https://datahubproject.io logo
Join Slack
Powered by
:wave: Hello, team! What would be the easiest way...
# troubleshoot
c
👋 Hello, team! What would be the easiest way to ADD AWS secret and key and token to the docker compose ENVs? (I want to ingest athena data and we using AWS credentials to access, not user and password). thanks a lot
b
Hey @cold-father-66356, Do you have docker-compose deployment? Are you using managed ingestion or running the ingestion process outside of the deployment?
c
I am using the webui and docker
I am getting this error now, after adding the token and keys to the envrinonment on docker compose
Copy code
'(UnrecognizedClientException) when calling the StartQueryExecution operation: The security token included in the request is invalid.\n'
maybe my token expired
ok, I updated the token but still not workin
b
you can supplied extra envs to your quickstart deployment : https://datahubproject.io/docs/quickstart/#customization
Managed ingestion runs in the actions container
so the required env vars needs to be added to that container
c
amazing, which containers was running ingestion was my question 🙌
b
c
@bulky-electrician-72362 would that be the place?
b
Yes
c
strange, for some reason it is not getting the token, Some tools do not accept the token, would that be the case? Are you guys using sqlalchemy or boto?
Worked 🙂. double quotes were the issue
b
Nice one. teamwork
c
@bulky-electrician-72362 quick question. What is the trick to configure this field? table_pattern.
I am trying.: table_pattern: [.*zeppelin’]. but I am getting an error
b
s3 igenstion?
c
I might gave screwed up my config. Give me a minute to be more precise on the error
The profiling enabled is not working:
Athena
b
What's the error you are getting?
c
Copy code
'[2022-12-16 17:05:15,318] ERROR    {datahub.entrypoints:183} - Failed to configure source (athena): 1 validation error for AthenaConfig\n'
           'profiling.enabled\n'
           '  extra fields not permitted (type=value_error.extra)\n',
Looks like Athena connector does not support this property?
b
cc: @dazzling-judge-80093 based on docs athena ingestion has profiling, is that true or is this a bug?
d
@cold-father-66356 please, can you try in this form?
Copy code
profiling: 
      enabled: true
c
Checking
@dazzling-judge-80093 worked 🙂. You guys rock 🙌
Quick question: Regarding spark lineage , do you have a special channel? I noticed that we do not have column lineage yet, and I would like to create dataassets name derived from the path on S3. (special use case) I have been using spline agent (they have amazing spark lineage), thanks a lot
d
Do you integrate Spline with Datahub or using separately?
c
Separately, Today is my first datahub day 🙂. Looks promising, In a few minutes I was able to get all metadata from Athena 1K tables, and now I am checking the spark lineage 🙂
Spline is pretty amazing regarding the spark-agent 🙂. but. I would prefer to use one tool 😂,
d
Yes, I was interested if you use spline-agent somehow with Datahub 🙂
c
well, I saw it is just a matter of sending the lineage 🙂
but the format from Spline is complex and very detailed
but that was my intention 😂 , send spline lineage to datahub 🙂
d
Column-level lineage is not supported yet with our spark agent, unfortunately, but I’m all ears for any solution proposal 🙂
c
If datahub would some how delegate the data lineage to spark spline agent would be amazing (no, I do not work on the code of spline 😂 )
do you have the datahub agent github link easily?
d
You mean, our spark lineage agent?
c
yes
d
c
thanks
Anyway, Great work regarding datahub. The company that I work for is inclined to use Atlan, but datahub is looking very promising and open source. (we would pay to be hosted for sure 😂 ,but open source is nice.)
@bulky-electrician-72362 quick question, what is the tab Query? Why I can not enable it? Can we use the datahub to query data?
d
The queries tab is enabled when there are usage data ingested. Usage is supported on certain sources like Redshift, BigQuery, Snowlake out of the box.
c
understood, That is what I thought 🙂. thanks a lot,
5 Views
Open in Slack
PreviousNext
Powered by