https://datahubproject.io logo
Join Slack
Powered by
Does anyone know when ingesting DBT files here - <...
# ingestion
r
Does anyone know when ingesting DBT files here - https://datahubproject.io/docs/metadata-ingestion/source_docs/dbt, if I can specify S3 files?
b
Currently I believe the file needs to be local to the fs where the source is being executed. However, I think this would make for a useful extension of this source..
m
Makes sense to me. Would be a great contrib if there is interest!
w
https://github.com/linkedin/datahub/pull/3739 Got a strange error though
Copy code
+ exit 1
docker-compose-with-elasticsearch.quickstart.yml is out of date.
Error: Process completed with exit code 1.
m
interesting... @witty-butcher-82399 does this work with s3 URI-s ?
I assumed some s3 specific boto3 lib would be needed
w
Test failed because of the following diff
Copy code
āžœ  quickstart git:(dbt_config_uris) āœ— diff docker-compose-with-elasticsearch.quickstart.yml temp-with-elasticsearch.quickstart.yml
74c74
<     - ES_JAVA_OPTS=-Xms256m -Xmx256m
---
>     - ES_JAVA_OPTS=-Xms256m -Xmx256m -Dlog4j2.formatMsgNoLookups=true
Should be fixed now: https://github.com/linkedin/datahub/pull/3739/commits/5de627a0d26c932e41f16033d30d3969f46287ef
No, current proposal is not working with S3 URIs. Of course, metadata files could be in S3 and make them accessible if public HTTP access is enabled for them. But my intention is to make them accessible from a git repo or similar. I considered S3 and boto3, but that would require also a role being granted for having access. Not sure if that would make things easier. I would like to have feedback from the community on that.
b
we will be putting a fix up to address this^
cc @green-football-43791
w
I considered S3 and boto3, but that would require also a role being granted for having access. Not sure if that would make things easier. I would like to have feedback from the community on that.
I also thought that S3 would have little value for those working on Google Cloud Platform, which is a common case for DBT, if Iā€™m not wrong.
Anyway, I have added the missing tests here https://github.com/linkedin/datahub/pull/3739
m
IF you use the dbt cloud service, they surface all of those artifacts both through URL as well as via their GraphQL api (you can crawl the DAG via GQL).
m
Thanks @witty-butcher-82399 šŸ‘€ ... @mysterious-zebra-62364: presumably those URLs are protected via creds... how should we pass this info to the dbt source?
m
I'd just add their API key into the config just as with the other ingestion sources
p
An integration with DBT Cloud via the API key would be great!
b
We should definitely add this. Seems like low hanging fruit
w
I have tested by fetching from github repos. Now I would like to include authentication too so I will need to include the token in the DBT source config.
Copy code
curl -H "Authorization: token ${PAT}" <https://raw.githubusercontent.com/user/repo/main/file.txt>
@purple-ghost-64569 How does auth work in the case of DBT cloud?
p
@witty-butcher-82399 I am currently not quite sure how to download the artifacts from dbt Cloud. I thought I read that this can be accomplished but I haven't found it in the docs: https://docs.getdbt.com/dbt-cloud/api-v2
m
@purple-ghost-64569 - Its in the API docs for 
Runs
= https://docs.getdbt.com/dbt-cloud/api-v2#operation/getArtifactsByRunId
w
Interesting! We have tested by placing the manifest/catalog file in a github repo.
Copy code
source:
  type: dbt
  config:
    env: DEV
    # Coordinates
    manifest_path: https://${CATALOG_TOKEN}:@raw.github.mpi-internal.com/...org.../...repo.../master/manifest.json
    catalog_path: https://${MANIFEST_TOKEN}:@raw.github.mpi-internal.com/...org.../...repo.../master/catalog.json
Github also uses token authorization and we made it work by passing the token in the URL. This should also work with DBT. It would be great if someone could confirm šŸ˜…
m
@witty-butcher-82399 - I'll try to test this on the dbt cloud api later this week.
w
šŸ‘
p
Did you ever dig into implementing this for dbt cloud?
w
I just did the implementation for http fetching and we are using it with github repos, never tested with dbt cloud
p
Gonna be easy enough to spin up an airflow task for this.
5 Views
Open in Slack
PreviousNext
Powered by