A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi everyone,

I need help ingesting *dbt* in my Datahub. My team deployed it in *Kubernetes* (GCP GKE) and now I don’t know where I should store manifest.json and catalog.json. I’ve tried in a bucket on Google Cloud Storage and passed the URI as an argument, but it didn’t work. Now I am wondering if I should have stored them in a bucket inside GKE (If there is one, sorry I know almost nothing about Kubernetes yet).

I see another person had the same problem in the past, but couldn’t find a solution: <https://datahubspace.slack.com/archives/CV2KB471C/p1656928845742369>

<@U041JQ60ZL5> we don’t yet have support for gcs as a source for the dbt artifact files, this would be a very welcome contribution to the source if you are interested!

<@U041JQ60ZL5> maybe you can mount the gcs bucket to file system of the pod running the ingestion using something like FUSE

Hi, <@UV0M2EB8Q>, thanks for your answer. Is this fully operational in other clouds (Heroku, AWS)?

<@U030H0S9QUV>, is this <https://github.com/libfuse/libfuse|FUSE> you are referring to?

<@U041JQ60ZL5> <https://github.com/GoogleCloudPlatform/gcsfuse> this is the one

You might need to update the action image

Thank you, <@U030H0S9QUV>, I’ll try this approach.

Thanks <@U030H0S9QUV> this looks like a promising approach until we add native gcs support

<@UV0M2EB8Q>, if the cloud were AWS, this won’t be a problem, am I correct?

more importantly, the files just need to be on S3

so if your company has an S3 bucket where you could drop your dbt artifacts, it would work