Hi team I have dbt manifest generated by airflow and I want to ingest that in datahub So initially we thought to put this manifest file on s3 and use recipe with source type as s3 and read from there and then ingest in datahub but it seems like this is not correct way So can we use this way or there is some other way to automate this process ? And I can stateful ingestion option in dbt recipe When and which case we prefer that ?

S3 source is for ingesting metadata from s3 files. You need to copy the manifest file from s3 to local filesystem and use the dbt ingestion source to ingest dbt metadata.

And what is stateful ingestion

stateful ingestion is more for checkpointing where you save a state and next time you don’t start from a clean slate but from the checkpoint. This way you can continue where you left off ->

@dazzling-judge-80093 we recently merged in a change to support reading manifest files from s3 for the Dbt source.

ahh, sorry, I was not aware of this

Thanks for this But one thing how we will ingest if have multiple glossary files then we have to write multiple recipe files?

is this last question related to dbt source? seems like a question about business glossary

Now I have asked related to glossary

were you able to verify that the dbt source works with your s3? just don't want to lose track of that item. Maybe start a different thread around glossary and multiple recipes?