A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi, I am trying to ingest DBT docs from S3 into datahub. Is there a way to do it without having to download the files into the local container and then bring it to datahub? In other words, can I directly fetch these files from S3 and put it on datahub?

The dbt source can read from s3! Take a look at the `aws_connection` config param for details

Thanks <@U01GZEETMEZ>, so instead of using <https://datahubproject.io/docs/generated/ingestion/sources/dbt/|this> , I can ingest directly from s3  and follow <https://datahubproject.io/docs/generated/ingestion/sources/s3/|this>?

If you're trying to ingest dbt metadata, you'll need to use the dbt source and put the <s3://bucket/file.json> paths + aws credentials in the config

Thank you!
```source:
  type: "dbt"
  config:
    # Coordinates
    # To use this as-is, set the environment variable DBT_PROJECT_ROOT to the root folder of your dbt project
    manifest_path: "${DBT_PROJECT_ROOT}/target/manifest_file.json"
    catalog_path: "${DBT_PROJECT_ROOT}/target/catalog_file.json"
    sources_path: "${DBT_PROJECT_ROOT}/target/sources_file.json" # optional for freshness
    test_results_path: "${DBT_PROJECT_ROOT}/target/run_results.json" # optional for recording dbt test results after running dbt test```
just to be very sure, so in the above, along with aws connection details, instead of DBT_PROJECT_ROOT we can point it to s3 paths, is that what you mean?

Hi <@U01GZEETMEZ>, sorry to ping you 1:1 but I have been trying to get an answer for this for quite some time now. It would be great if you could help me out here.

Do we know what perms does the user need for mongodb integration? Is there anything specific that we need to keep in mind before we create a user? cause I am getting this error:
```pymongo.errors.OperationFailure: not authorized on core to execute command ```
the user has read-only perms on the cluster.