https://datahubproject.io logo
#ingestion
Title
# ingestion
s

salmon-area-51650

06/15/2022, 5:09 PM
👋Hey there! I have an issue with
dbt
ingestion. The cronjob which is executing the integration was ok (no errors) but I cannot see dbt platform in the UI. This is the configuration of the job:
Copy code
source:
      type: "dbt"
      config:
        # Coordinates
        manifest_path: "<s3://bucket/manifest.json>"
        catalog_path: "<s3://bucket/catalog.json>"
        sources_path: "<s3://bucket/sources.json>"

        aws_connection:
          aws_region: "eu-west-2"

        # Options
        target_platform: "snowflake"
        load_schemas: True # note: if this is disabled
        env: STG
    sink:
      type: "datahub-rest"
      config:
        server: "<http://datahub-datahub-gms:8080>"
And this is the output of the job:
Copy code
'failures': {},
  'cli_version': '0.0.0+docker.b4bf1d4',
  'cli_entry_location': '/usr/local/lib/python3.8/site-packages/datahub/__init__.py',
  'py_version': '3.8.13 (default, May 28 2022, 14:23:53) \n[GCC 10.2.1 20210110]',
  'py_exec_path': '/usr/local/bin/python',
  'os_details': '...-glibc2.2.5',
  'soft_deleted_stale_entities': []}
 Sink (datahub-rest) report:
 {'records_written': 2048,
  'warnings': [],
  'failures': [],
  'downstream_start_time': datetime.datetime(2022, 6, 15, 16, 28, 19, 669215),
  'downstream_end_time': datetime.datetime(2022, 6, 15, 16, 29, 23, 420550),
  'downstream_total_latency_in_seconds': 63.751335,
  'gms_version': 'v0.8.38'}

 Pipeline finished with 1061 warnings in source producing 2048 workunits
And there is no any `dbt`platform in the UI Any idea? Thanks in advance!
b

bulky-soccer-26729

06/15/2022, 5:27 PM
Hi Miguel! so is everything that you just ingested successfully showing up as a snowflake platform?
s

salmon-area-51650

06/16/2022, 5:20 AM
Hi @bulky-soccer-26729 I don’t know to be honest. I would expect having that information as a new
dbt
platform
I’ve moved this question to #troubleshoot https://datahubspace.slack.com/archives/C029A3M079U/p1655372894852579 I think that’s the right forum 🙂
👍 1
9 Views