A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

IMG_20230518_155845.jpg

Hi, I am getting this error while running recipe file, It is saying pipeline_name required if stateful_ingestion enabled, what is pipeline_name, not mentioned in the documentation

Hi <@U054NUV9D9T> 
Please see below sample recipe , pipeline_name as top level attribute in recipe 

source:
  type: "snowflake"
  config:
    username: &lt;user_name&gt;
    password: &lt;password&gt;
    role: &lt;role&gt;
    host_port: &lt;host_port&gt;
    warehouse: &lt;ware_house&gt;
    # Rest of the source specific params ...
# This is mandatory. Changing it will cause old telemetry correlation to be lost.
_*pipeline_name: "my_snowflake_pipeline_1"*_

# Pipeline-level datahub_api configuration.
datahub_api: # Optional. But if provided, this config will be used by the "datahub" ingestion state provider.
    server: "<http://localhost:8080|http://localhost:8080>"
    
sink:
  type: "datahub-rest"
  config:
    server: '<http://localhost:8080|http://localhost:8080>'

reporting:
    ◦ type: "datahub" # Required
    config: # Optional. 
      datahub_api: # default value
        server: "<http://localhost:8080>"