https://datahubproject.io logo
Join Slack
Powered by
I have this job definition ``` # Snowflake to Data...
# troubleshoot
k
I have this job definition
Copy code
# Snowflake to Datahub recipe configuration
# To run an ingestion run: datahub ingest -c ./metadata-ingestion/recipes/snowflake_to_datahub_rest.yml
# pipeline_name: "my_snowflake_pipeline_1"
source:
  type: snowflake
  config:
    # Coordinates
    host_port: ${SNOWFLAKE_ACCOUNT}
    warehouse: 'AGGREGATION_COMPUTE'

    # Credentials
    username: ${SNOWFLAKE_USERNAME}
    password: ${SNOWFLAKE_PASSWORD}
    role: 'XADMIN'

    env: "PROD"

    profiling:
      enabled: False

    database_pattern:
      allow:
        - "DWXX"
        - "VISIBILITY"
        - "STRATEGY_AND_PLANNING"
        - "ABC_SHIPPER_STRATEGY_AND_PLANNING"
        - "XYZ"
        - "MARKETING"
        - "GLOBAL_OPERATIONS"
        - "CENTRAL_STRATEGY_AND_PLANNING"
        - "FINANCE"
      deny:
        - "DEV"
        - "ANALYST_DEV"

    table_pattern:
      ignoreCase: False

    include_tables: True
    include_views: True
    include_table_lineage: False


    stateful_ingestion:
      enabled: True
      remove_stale_metadata: True



sink:
  type: "datahub-rest"
  config:
    server: ${DATAHUB_GMS_HOST}:8080
I get this validation error:
Copy code
1 validation error for SnowflakeConfig                                                                                                              │
│ stateful_ingestion                                                                                                                                  │
│   extra fields not permitted (type=value_error.extra)
which is really vague, I have not any idea of what I am doing wrong
h
Hi @kind-psychiatrist-76973, I'm taking a look. Will update you shortly.
k
thanks
h
Unless something weird is coming via the environment variables being referenced, this recipe should work (with pipeline_name uncommented).
I have been able to successfully run this recipe from my end, with the vars appropriately configured.
Could you try commenting out non-required fields and replacing the environment variables with their actual values to see what is causing the issue?
k
OK I am away from work (it’s 18:27 in the UK). I will try again tomorrow, but with or without 
pipeline_name: "my_snowflake_pipeline_1"
it was not working so I will check the if environment is passing something weired.
👍 1
m
Also hopefully you are on a new-ish datahub cli version that supports stateful_ingestion 🙂
m
We are still facing the same issue
k
the version is 0.8.33
and yes stateful_ingestion equals to true
m
@kind-psychiatrist-76973 is my colleague and we are working on this state full ingestion. We tried explicitly running and we still have the same error
k
Copy code
root@sh:/datahub# curl ${DATAHUB_GMS_HOST}/config
{
  "models" : { },
  "versions" : {
    "linkedin/datahub" : {
      "version" : "v0.8.33",
      "commit" : "72046bf43d39b1b1db6457ab567d2ca9e3b78bbe"
    }
  },
  "managedIngestion" : {
    "defaultCliVersion" : "0.8.33",
    "enabled" : true
  },
  "statefulIngestionCapable" : true,
  "supportsImpactAnalysis" : true,
  "telemetry" : {
    "enabledCli" : true,
    "enabledIngestion" : false
  },
  "datasetUrnNameCasing" : false,
  "retention" : "true",
  "noCode" : "true"
}
I found the issue: the cron image had still the old version of the Python acryl-datahub package, we updated the requriements.txt file and that it started to work. Sorry about this.
👍 1
2 Views
Open in Slack
PreviousNext
Powered by