https://datahubproject.io logo
Join Slack
Powered by
Hello, Schema information not being ingested for s...
# troubleshoot
s
Hello, Schema information not being ingested for snowflake tables. I have the tables on the UI but not the columns metadata. I have as well the following erros (attached). Any ideia on how to fix this ? See here my ingestions yaml:
Copy code
source:
  type: snowflake
  config:
    account_id: ${DH_SNOWFLAKE_ACCOUNT_ID}
    warehouse: ${DH_SNOWFLAKE_WAREHOUSE}
    username: ${DH_SNOWFLAKE_USER}
    password: ${DH_SNOWFLAKE_PASSWORD}
    role: ${DH_SNOWFLAKE_ROLE}
    include_tables: True
    include_views: True

    ignore_start_time_lineage: true

    stateful_ingestion:
      enabled: True 
      remove_stale_metadata: True 

    profiling:
      enabled: true
    profile_pattern:
      allow:
        - 'DATABASE_NAME.SCHEMA_NAME.*'
    database_pattern:
      allow:
        - "OIP"
    schema_pattern:
      allow:
        - "SCHEMA_NAME"
    table_pattern:
      deny:
       - '.*\._AIRBYTE_.*'

pipeline_name: "snowflake_ingestion"

sink:
  type: datahub-rest
  config:
    server: ${DATAHUB_SERVER}
Thank you guys in advance !
It seems to be a problem linked with pyarrow. If a uninstall pyarrow and install greenlet, it works just fine. Pyarrow is used at some point by acryl-datahub[snowflake-usage, snowflake]. But if we uninstall it, it doesn't cause any error and correct this column names problem.
d
snowflake’s python connector needs pyarrow
s
@dazzling-judge-80093 Do you know which is the exact version of 
pyarrow
that works fine the 
acryl-datahub[snowflake-usage, snowflake]
?
d
pyarrow>=6.0.1
should be fine
2 Views
Open in Slack
PreviousNext
Powered by