when ingesting Snowflake data, I see this error re...
# troubleshoot
r
when ingesting Snowflake data, I see this error reported in the terminal
Copy code
[2021-11-02 13:27:38,054] WARNING  {datahub.ingestion.source.sql.snowflake:183} - Extracting lineage from Snowflake failed.Please check your premissions. Continuing...
Error was (snowflake.connector.errors.ProgrammingError) 000904 (42000): SQL compilation error: error line 13 at position 33
invalid identifier 'T.OBJECTS_MODIFIED'
[SQL:
WITH table_lineage_history AS (
    SELECT
        r.value:"objectName" AS upstream_table_name,
        r.value:"objectDomain" AS upstream_table_domain,
        r.value:"columns" AS upstream_table_columns,
        w.value:"objectName" AS downstream_table_name,
        w.value:"objectDomain" AS downstream_table_domain,
        w.value:"columns" AS downstream_table_columns,
        t.query_start_time AS query_start_time
    FROM
        (SELECT * from snowflake.account_usage.access_history) t,
        lateral flatten(input => t.BASE_OBJECTS_ACCESSED) r,
        lateral flatten(input => t.OBJECTS_MODIFIED) w
    WHERE r.value:"objectId" IS NOT NULL
    AND w.value:"objectId" IS NOT NULL
    AND w.value:"objectName" NOT LIKE '%.GE_TMP_%'
    AND t.query_start_time >= to_timestamp_ltz(1635724800000, 3)
    AND t.query_start_time < to_timestamp_ltz(1635811200000, 3))
SELECT upstream_table_name, downstream_table_name, upstream_table_columns, downstream_table_columns
FROM table_lineage_history
WHERE upstream_table_domain = 'Table' and downstream_table_domain = 'Table'
QUALIFY ROW_NUMBER() OVER (PARTITION BY downstream_table_name, upstream_table_name ORDER BY query_start_time DESC) = 1        ]
(Background on this error at: <http://sqlalche.me/e/13/f405>).
Based on snowflake’s docs, I dont see OBJECTS_MODIFIED field in the snowflake.account_usage.access_history. Can this be a bug?
seems to be related to this PR - https://github.com/linkedin/datahub/pull/3331, but for some reason I do not see that column in our tables
l
@red-pizza-28006 We will fix this
r
thank you @loud-island-88694
any ETA on this @loud-island-88694?
l
This issue will not fail the job right now - it is only a warning. We will get to this next week
r
Correct, but i was really curious to see the table lineage from within Snowflake, thats why i asked 😬
@loud-island-88694 any ETA on this? I am still seeing this exception?
l
@red-pizza-28006 Please expect an update later this week
are you running DataHub in production now?
b
Is this fixed ? I am getting this error for Lineage.