A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello , I was wondering if Snowflake ingestion can get away with <https://docs.snowflake.com/en/user-guide/security-access-control-privileges.html#:~:text=enables%20viewing%20the%20structure%20of%20a%20table%20(but%20not%20the%20data)%20via%20the%20DESCRIBE%20or%20SHOW%20command%20or%20by%20querying%20the%20Information%20Schema|REFERENCES> access instead of Select as it  only requires access to Metadata.

<https://docs.snowflake.com/en/user-guide/security-access-control-privileges.html#:~:text=enables%20viewing%20the%20structure%20of%20a%20table%20(but%20not%20the%20data)%20via%20the%20DESCRIBE%20or%20SHOW%20command%20or%20by%20querying%20the%20Information%20Schema>.

Lineage  runs a select sql which needs select rights.

Also profiling would require select access to profile the data

Other than that it might work. But will need to test it out

<@U027B7R23J4> I tried ingestion recipe with default configs by giving only REFERENCES access. As expected , only Lineage failed.

Good to know <@U02NFBYCVUL>. This will surely help other people in community

<@U027B7R23J4>. I am trying to get lineage by giving `imported privileges` access to only internal SNOWFLAKE DB. I am getting following error.

`grant imported privileges on database snowflake to role datahub_role;`

```[2022-01-21 16:01:50,718] WARNING  {datahub.ingestion.source.sql.snowflake:181} - Extracting lineage from Snowflake failed.Please check your premissions. Continuing...
Error was (snowflake.connector.errors.ProgrammingError) 000904 (42000): SQL compilation error: error line 13 at position 33
invalid identifier 'T.OBJECTS_MODIFIED'
[SQL: 
WITH table_lineage_history AS (
    SELECT
        r.value:"objectName" AS upstream_table_name,
        r.value:"objectDomain" AS upstream_table_domain,
        r.value:"columns" AS upstream_table_columns,
        w.value:"objectName" AS downstream_table_name,
        w.value:"objectDomain" AS downstream_table_domain,
        w.value:"columns" AS downstream_table_columns,
        t.query_start_time AS query_start_time
    FROM
        (SELECT * from snowflake.account_usage.access_history) t,
        lateral flatten(input =&gt; t.BASE_OBJECTS_ACCESSED) r,
        lateral flatten(input =&gt; t.OBJECTS_MODIFIED) w
    WHERE r.value:"objectId" IS NOT NULL
    AND w.value:"objectId" IS NOT NULL
    AND w.value:"objectName" NOT LIKE '%.GE_TMP_%'
    AND t.query_start_time &gt;= to_timestamp_ltz(1642636800000, 3)
    AND t.query_start_time &lt; to_timestamp_ltz(1642723200000, 3))
SELECT upstream_table_name, downstream_table_name, upstream_table_columns, downstream_table_columns
FROM table_lineage_history
WHERE upstream_table_domain = 'Table' and downstream_table_domain = 'Table'
QUALIFY ROW_NUMBER() OVER (PARTITION BY downstream_table_name, upstream_table_name ORDER BY query_start_time DESC) = 1        ]```

`OBJECTS_MODIFIED`  column does not exist in our DB. I think It’s only included in preview feature.
``` This behavior change is included in the 2021_10 bundle introduced in the Snowflake 5.40 release. The bundle is disabled by default in 5.40, but can be enabled for testing purposes.

The bundle is planned to be enabled by default in January 2022; however, the date is subject to change.```
<https://community.snowflake.com/s/article/Pending-Behavior-Change-Log>

Given that Jan 2022 is about to end can you try to enable this and test this again? By the time we ship this it might get enabled by default

It worked after enabling preview feature . Thanks <@U027B7R23J4> for your help