Hello Does Datahub support Redshift Serverless or there is a DataHub #troubleshoot

Hello, Does Datahub support Redshift Serverless ?...

lively-minister-2773

12/19/2022, 1:38 PM

Hello, Does Datahub support Redshift Serverless ? or there is a work around as stl_insert is not available in Redshift Serverless https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-monitoring.html You can’t query STL, STV, SVCS, SVL, and some SVV system tables and views with Amazon Redshift Serverless, except the following: …. Thank you

dazzling-judge-80093

12/19/2022, 3:50 PM

I think if you disable lineage/usage extraction it should work

lively-minister-2773

12/19/2022, 7:02 PM

thank you Tamas, I’ll give it a try

lively-minister-2773

12/20/2022, 9:03 AM

That worked @dazzling-judge-80093 thank you for your help

miniature-waiter-20271

01/23/2023, 12:34 PM

For me too, thanks both!

brainy-balloon-97302

05/26/2023, 4:45 PM

Hi guys here is my config but mine is still failing. Could you please help or have any ideas?

Copy code

source:
    type: redshift
    config:
        host_port: '<http://XXXXXXXXX-corpvpc.XXXXXXXXX.us-west-2.redshift-serverless.amazonaws.com:5439|XXXXXXXXX-corpvpc.XXXXXXXXX.us-west-2.redshift-serverless.amazonaws.com:5439>'
        database: e2e_datahub_dev2
        username: datahub_user
        table_lineage_mode: stl_scan_based
        include_table_lineage: false
        include_usage_statistics: false
        include_tables: true
        include_views: true
        profiling:
            enabled: true
            profile_table_level_only: true
        stateful_ingestion:
            enabled: true
        password: '${redshift_db_sa}'

brainy-balloon-97302

05/26/2023, 4:47 PM

Keep getting this error:

Copy code

redshift_connector.error.ProgrammingError: {'S': 'ERROR', 'C': '42501', 'M': 'permission denied for relation stl_insert', 'F': '../src/pg/src/backend/catalog/aclchk.c', 'L': '2945', 'R': 'aclcheck_error'}

brainy-balloon-97302

05/26/2023, 4:47 PM

using cli v0.10.2.1

brainy-balloon-97302

05/26/2023, 5:15 PM

It looks like the query is still including the

stl_insert

table.

Copy code

[2023-05-26 17:14:35,434] DEBUG    {datahub.ingestion.source.redshift.redshift_schema:89} - Query : 
        select
            database,
            schema,
            "table",
            size,
            tbl_rows,
            estimated_visible_rows,
            skew_rows,
            last_accessed
        from
            pg_catalog.svv_table_info as ti
        left join (
            select
                tbl,
                max(endtime) as last_accessed
            from
                pg_catalog.stl_insert
            group by
                tbl) as la on
            (la.tbl = ti.table_id)
       ;

brainy-balloon-97302

05/26/2023, 5:24 PM

Here is my latest config that is still failing. Seems like no matter what I do it still runs the query with

stl_insert

Copy code

source:
    type: redshift
    config:
        host_port: '<http://XXXXXXXXX-corpvpc.XXXXXX.us-west-2.redshift-serverless.amazonaws.com:5439|XXXXXXXXX-corpvpc.XXXXXX.us-west-2.redshift-serverless.amazonaws.com:5439>'
        database: e2e_datahub_dev2
        username: datahub_user
        include_table_lineage: false
        include_copy_lineage: false
        include_table_location_lineage: false
        include_unload_lineage: false
        include_usage_statistics: false
        incremental_lineage: false
        store_last_usage_extraction_timestamp: false
        include_tables: true
        include_views: true
        profiling:
            enabled: false
            profile_table_level_only: false
        stateful_ingestion:
            enabled: false
        password: '${redshift_db_sa}'

brainy-balloon-97302

05/26/2023, 7:22 PM

Just for others, I was not able to get the type

redshift

to work only

redshift-legacy

here is my config

Copy code

source:
              type: redshift-legacy
              config:
                # Coordinates
                host_port: <http://XXXX-XXXXX-corpvpc.XXXXX.us-west-2.redshift-serverless.amazonaws.com:5439|XXXX-XXXXX-corpvpc.XXXXX.us-west-2.redshift-serverless.amazonaws.com:5439>
                database: XXXX
            
                # Credentials
                username: datahub_user
                password: '${DATAHUB_USER_PASS}'
            
                include_table_lineage: false
                include_table_location_lineage: false
                include_unload_lineage: false

            
                profiling:
                  enabled: false
                stateful_ingestion:
                  enabled: false
                  



            sink:
              type: datahub-rest
              config:
                token: '${DATAHUB_GMS_TOKEN}'
                server: '${DATAHUB_GMS_URL}'
            pipeline_name: "redshift_e2e_XXXXX"

6 Views

Open in Slack

Previous Next