Hi , I am trying to ingest data from superset to datahub. charts, dashboards and their internal ineage was loaded successfully, but not able to find any inputs or the source datasets to get the end - to - end lineage. Any help would be much appreciated .. For reference, config_file

source:
  type: superset
  config:
    # Coordinates
    connect_uri: <>

    # Credentials
    username: admin
    password: admin

sink:
  type: "datahub-rest"
  config:
    # TODO : Parameterizing the sink url from single config
    server: <>

under lying source for superset datasets is redshift ..

When you say end-to-end lineage what were you expecting that is missing?

source-datasets [lying in redshift] -> [dataset in superset] -> [charts] -> [dashboards]

lineage part from

source-datasets [lying in redshift] -> [dataset in superset] -> [charts]

is the missing link ..

source-datasets [lying in redshift]

and

[dataset in superset]

are the same thing. Just represented differently in superset (as connection) vs original source. For getting schema from redshift you will have to use redshift source.

Yes, i have redshift source ingested as well , but lineage from

redshift to superset chart

is not there..

can you run the recipe, write output to file and check if there is anything called

chartInfo

or something like that there? The code is there for this lineage https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/superset.py#L290 so it should be coming

okay thanks checking the same.

Is the database name configured in redshift connection in superset?

yes i checked basically, datahub cli uses

Display Name

, which was different from the original Database Name in my case causing the issue . but now i checked the backend (entity type) is external in the above case , it should be redshift any idea to change the same updated (current -state) :

"inputs": [
                            {
                                "string": "urn:li:dataset:(urn:li:dataPlatform:external,mydb.<schema-name>.<table-name>,PROD)"
                            }
                        ],

expected :

"inputs": [
                            {
                                "string": "urn:li:dataset:(urn:li:dataPlatform:redshift,Amazon Redshift.<schema-name>.<table-name>,PROD)"
                            }
                        ],

Can you tell what your sqlalchemy url looks like? This https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py#L84 controls platform name

sqlalchemy_uri.startswith("jdbc:postgres:")
        and sqlalchemy_uri.index("redshift.amazonaws") > 0

Does your uri not satisfy either of these conditions?

Need to check what url is generated by superset under the hood.. (because on ui it accepts only host and port and credentials) checking the same on superset docs.

Found it https://superset.apache.org/docs/databases/redshift

yes , https://superset.apache.org/docs/databases/installing-database-drivers

redshift+psycopg2://<userName>:<DBPassword>@<AWS End Point>:5439/<Database Name>

Raised PR for the fix https://github.com/linkedin/datahub/pull/3944. Once this is merged and a release is made the problem with

external

should be fixed

Thanks for all the support.

The python client is released very frequently https://github.com/acryldata/datahub/releases