Hello, hope you are doing well, i got a query : for DB we are using Redshift but for pipeline we are not using Airflow.. we are using LAMBDA..so can we ingest our lineage

Here are some example python code to generate pipeline lineage programatically -> https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_job_dataflow_new_api.py https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_job_dataflow_new_api_simple.py

thank you for sharing

We currently don’t have Quicksight ingestion source but any contribution is welcomed.

Thank you @dazzling-judge-80093 for your response

For queries you need to run the redshift usage source -> https://datahubproject.io/docs/generated/ingestion/sources/redshift#module-redshift-usage For ingestion: You need to enable

include_table_lineage

in your redshift recipe, and this property should be enabled by default. It should capture lineage based on the insert/create table as/etc queries you ran on Redshift. Redshift’s query history is just a couple of days, so make sure you recently run such queries. Stats is enabled if you run profiling on the table.

@dazzling-judge-80093 Hello, hope you are doing well. What could be the possible reasons for validation and stats not getting enabled though we have ensured the enabling steps?

Can you share the ingestion summary log? Hopefully from that we can see what is happening

@dazzling-judge-80093 Hello, hope you are doing we;;. Sharing the log