Hi we are testing out Redash metadata ingestion library Is t DataHub #ingestion

Hi, we are testing out Redash metadata ingestion l...

thousands-tailor-5575

09/15/2021, 3:31 PM

Hi, we are testing out Redash metadata ingestion library. Is there a posibility to extract chart -> dataset lineage? It looks like there is only Chart -> Data Platform lineage being retrieved.

mammoth-bear-12532

09/15/2021, 3:50 PM

@square-greece-86505: might have some ideas.

mammoth-bear-12532

09/16/2021, 1:15 AM

@square-greece-86505: do you have some suggestions here?

adamant-pharmacist-61996

09/16/2021, 4:46 AM

@thousands-tailor-5575 we’ve been looking at the same thing. It would be great to refactor the code so that we can inject sql parsers (I think the LookML recipe might have this functionality)

square-greece-86505

09/16/2021, 2:57 PM

yes we can use the code from LookML. I am planning to work on it on this weekend

square-greece-86505

09/20/2021, 11:43 AM

Hi everyone, I made a PR for Redash: • Add feature parse table names from SQL • Adding

QualifiedNameParser

and several extended dataclasses for parsing table name into fully qualified name based on database type. As an example, table name

orders

in Postgres database should be normalized into

databaseName.public.orders

if only table name part given • Some small refactors to improve Would be super helpful for any comment, suggestion or even give it a try on your environment 😊 https://github.com/linkedin/datahub/pull/3267 cc: @adamant-pharmacist-61996 @thousands-tailor-5575 @mammoth-bear-12532

adamant-pharmacist-61996

09/21/2021, 6:56 AM

hey @square-greece-86505 I’ve put a ticket into our sprint to run the evaluation, should be done in a week 🙂 Sorry! Things are a bit stretched at the moment

square-greece-86505

09/21/2021, 3:13 PM

thanks @adamant-pharmacist-61996 😉

adamant-pharmacist-61996

10/07/2021, 1:09 AM

hey guys, just an update. We’re now looking at the PR and will get back y’all shortly

mammoth-bear-12532

10/07/2021, 1:39 AM

Hey @adamant-pharmacist-61996 🙂 I merged the PR last night 😱

mammoth-bear-12532

10/07/2021, 1:39 AM

so you can test on latest … and we can fix forward if you find any issues

adamant-pharmacist-61996

10/07/2021, 1:43 AM

adamant-pharmacist-61996

10/07/2021, 1:44 AM

Okey doke. Let's get it a whirl

square-greece-86505

10/12/2021, 6:34 AM

Thanks @mammoth-bear-12532 Please let us know for any issues @adamant-pharmacist-61996

adamant-pharmacist-61996

10/12/2021, 6:35 AM

We're looking at it now😃

adamant-pharmacist-61996

10/12/2021, 6:35 AM

I'll ask our engineer to feedback soon

❤️ 1

Open in Slack

Previous Next