Hi There, Curious if Datahub supports Liquid Variable logic in LookML where an explore can be derived from one of 2 underlying tables?

We’d like to be able to support this, but it’s not yet clear to me that we actually have the information to resolve the liquid variable references when we’re parsing the LookML

is that a limitation due to the data you receive via LookML

Like in this example, how can we actually determine the value of

%28traffic_analysis_summary.post_uri._in_query

? We only receive the actual LookML file’s contents

i think my expectation is that it would see that its derived from

metric_stores_modeled.traffic_summary.traffic_analysis_page_daily_summary

OR

metric_stores_modeled.traffic_summary.traffic_analysis_site_daily_summary

OR 

metric_stores_modeled.traffic_summary.traffic_analysis_site_monthly_summary

I’m not quite sure that’s the right solution - if we did that, the meaning of a lineage edge would be “might be derived from” rather than “is derived from”.

right now just in query

Yes, I would say in our scenario it's derived from both as both sources are actively used - in Looker it's dynamic based upon how the Explore user chooses to filter their query, either getting daily granularity or monthly

➕

That makes sense - we’re looking into it

thank you!

it seems like this is a common enough pattern that we should handle it correctly, even if we don’t build out full support for liquid variables

Thank you!

It’s currently in our backlog. I’d be open to help 🙂

what do you mean by open to help?

ah sorry I should’ve been more clear - we’d love contributions/PRs 🙂

sorry i still dont know that i understand

Hey guys - just catching up apologies - @swift-painter-68980 we could submit an implementation of this feature ourselves in an effort to speed things along and push upstream as a contribution to the OSS project. @gray-shoe-75895 I'd have to check internally if we're open to that, though I think the constraint for us is likely to be be Java skills on the team as I believe this would be a change in the CLI components (which is Java 11) and we're mostly a python team. I was trying to build locally yesterday evening out of curiosity, though I was having issues getting the Gradle version to build on my M1 Mac.

Thanks for the input tom!

We should definitely figure out the M1 / gradle issues (you can DM me some error messages). However, the ingestion subsystem is completely written in python. The part that you’d need to change is here https://github.com/datahub-project/datahub/blob/c5fc2ea7980b2d656fe41ab21228802213[…]-ingestion/src/datahub/ingestion/source/looker/lookml_source.py, where we’re currently skipping lookml sql queries with embedded liquid variables

That's great - thanks for the pointer @gray-shoe-75895!