Hi There, Curious if Datahub supports Liquid Vari...
# troubleshoot
s
Hi There, Curious if Datahub supports Liquid Variable logic in LookML where an explore can be derived from one of 2 underlying tables?
@lemon-yacht-62789 fyi
For context this is what liquid variable looks like in datahub currently
Copy code
{% if %28traffic_analysis_summary.post_uri._in_query or traffic_analysis_summary.article_title._in_query%29 %} metric_stores_modeled.traffic_summary.traffic_analysis_page_daily_summary {% elsif %28traffic_analysis_summary.date_date._in_query or traffic_analysis_summary.date_week._in_query%29 %} metric_stores_modeled.traffic_summary.traffic_analysis_site_daily_summary {% else %} metric_stores_modeled.traffic_summary.traffic_analysis_site_monthly_summary {% endif %}
g
We’d like to be able to support this, but it’s not yet clear to me that we actually have the information to resolve the liquid variable references when we’re parsing the LookML
s
is that a limitation due to the data you receive via LookML
g
Like in this example, how can we actually determine the value of
%28traffic_analysis_summary.post_uri._in_query
? We only receive the actual LookML file’s contents
s
i think my expectation is that it would see that its derived from
Copy code
metric_stores_modeled.traffic_summary.traffic_analysis_page_daily_summary

OR

metric_stores_modeled.traffic_summary.traffic_analysis_site_daily_summary

OR 

metric_stores_modeled.traffic_summary.traffic_analysis_site_monthly_summary
and you would see those within our snowflake instance
g
I’m not quite sure that’s the right solution - if we did that, the meaning of a lineage edge would be “might be derived from” rather than “is derived from”.
A question: is
_in_query
the only type of
{%
liquid variable reference that you’re using, or are there others as well?
e.g. are you using
_user_attributes
or other variables as well?
s
right now just in query
the explore tho is derived from both tables and depending how you query the explore in looker would determine which table it uses, i dont think it would be incorrect neccesarily to say both are applicable
l
Yes, I would say in our scenario it's derived from both as both sources are actively used - in Looker it's dynamic based upon how the Explore user chooses to filter their query, either getting daily granularity or monthly
s
g
That makes sense - we’re looking into it
s
thank you!
This documentation might be useful as it paints exactly what we are doing in Looker
g
it seems like this is a common enough pattern that we should handle it correctly, even if we don’t build out full support for liquid variables
s
Thank you!
Out of curiosity @gray-shoe-75895 is this queued up in near term or in the backlog?
g
It’s currently in our backlog. I’d be open to help 🙂
s
what do you mean by open to help?
g
ah sorry I should’ve been more clear - we’d love contributions/PRs 🙂
s
sorry i still dont know that i understand
l
Hey guys - just catching up apologies - @swift-painter-68980 we could submit an implementation of this feature ourselves in an effort to speed things along and push upstream as a contribution to the OSS project. @gray-shoe-75895 I'd have to check internally if we're open to that, though I think the constraint for us is likely to be be Java skills on the team as I believe this would be a change in the CLI components (which is Java 11) and we're mostly a python team. I was trying to build locally yesterday evening out of curiosity, though I was having issues getting the Gradle version to build on my M1 Mac.
s
Thanks for the input tom!
g
We should definitely figure out the M1 / gradle issues (you can DM me some error messages). However, the ingestion subsystem is completely written in python. The part that you’d need to change is here https://github.com/datahub-project/datahub/blob/c5fc2ea7980b2d656fe41ab21228802213[…]-ingestion/src/datahub/ingestion/source/looker/lookml_source.py, where we’re currently skipping lookml sql queries with embedded liquid variables
l
That's great - thanks for the pointer @gray-shoe-75895!