Hi all, I'm evaluating datahub for a POC and had a...
# getting-started
f
Hi all, I'm evaluating datahub for a POC and had a few questions about how different metadata revisions are reflected in the UI. All of the questions assume we're directly hitting gms and synchronously emitting metadata to datahub (not pulling/crawling any services): • At t=0, I push the following lineage metadata: A --> B • At t=1, I push A' --> B (with A no longer existing/deleted from the system) B's authoritative source (from the POV of the code, not necessarily the data warehouse) at t=1 is now A', but the edges A --> B and A' --> B will both be reflected in the UI, and I only see the option to filter lineage by time (at the granularity of days). Does datahub have a concept of a particular "version" of the world / the ability to filter entities in the UI by that? Trying to think about how to reconcile the "CI-time" view of our data system where we know the full DAG of pipelines, vs the "runtime" view where a dataset may not materialize until some time after deploying pipelines
Think we were able to resolve these concerns w/ some (ab)use of the various APIs & got to a workable setup
💯 2
Also since the lineage DAG represents the latest state pushed (in our case with respect to the output dataset from a pipeline) that isn't an issue for us
r
@kind-caravan-32909 can you assist? Thx
👍 1