https://linen.dev logo
Join Slack
Powered by
Anyone here has strategy on collecting Druid query...
# general
m
Anyone here has strategy on collecting Druid query lineage? Should we support Openlineage and build something similar to https://trino.io/docs/current/admin/event-listeners-openlineage.html ?
g
i'm just hearing of this for the first time (from you), looks interesting but i'm wondering how it'd work
it says the goal is to "capture every query that creates or modifies tables" which seems very intense if by "modify" that includes inserts from realtime ingest. Seems like it would be a huge amount of modifications to report
or does "modify" mean something else in this context?
could possibly emit something any time a segment is published or dropped
j
or does "modify" mean something else in this context?
I think "every query" means every user-generated action. We can get datasource-lineage through metrics, etc. but user-initiated changes through queries are what this open lineage integration is trying to do.
@Gian Merlino ^
g
ah, i see. so it's sort of a logging standard? for very specific kinds of logs?
possibly we could add some hooks to streaming supervisor start/stop and to batch ingestion tasks that could be used to generate this sort of lineage info
j
ah, i see. so it's sort of a logging standard? for very specific kinds of logs?
Yeah
I was thinking it could hook into existing 
Emitter
logic – although it would need more complex info about the plan itself (what columns on what datasources are touched, etc.)
Kind of like an "open telemetry"-like thing for query lineage in sql-compliant engines
g
that might work, lineage could be a kind of 
Event
. this would allow it to flow to regular emitters (such as our 
KafkaEmitter
), and you could also add an 
OpenLineageEmitter
that emits 
LineageEvent
only to an open lineage server
👍 1
Open in Slack
PreviousNext
Powered by