Hi team π β proposing engine-native column-level lineage via the audit log
I've opened an RFC + draft PR to emit column-level lineage directly from the FE analyzer into AuditEvent, so downstream tools (DataHub / Atlas / OpenLineage) don't need to re-parse SQL and replicate our catalog logic offline.
The gist:
β’ Register source columns during the analyzer pass (new SourceColumn on AnalyzeState)
β’ Propagate through existing relation visitors β tables, views, MVs, CTEs, joins, unions
β’ Unified OutputColumn abstraction across INSERT / CTAS / view creation
β’ Gated behind a session variable, with size caps; estimated <2% analyze-time overhead
β’ Wire format aligned with OpenLineage facets
Open questions I'd love input on:
1. Wire format β stick with OpenLineage facets, or something lighter?
2. View expansion depth β fully expand, or stop at the view boundary?
3. Anything else you'd want covered before this lands?
Issue / design:
https://github.com/StarRocks/starrocks/issues/71920
Any feedback β on the approach, the scope, or the corner cases β would be super helpful. Happy to iterate. π