Hi team :wave: β€” proposing engine-native column-le...
# contributing-to-starrocks
y
Hi team πŸ‘‹ β€” proposing engine-native column-level lineage via the audit log I've opened an RFC + draft PR to emit column-level lineage directly from the FE analyzer into AuditEvent, so downstream tools (DataHub / Atlas / OpenLineage) don't need to re-parse SQL and replicate our catalog logic offline. The gist: β€’ Register source columns during the analyzer pass (new SourceColumn on AnalyzeState) β€’ Propagate through existing relation visitors β€” tables, views, MVs, CTEs, joins, unions β€’ Unified OutputColumn abstraction across INSERT / CTAS / view creation β€’ Gated behind a session variable, with size caps; estimated <2% analyze-time overhead β€’ Wire format aligned with OpenLineage facets Open questions I'd love input on: 1. Wire format β€” stick with OpenLineage facets, or something lighter? 2. View expansion depth β€” fully expand, or stop at the view boundary? 3. Anything else you'd want covered before this lands? Issue / design: https://github.com/StarRocks/starrocks/issues/71920 Any feedback β€” on the approach, the scope, or the corner cases β€” would be super helpful. Happy to iterate. πŸ™
thumbup rocky 1
h
@Simo Wang please take a look