A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi guys, not sure if this is the best place to ask this question(Please correct me if wrong):

Is there any plan/existing feature to support _*Flink lineage*_? Potentially we can build one similar to <https://github.com/linkedin/datahub/tree/master/metadata-integration/java/spark-lineage|spark-lineage> by ourself, but to prevent re-invent the wheel, just want to double check if someone's working on it already.

At least <https://feature-requests.datahubproject.io/roadmap|Q1 2022 Road Map> doesn't has related feature IIUC.

Well, looks like flink execution plan doesn't contains necessary metadata to identify details of up/down stream. Thus, spark-lineage way might not working.

Hi <@U02TN2N66UA>! We don’t have Flink on our immediate roadmap, and I can’t think of anyone in the community off the top of my head that might be working on it.. That said, I’d be more than happy to help facilitate conversations/guidance with the Core DataHub team! It’s always suuuper helpful when community members can contribute back - you can take a look at <#C02SRNN11EG|integration-iceberg-datahub> as an example of a community-lead integration with support from the Core Team

In the meantime, I created a feature request for Flink so we can get a better sense of community interest :slightly_smiling_face: <https://feature-requests.datahubproject.io/b/feedback/p/flink-integration>

Gotcha! Thanks a lot for the response, Maggie!