Hey all, Is there a way to get the data lineage information from Flink. As in we need to know what sources (and its properties eg for kafka source ->Kafka topic name, cluster etc.) ,and the sinks (along with their properties) for audit purposes. Tried to analyse Jobgraph(/Streamgraph) but we get limited information from the execution plan(as the name of the source can be modified in the Flink code). Is this lineage a first class entity in Flink as of now? If not is there a way to analyse the flink code and extract it, or maybe use the transformations present in the StreamExecutionEnvironment?

There isn’t yet but there is a FLIP planned on this topic for the next Flink release

Thanks we have a major usecase and would be glad to contribute to it. If not already we can go ahead and create one.

@Martijn Visser Can we link to the relevant FLIP?