𝐄𝐧𝐝-𝐭𝐨-𝐄𝐧𝐝 𝐃𝐚𝐭𝐚 𝐋𝐢𝐧𝐞𝐚𝐠𝐞 𝐰𝐢𝐭𝐡 𝐊𝐚𝐟𝐤𝐚, 𝐅𝐥𝐢𝐧𝐤, 𝐒𝐩𝐚𝐫𝐤, 𝐚𝐧𝐝 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐮𝐬𝐢𝐧𝐠 𝐎𝐩𝐞𝐧𝐋𝐢𝐧𝐞𝐚𝐠𝐞 I've created a complete, hands-on tutorial that shows how to capture and visualize data lineage from the source all the way through to downstream analytics. The project follows data from a single Apache Kafka topic as it branches into multiple parallel pipelines, with the entire journey visualized in 𝐌𝐚𝐫𝐪𝐮𝐞𝐳. The guide walks through a modern, production-style stack: ✅ 𝐀𝐩𝐚𝐜𝐡𝐞 𝐊𝐚𝐟𝐤𝐚 - Using Kafka Connect with a custom 𝐎𝐩𝐞𝐧𝐋𝐢𝐧𝐞𝐚𝐠𝐞 SMT for both source and S3 sink connectors. ✅ 𝐀𝐩𝐚𝐜𝐡𝐞 𝐅𝐥𝐢𝐧𝐤 - Demonstrating 𝐎𝐩𝐞𝐧𝐋𝐢𝐧𝐞𝐚𝐠𝐞 integration for both real-time analytics (DataStream API) and data integration (Table API) jobs. ✅ 𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 - Ingesting streaming data from Flink into a modern lakehouse table. ✅ 𝐀𝐩𝐚𝐜𝐡𝐞 𝐒𝐩𝐚𝐫𝐤 - Running a batch aggregation job that consumes from the Iceberg table, completing the lineage graph. This project demonstrates how to build a holistic view of your pipelines, helping answer questions like: • Which applications are consuming this topic? • What's the downstream impact if the topic schema changes? The whole setup is fully containerized and easy to run. Want to see it in action? The full source code and a detailed walkthrough are available on GitHub. • 𝐒𝐞𝐭𝐮𝐩 𝐭𝐡𝐞 𝐝𝐞𝐦𝐨 𝐞𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭: https://github.com/factorhouse/factorhouse-local • 𝐅𝐨𝐫 𝐭𝐡𝐞 𝐟𝐮𝐥𝐥 𝐠𝐮𝐢𝐝𝐞 𝐚𝐧𝐝 𝐬𝐨𝐮𝐫𝐜𝐞 𝐜𝐨𝐝𝐞: https://github.com/factorhouse/examples/blob/main/projects/data-lineage-labs/lab2_end-to-end.md