邢云
05/15/2025, 6:03 PM``
`schema:`
`- user_id: string, 5M cardinility`
`- item_id: string`
``
2. Enrich each record via joining Source B, C, D on key item_id. (stream-table join)
3. Explode and then aggregate, the aggregated schema could be:
``
`schema:`
`- user_id`
`- agg_id: unique id, high data skew, large users could have 25B agg_id, small users could have only one`
``
Current architecture
• Medallion architecture, Spark Structured Streaming(AvailableNow trigger) + Delta Lake
``
`Bronze A(parquet) -> Silver A(Delta)`
`Bronze B(parquet, 50B) -> Silver B(Delta)`
`Bronze C(parquet, 20B) -> Silver C(Delta)`
`....`
`Gold A = Stream from A, join Silver B, C, D... in each micro-batch, then merge into Gold A, Result is 600B(yes, explosions.)`
``
• Merge via Delta Lake merge
• Sink Gold A into ADX or other placesing for serving purpose. Gold A is business ready
Questions on architecual design with Flink 2.0
1. For now I'm thinking Flink 2.0: KeyedProcessFunction + Managed State + Event-Time Timers
2. I'm afraid this would be an architectural change:
• Does Medallion architecture still apply? or Can I keep using Delta Lake or switch to Apache Paimon?
• Maybe in a hybrid mode: I keep silvers, they're still handled by spark, then I re-build gold with Flink. I prefer this as I can gradually change my architecture with confidencde
Other thoughts
Here're some my understandings, are they correct?
• I didn't mention ForSt because it's state backend of Flink disaggregated state management, but not Data Format of LakeHouse architecture.
• My Hive catalog shall be re-built as its underlying format is delta lake.
• I cannot use Delta Lake merge feature, I need to do that in Flink instead.
Any advice or best practice is welcomed! Thank youPedro Mázala
05/26/2025, 8:47 AM