Hello Flink Team, I want to migrate an existing st...
# troubleshooting
u
Hello Flink Team, I want to migrate an existing streaming application(based on Medallion architecture) to Flink 2.0, may I get advice/best practice from architectural level? Challengage(Data Flow): 1. Stream from Source A (parquet files on cloud storage, 80B)
Copy code
``
    `schema:`
      `- user_id: string, 5M cardinility`
      `- item_id: string`
``
2. Enrich each record via joining Source B, C, D on key item_id. (stream-table join) 3. Explode and then aggregate, the aggregated schema could be:
Copy code
``
    `schema:`
      `- user_id`
      `- agg_id: unique id, high data skew, large users could have 25B agg_id, small users could have only one`
``
Current architecture • Medallion architecture, Spark Structured Streaming(AvailableNow trigger) + Delta Lake
Copy code
``
 `Bronze A(parquet) -> Silver A(Delta)`
 `Bronze B(parquet, 50B) -> Silver B(Delta)`
 `Bronze C(parquet, 20B) -> Silver C(Delta)`
 `....`
 `Gold A = Stream from A, join Silver B, C, D... in each micro-batch, then merge into Gold A, Result is 600B(yes, explosions.)`
``
• Merge via Delta Lake merge • Sink Gold A into ADX or other placesing for serving purpose. Gold A is business ready Questions on architecual design with Flink 2.0 1. For now I'm thinking Flink 2.0: KeyedProcessFunction + Managed State + Event-Time Timers 2. I'm afraid this would be an architectural change: • Does Medallion architecture still apply? or Can I keep using Delta Lake or switch to Apache Paimon? • Maybe in a hybrid mode: I keep silvers, they're still handled by spark, then I re-build gold with Flink. I prefer this as I can gradually change my architecture with confidencde Other thoughts Here're some my understandings, are they correct? • I didn't mention ForSt because it's state backend of Flink disaggregated state management, but not Data Format of LakeHouse architecture. • My Hive catalog shall be re-built as its underlying format is delta lake. • I cannot use Delta Lake merge feature, I need to do that in Flink instead. Any advice or best practice is welcomed! Thank you
p
Hello there @邢云 TLDR it looks nice. One thing that would help me understanding the approach and reasons would be: why is this change taking place? Which kind of problems are you expecting Flink to solve? 1. I think this is a good approach. How are you planning on joining the streams? It's important to consider data skew and temporal joins if you're to use SQL/tableAPI side of things. 2. On the migration is it possible to start with some specific pipelines first? Create some sort of shadow processing to gain confidence and increase the team knowledge without going for the full turn. 3. The Medallion architecture here would still apply. Medallion is a conceptual pattern, not tied to a specific tool. You can still use Delta here, but Paimon is growing strong and it has a lot of support dedicated to Flink. Perhaps the change is good. It will depend on how much changes you want to create. 4. This would be a good approach. Also, just coming back to point 2, you could try to run the new pipeline as a shadow and compare results to make sure migration is moving on the right direction. I like the approach of starting on the leaves (gold) because otherwise you'd have to handle with a lot downstream. Points to consider: • Data skew will be high due to your aggregation. I'm not sure if there is a way to play with heavy hitter detection in Flink, it would be worth exploring. • Shadow pipelines with testing • temporal joins