Hello friends, I’ve run into an issue with my Flin...
# troubleshooting
m
Hello friends, I’ve run into an issue with my Flink job designs, and I could use a bit of help. Basically, we can’t have an operator with more than two inputs. However, there are cases where we need to join three topics together. The solution that comes to mind is to join the first two topics, then join the result with the third topic. But there’s a problem here. For example, I’m joining the price and quantity topics together, and both of them are stored in their respective states. Now, in the next operator, I join the result with the third topic. In the second join, I need to store both the result of the previous join and the third topic! This means I’m storing something again that I already had in the previous step. Now, imagine this requirement keeps growing and I might need more joins — but in every step, I’m repeatedly storing duplicate data. What’s the proper way to solve this problem? Is there something I don’t know that’s causing me to hit this issue, or is this a normal problem — and if so, is there a better solution for it? For context: my code is written using the DataStream API.
p
Basically, we can’t have an operator with more than two inputs.
Is this a restriction by your team? What inputs are you using?
The solution that comes to mind is to join the first two topics,
then join the result with the third topic.
If all of them come from the same broker, you can comma split the topic names and Flink will read from all of them in a single shot.
m
No, this is not my limitation, it’s a limitation in, for example, KeyedCoProcessFunction. I can’t pass more than two types to it. All my inputs are from Kafka.
p
What I did to solve this was having a parent type named Kafka record. It would store the record in a pojo like generic way. this way I had full control over it. The detail here lives on creating a class that "absorbs" the other ones
Copy code
KafkaSource<OrderEvent> kafkaSource =
        KafkaSource.<OrderEvent>builder()
            .setBootstrapServers(bootstrapServers)
            .setProperties(properties)
            .setTopics("a,b,c,d,e,f,g,h")
            .setGroupId("###")
            .setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST))
            .setValueOnlyDeserializer(new EventDeserializerSchema())
            .build();
My object OrderEvent may be 8 different types here. I can even achieve this without Kryo to ensure compatibility in the future
m
So, is this a logical approach? Because this is the method that came to my mind, but I thought maybe it’s the wrong way. There won’t be any problems with this method, right?
p
No problems. I ran multiple pipelines like this. For me it sounds logical. But OFC there is always the tradeoff, right?! In this approach, you will see more generic objects floating around the pipeline. It increases code complexity in a sense of having to test the input before using it, while in the other hand, you got this solved on deserialization time, but have a more complex way of mixing them up. IMO, everything is about choosing your poison 😅
a
I've also most often leaned towards the union-approach similar to what Pedro describes above - but with concrete types rather than a generic one. Usually I ingest records and map them to types that most precisely describe them, e.g.,
ItemSold
,
ItemPicked
,
ItemShipped
, etc. Let's say I need to perform an operation that involves all three of the above. In this case, I would create a union type with an appropriate name, e.g.,
ItemSoldPickedShippedUnion
(naming is hard, right?!). This union type contains all relevant fields from each of the concrete input records. Then I'd map each record input stream to the union type. For example:
Copy code
var itemSoldUnionStream = itemSoldStream
  .map(ItemSoldPickedShippedUnion::fromItemSold);

var itemPickedUnionStream = itemPickedStream
  .map(ItemSoldPickedShippedUnion::fromItemPicked);

var itemShippedUnionStream = itemShippedStream
  .map(ItemSoldPickedShippedUnion::fromItemShipped);

var itemSoldPickedShippedUnionStream = itemSoldUnionStream
  .union(itemPickedUnionStream)
  .union(itemShippedUnionStream);

// do stuff with itemSoldPickedShippedUnionStream
(I left out calls to
.uuid
,
.keyBy
etc for brevity) I prefer having concrete types, as opposed to generic ones, as it makes the intention of the code more explicit. It's more code and more verbose, but when using Java record types, the boilerplate is kept to a minimum.
💡 1
👍 1
I can only echo what Pedro said: as the API stands today you either have to daisy chain joins or use union types 🤷 I mostly pick the latter