Pretty new to Flink but have spent a few weeks hac...
# troubleshooting
t
Pretty new to Flink but have spent a few weeks hacking around with it. I have what feels like a common case, but I'm having trouble figuring out the most reasonable way to address this. I have an IoT device that emits different types of data. There is a topic that emits to denote that it has started a certain task, which signals both that the task is now loaded, and what that particular task is. On another topic I have the state it is in running that task such as paused, running, faulted, complete. I'm basically trying to connect these 2 data sources to accumulate the total time spent in each state for each task execution. So basically I end up with data such as "Task W that started at X ran for Y amount of time running and Z amount of time faulted"
d
have you tried to sync topics using watermark alignment? It can be configured in DataStream API
t
I'm not aware of this. Does this alignment apply to connected streams?
d
you define group across sources
so you can define in couple input topics
t
In [the docs](https://nightlies.apache.org/flink/flink-docs-master/docs/learn-flink/etl/#example) it seems to suggest that I can't really control the order that a CoFlatMap is executed
It is important to recognize that you have no control over the order in which the
flatMap1
and
flatMap2
callbacks are called. These two input streams are racing against each other, and the Flink runtime will do what it wants to regarding consuming events from one stream or the other. In cases where timing and/or ordering matter, you may find it necessary to buffer events in managed Flink state until your application is ready to process them. (Note: if you are truly desperate, it is possible to exert some limited control over the order in which a two-input operator consumes its inputs by using a custom Operator that implements the InputSelectable
d
you can of course there is some performance penalty
t
I'd rather not buffer things, as it seems wasteful
d
but I haven't analysed details how it exactly works. I'm going to od this next week
having sync sources you can try to do interval join
I'm doing it
initially it works in my tests
t
That seems relevant, but I've just not found anything suggesting how this might impact the ordering of 2 connected streams
d
but need to perform more detailed tests
In my case I have topic with domainEvent and technicalEvent containing metadata
and I have to join them using some requestId
they are sync using event_time
t
I've been using
BoundedOutOfOrdernessWatermarks
to try to keep things in order, but it doesn't seem to keep the events from 2 different streams in order with eachother
even using my own event time assigner it doesn't seem to make a difference
d
watermark alignnment across streams is needed
t
There is some preprocessing to the data with a
map
, I assume the data produced by the map inherits the watermark?
d
Copy code
When enabling the alignment, you need to tell Flink, which group should the source belong. You do that by providing a label (e.g. alignment-group-1) which bind together all sources that share it.
I haven't check implementation but I guess that it pause a source when it produces the data too fast
a ye here it is how it works
Copy code
In order to achieve the alignment Flink will pause consuming from the source/task, which generated watermark that is too far into the future. In the meantime it will continue reading records from other sources/tasks which can move the combined watermark forward and that way unblock the faster one.
t
yeah this is exactly what I want, buffering seems like a needlessly expensive way to do this given I don't really need that buffered data until I've received the data from the other topic
I'll give this a whirl and see if I can get it to work out
👍 1