Hi All I have devices with sensors that could buff...
# troubleshooting
n
Hi All I have devices with sensors that could buffer data when network unreachable and send buffered data when online. So those events are not late in term of single sensor stream, but when I have few online sensors Flink update watermark and all buffered data is 'late' for operator and ignored. Setting latteness on window isn't an option in my case because of it break processing window logic. So is the way to say flink to use watermarks per key? Or if no way then how I should use flink in such case to process buffered on device data for 2 weeks with window aggregations?
y
Hi Nick I think you can use Bounded Stream and compute fixed-time windows using event time.
p
Hello there Nick! I have some follow-up questions to understand better your usecase. 1. What’s the typical out‐of‐order bound (max lateness) you expect for each sensor stream? and how long can a device be offline and buffering before coming back online? 2. What window size and type are you using (e.g. tumbling, sliding, session)? 3. How are you keying your stream? One key per device, per sensor type, or something else? How many distinct keys (roughly) and what parallelism do you run at? 4. Which
WatermarkStrategy
(or custom generator) are you using now? 5. If data arrives after the watermark (i.e. “late”), do you need to re‐compute windows or simply side‐output them for offline processing? Would you be open to capturing late elements in a side output and then re‐feeding them through a separate job?
n
1. lateness is not a choice 2. in different jobs I have 10-30-60 sec sliding with 1 sec slide 3. keying by device_id-sensor_id for measurements. Now around 600, 400 of them are sending 1 event per sec. Parallelism doesn't matter it must work with any from 1 to any other 4. Nothing because it didn't work 5. as in 1 lateness is not a choice, I must process data with windows based on event time through key. Refeeding window aggregation break processing logic
I need answer how to make flink use watermarks by keys or how to archive same logic
p
1. What do you mean by lateness is not a choice? The sensors that hold the data and offload later are "late" processing events. Do your events have the timestamp of generation and the timestamp of being written to your source? 2. - 3. "Parallelism doesn't matter it must work with any from 1 to any other" I did not quite fully got this sentence. 4. So what's your source today? 5. So what happens when
device_id-sensor_id
posts some events in a window and then gap for a while? Which behavior should it make? i.e. if a sensor dies, making the time never moving forward, the last window of that sensor will be discarded?
n
1. Lateness in flink meaning not a choice. It has unusable semantic in my case. Event have timestamp when it was created on device. 3. That mean that I must get same result with running on same dataset with 1,2,4,10 or 100 parallelism. So for me parallelism doesn't matter because nothing should depend on it 4. I tried everything combination of WM start and no one help. So that mean that I have no WM strat and I'm searching solution. May be it must be some custom strat, but as I think it is Flink arch problem. 5. Window should be opened for a while until data come or something happen and state would be cleared. Anyway when old data will come it must be processed as actual online data with same window logic without any 'lateness' semantic. And logic must be same even with timestamps about 2 weeks ago are processed near to online data
p
Got it. Indeed, I think a custom KeyedProcessFunction will be your way to go. There you may develop a custom window based on the highest timestamp of a key. Then doing the triggering/windowing part based on that value. I've built something similar to it in the post The case for a Custom Window in Flink: Expanding your streaming use-cases. I do not use custom watermark logic there but I think it would be fairly straight forward to add. If you need help building it, let me know and we can collaborate.
n
Thanks for clearing situation. So as expected I must write my own windowing logic in process function
p
I think so yes. I have some code already in this area. If you want to collaborate, lmk.
n
I found https://github.com/TawfikYasser/Keyed-Watermarks-in-Apache-Flink and will try do something like that
👍 1
@Pedro Mázala do you know why Flink didn't realize keyed watermarks as suggested in repo above and in https://scholar.google.com/scholar?q=10.1016/j.future.2025.107796
p
So, the way features land in Flink is by community support. Community members propose them through FLIPs, they get voted, and when approved, someone from community needs to implement. If you feel this could help other folks perhaps it's worth making the proposal through the process.
n
I read and trying to use but two big problem: 1. It's for 2.0 that I can't use because no needed connectors available now 2. WM here a granulated but not by events key. It add some additional kinds of WM that could be used but it's not WM-per-key
message has been deleted