hey guys, in flink we dont have a callback when a map, filter or other function gives error. its possible to introduce that kind of feature in flink and pyflink in a future release?

Which kind of error are you mentioning? Wouldn't a try/catch help you there?

its not that. i want a side output to put error events to publish on a DLT topic

Have you checked flatMap instead? IIRC, they do support side/multiple outputs

in pyflink-?

Oh! I was checking the docs and apparently, on pyflink only a process funtion will achieve this behavior

yes, that's why we were asking if its possible to add that functionallity or a function error as argument to redirect the discarded records to a DLT topic

Yeah. From current implementation no. You'd have to rely on ProcessFunction. If you think more folks could support, you can always open a new flip discussion

where i can open it ?

I'm curious about the statement that process functions are less performant than map. How did you measure this?

the implementation that we had on process functions were giving less rate of process records than the map function. i dont have the prints now ...but we saw that was about 10k