Hey all, We're using Flink to run a 'per consumer' processing job that takes a bunch of Kafka inputs and writes to a single Kafka sink. Using Kafka for 1:1 communication comes with some downsides, so I've been trying to figure out if it would be possible to have a Flink Sink based on a direct "socket" connection to the single consumer of the output. Ideally the flink job would listen on a server-socket so that the consumer can initiate the connection. Have been looking at the built-in connectors, and there does not seem to be any sink making use of such approach. Anyone have any knowledge about direct-to-consumer sinks, and perhaps even sinks that start by listening for an incoming connection? Open source examples would be nice - but mostly just looking to understand if this is a sensible thing to pursue.

I don't see how that would work while maintaining any processing guarantees. What if your Flink job/cluster needs to restart?

Processing is only on live data and consumer process specific, and the processing ('logic') of the job is parameterized by rules being sent from the specific consumer. We anyhow have have to restart-and-re-initialize when these change, or if consumer is restarted. We have our own mechanism for creating the initial state for a given 'cut in' point in the live stream. The consumer processes are all spawned without any prior state, so there is no at-least-one-delivery or exactly-once-processing concerns across restarts / failure. (Of course within a job:consumer pairing we want at-least-once delivery until a failure point) In essence the flink job acts as a side-car with same life-time as the consumer of the output. If either 'dies', both are restarted.

I see, but that kind of beats the point of Flink in terms of guarantees 🙂 I'm not aware of anyone using Flink for such a use case

Thank you for the answers 🙂