hey, has anyone managed to abstract over Flink for 'easier' stateful stream processing for the less technical folks in their org (e.g. data science)?

Yes, we've had some success creating various kinds of

Alarm

abstractions that hide away the more intricate details of watermarks and window process, instead just lumping them into a window size/slide API etc. For example, a count-based alarm is easy to implement and easier for folks not familiar with stream processing to understand and make use of

what do you mean by alarm here? is this for monitoring and alerting specifically or something else? 😄 how did you abstract? with a DSL/config or constructs in the app/language level?

Yes, specifically for monitoring and alerting. We made an Alarm base class and then allowed devs to build their own alarms on top of that base class. The source streams were essentially fixed, and the abstraction layer was just done in Java via the Datastream API

A good approach is to use pyflink and to wrap the flink specific calls into a common library and just have the data science people work on the query logic. Also, use pyflink SQL if possible because everyone knows SQL.

We did abstract a few common things like Keybys and aggregations to make it easy for other people to onboard usecases.