As I showed there, yes, that is somewhat possible 🙂 We build the “dynamic rules” system for Kcell - a Telco from Kazakhstan, which still runs in production and is a platform for many use cases in the organization.

But it’s not fully dynamic as you probably could think. We had preimplemented tens of rules templates (Very configurable) in Java. Then users could instantiate many of these rules dynamically (via API) on the running Flink system. They run more than a hundred of those rules.

It’s not ideal. The holy grail for us is to build a system which allows defining any rules with power of a language like SQL (with MATCH_RECOGNIZE clause, which runs CEP behind the scenes). That’s why in different contexts we build a platform that provide to run easily Flink SQL jobs (separately or on session cluster). We trade dynamicity and resource efficiency for this power of SQL. This is often good enough for our clients. But what if we could get the best of both worlds? We did an experiment trying to answer a question “would it be possible to have dynamic rules system of Flink SQL queries, running inside of single Flink job?” This blogpost came out of this https://getindata.com/blog/dynamic-sql-processing-with-apache-flink/ Based on that experiment, I would say “yes, it is possible” 🙂 It just needs some work, Flink contributions to make it functional.