Is there support to dynamically create tasks or jo...
# randomd
Damo
07/27/2023, 4:28 PMIs there support to dynamically create tasks or jobs to process streams and perform calculations, without having to rely on code changes and deploying jars?
n
Nathanael England
07/27/2023, 4:30 PMI haven't seen anything like that for jobs. Those always seem to come with an implicit rebuild/deploy requirement. Flink stateful functions seem to be much more flexible in deployment, but you lose out on some of the benefits of the datastream/table API for traditional jobs.
d
Damo
07/27/2023, 4:40 PMI see. What I am looking to do is to be able to provide a service that performs some aggregations on streams, but allow users to define what these calculations are. The flexibility is dampened should compilation and deployment be required.
m
Martijn Visser
07/27/2023, 7:37 PMIsn’t this exactly what you can achieve with SQL?
n
Nathanael England
07/27/2023, 7:43 PMDoesn't changing out the sql require redeploying a job?
m
Martijn Visser
07/27/2023, 8:52 PMMaybe, but it would probably end up as a semantic discussion: SQL is declarative, same as “defining what calculations are”. While functionally the way you put these changes in production (e.g. stop/start a SQL job or use a Web UI/any other tool to push the calculation definitions to prod) can be different, both imho are still a deployment because you change the configuration/logic
Martijn Visser
07/27/2023, 8:55 PMFrom my experience, the primary reason why people like to avoid pipelines/deployments isn’t a functional requirement, but to bypass IT departments or IT requirements.
Martijn Visser
07/27/2023, 8:57 PMCurious on other opinions though :)
n
Nathanael England
07/27/2023, 9:09 PMFor us, it's just speed of development. If I could leave the overall job unperturbed and more quickly swap out a small portion of logic, that could be conceivably way faster than rebuilding a big complex job.
m
Martijn Visser
07/27/2023, 9:51 PMI think I can more quickly iterate with Table API/SQL then I can with Datastream API in general. Doing it in SQL requires me no compilation, by far the fastest
d
Damo
07/28/2023, 8:00 AMAlso offering a "service" or platform, you may want to offer stakeholders or teams the ability to run tasks or calculatons on top of defined streams, without the platform maintainers needing to be involved or deploy said calculations.
Damo
07/28/2023, 8:03 AM@Martijn Visser interesting, I will look at the SQL route. I am guessing Flink allows me to connect an SQL client to create and manage my jobs?
g
Gaurav Miglani
07/28/2023, 8:45 AMwe did it using our custom dags
Gaurav Miglani
07/28/2023, 8:46 AMwe wrote an engine that take user input of dag like kafka source -> sql -> kafkasink and then create ddl for source and sink and views for sql operator and other custom operator
d
Damo
07/28/2023, 9:02 AMAnd you are able to dynamically kick off tasks with their own defined calculations/operators with this approach?
g
Gaurav Miglani
07/28/2023, 9:32 AMyes, sql, http lookup, custom calculation operators in the dag flow to solve custom usecases
d
Damo
07/28/2023, 10:16 AMHow are you submitting them to Flink or making Flink aware of incoming new use-cases without needing redeploys?
I have seen an article here that describes an approach using rules or configs as a separate data source and using broadcasts:
https://flink.apache.org/2020/03/24/advanced-flink-application-patterns-vol.2-dynamic-updates-of-application-logic/
g
Gaurav Miglani
07/31/2023, 6:18 AMyes, we have 2 primitives, streams and operator, an operator can take n number of streams as input and output m number of streams.
Gaurav Miglani
07/31/2023, 6:19 AMchaining of streams and operator create a flink realtime pipeline