hi all, I have one kafka source with events that m...
# troubleshooting
n
hi all, I have one kafka source with events that must be processed in many ways. Now I fork that input stream to 4 streams and then do different processing but it is one big job. I need in near future make new processing and it looks like not good practice. I think to split my mega jobs to many little but each must read same topic and each must do same preprocessing such deserialization and some common logic. What you usually do in such cases? What best practice?
p
Hello there, Nick! This is an interesting case. Indeed smaller jobs are more reliable and easy to maintain. When I had to do this, I created a monorepo with many entrypoint classes. The monorepo would generate a single jar (for the sake of simplicity) and when deploying, I'd specify the class to be the entrypoint for the job. All platform bits were shared among them and each one had its own consumergroup to kafka.
n
yeap, I also have one project for all flink jobs and only entrypoint in specific jobs are different