Hi team, I am starting to run flink on production ...
# troubleshooting
r
Hi team, I am starting to run flink on production what is the recommended method to deploy flink and with how much memory allocation. Basically I want to run my jar file right now when I'm running it in a EC2 cluster there's delay of more than 2 days in the realtime streaming data, I want my transformed stream to be in realtime. My code flow is in this manner: Reading kafka streams using kafkaSource and then doing processing by extracting key fields from kafka topic and using them to query on my Mysql DB using JDBC connection and fetching all the required fields from necessary tables, merging them and sending the merged topic back to kafka using kafkaSink. Right now when deploying it using java -jar myJarFile.jar I am getting a lot of delay of more than 36 hours. I want to make it more realtime how can I do it? Please help.
p
Hey there, Rajat! How is the back pressure of your pipeline? How many partitions your topics have? On your deployment, do you use any HA capability? I did not fully understand the deployment setup. On your MySQL query, how are you tackling it? Understanding those properly may help me to guide you through it
r
Hey, Back pressure of the source is realtime kafka topic that has 1 partition. So, right now I am running my java jars directly on the server using command:
Copy code
java -jar myJarFile.jar
On Mysql I am sending queries based on batches of 500, my java code will wait till it collects the kafka records, if it reaches 500 OR 2 second Timeout. Then creating the query and sending it to Mysql using JDBC connection
p
So, right now I am running my java jars directly on the server using command:
Copy code
java -jar myJarFile.jar
I'd suggest you check how Flink standalone works. Running Flink like this you lose a ton of it's capabilities.
Back pressure of the source is realtime kafka topic that has 1 partition.
Any reason on the single partition? Are you using Flink AsyncIO features to run the queries?
r
No, what are Flink AsyncIO features can you share any resource for that
We have this 1 partition for our production kafka topic we can't change that
p
I'd say your best shot is trying to parallelize flink tasks based on a keying strategy to keep the desired ordering
r
alright sure, will try and check. Thanks