Hi Team, I am configuring a Flink Job that can handle almost 1 million data per second, I have started with below configuration CPU: 4 cores Memory: 2GB Task Slots: 4 with only 30k logs per second But my job still goes too much busy and have a much backpressure, As far as I read that Flink can handle very large amount of data But here is some contradict, I might miss out some of the configuration, So can anybody help me to figure out it would be highly appreciate Thank you in advance 🙏

You can try increasing parallelism for operator which seems to be more busy, during heavy load.

@Nishant Goenka i have tried that but no success. Do you have any idea like what other thing i need to increase if i increase parallelism I have tried by increasing memory of Flink but again no success

which kind of sink are you using?

Kafka

i have earlier faced the similar kind issue with kafka sink(may not be same). that time turns out that job was having issue and was waiting for late events. in jour job can you add

senv.getExecutionPlan()

and collect the json response and check if there is anything you can find. Also, very different thought 1. Can you check the Bandwidth of the instance with kafka cluster

okay, other thing as I see flamegraph It looks like major issue is of serialization Like I want to perform some kind of enrichment and filter like thing on my incoming stream, So @Samrat Deb @Felipe Quirce Do you suggest any best serializer, I have tried for Kryo but it is taking too much time and CPU

Please avoid using kryo fallback. Are u using avro or protobuf formats?

@Sumit Nekar I have tried to avoid Kryo by disabling a generic type, But in my application I have implmented one genetic method and I am unable to register it to disable Kryo

Then its fine. Please make sure if ur kafka cluster can handle ur qps without any cpu throttling

I have tried to avoid Kryo by disabling a generic type, But in my application I have implmented one genetic method and I am unable to register it to disable Kryo

Can you share what difficulties you are facing while disabling Kyro

env.getConfig().disableGenericTypes();

reference :- https://flink.apache.org/2020/04/15/flink-serialization-tuning-vol.-1-choosing-your-serializer-if-you-can/

@Samrat Deb Look on this error, I have one generic function Command it causing a problem

@Keyur Makwana I'm also facing some performance issue and can't find the right setting so far. Any luck on your side since you posted your message? 👀

@Tiphanie Dousset I have tried to optimize my job by changing serialisation and other part is I am looking for is Table API, It also helped me to give some what high performance