Hello I am using flink for large data process, But...
# troubleshooting
k
Hello I am using flink for large data process, But when i pass around 50k logs per second, My flunk job goes to 100% busy. I have tried by increasing parallelism and flink memory but it did not work for me. Can anyone help me?
Here is the how my job looks like
m
Hi! I was in such situation with some project, and what helped me was to check memory for task managers and job manager. Increasing memory “in general” may not help, but increasing specifically JVM memory helped in my case. Try to change settings in flink.conf according to docs
j
You can check the flame graph of the operators to see where they are spending their time.
k
@Mikhail Spirin I have tried by triple memory but it is not showing significant changes
@Joe Karayusuf I show that write takes time, But I don’t understand how to optimize it
j
Interesting. Looks like your objects using kryo serialization. Either for state, network transfer or both.
I'd recommend updating your model so that flink can either use the POJO or case class serializer.
Here's the criteria your model needs to meet in order to use the POJO serializer. https://nightlies.apache.org/flink/flink-docs-release-1.10/dev/types_serialization.html#rules-for-pojo-types
If you can share the models that you're using, I'd be happy to take a look.
k
Actually i am converting my log to JSON Object I think that might be troubles for me
✅ 1
@Joe Karayusuf @Mikhail Spirin I need your opinion in this thing, I have input stream log that is of fixed schema(Around 1 million record per second), And I have stream of rule for enrichment to be applied on logs. So how would i efficiently implement this? Like for enrichment I need to add or remove field so that was a motivation for me to use JSON object, But now it is creating performance issue, So what would you suggest?
j
Few random thoughts: 1. I'd recommend removing the
.rebalance
between the co-process and sink function. Rebalances aren't cheap. They require flink to serialize the data, send it over the network, and then deserialize the data. You're rebalancing between the co-process function and the sink. Since the data is already rebalanced going into the co-process function, the second rebalance probably isn't doing much. 2. Once the rebalance is removed, try enabling object reuse.
env.getConfig.enableObjectReuse
. Assuming you're using immutable objects, this should be a nice performance win. It'll ensure that flink doesn't serialize and deserialize the objects between the co-processor and sink when they are in the same task group.
Based on your description it seems your use of broadcast function is great. You broadcast the enrichment rules to all operators, evenly distribute, and apply the rules to the incoming data. So the performance bottlenecks here will be: 1. Parsing the data. 2. Applying the rules. 3. Serializing the data to be sent to the sink.
If json is needed during the parsing and serialization, I'd recommend using jsoniter. We switched from
circe
to
jsoniter-scale
in our flink app and saw a huge performance improvement. I imagine switching from
jackson
or
gson
to
jsoniter
would show similar benefits, but can't confirm that.
As far as applying the rules goes, that'll come down to data structures. I imagine you already have this well structured since you've identified serialization as the bottleneck. Though, double check to make sure that identifying which rules to apply to which objects is as efficient as possible. Chances are, the rules are read a lot more than written. So whatever data structure you use here should favor fast reads over fast writes.
k
Okay
Yes as you told like i use broadcasting for dynamic rule
j
For the sink: 1. Are you using an async sink? Most common sinks should have a library that builds off of
AsyncSinkBase
. If not, take a look at implementing one. 2. Are you batching data to the sink? If not, I'd recommend batching the data to reduce the number of calls downstream.
k
Also i was thinking to try Table Api. It is totally different approach but l am thinking is is help for optimization
j
Out of curiosity, what is your source and sink?
Kafka
,
Kinesis
, something else?
k
Both source and sink is Kafka
✅ 1
@Joe Karayusuf I have tried for Async sink in Kafka but I don’t find any reference for that, Can you please help?
j
I'll look into it a bit later tonight.
k
Okay
@Joe Karayusuf I have tried by implementing Custom Sink, But I am not sure it is Async or not
public class DynamicKafkaSink extends RichSinkFunction<Tuple2<String, String>> { private transient KafkaProducer<String, String> kafkaProducer; @Override public void open(Configuration parameters) throws Exception { super.open(parameters); // Initialize Kafka producer Properties producerProperties = new Properties(); producerProperties.setProperty(“bootstrap.servers”, “localhost:9092"); producerProperties.setProperty(“key.serializer”, “customSerializer”); producerProperties.setProperty(“value.serializer”, “customSerializer”); kafkaProducer = new KafkaProducer<>(producerProperties); } @Override public void invoke(Tuple2<String, String> value, Context context) throws Exception { String topic = determineTopic(value.f0); // Determine the target Kafka topic based on some condition // Create a Kafka record with topic, key, and value ProducerRecord<String, String> record = new ProducerRecord<>(topic, value.f0, value.f1); // Send the record to Kafka kafkaProducer.send(record); } @Override public void close() throws Exception { super.close(); // Close the Kafka producer if (kafkaProducer != null) { kafkaProducer.close(); } } private String determineTopic(String key) { // Implement your logic to determine the Kafka topic based on the key // For example, you can use a lookup table or some conditional logic // to map the key to a topic name. return “dynamic_topic”; } }
@Joe Karayusuf Did you get chance to look for?
j
Hi friend! Not yet, sorry! Been sick the past few days. Will take a look soon!
k
Ohh, I hope you are fine now. Thank you so much for the reply friend