Hey team! Currently, I am looking into optimizing ...
# troubleshooting
t
Hey team! Currently, I am looking into optimizing for low latency a flink job. The job is very simple, reads from a topic with 4 partitions, does some mapping and sinks to another topic with 4 partitions. Nothing fancy... currently, I tweaked these configs on the Consumer side:
Copy code
max.poll.records=100
fetch.min.bytes=1
fetch.max.wait.ms=5
fetch.max.bytes=8388608
max.partition.fetch.bytes=2097152
and on the producer side:
Copy code
batch.size=16384
max.request.size=2097152
buffer.memory=33554432
linger.ms=0
Job parallelism is 4 and I already have 1 consumer per partition. So far, this worked it I managed to drop the latency, but looking at the consumer lag, there are still records in the topic... My goal would be to have as few as possible 😅 Any advice is very much welcomed! Thanks
p
I've got some questions before. Which deserializer are you using? POJO or Kryo? This could be something. Also a pipeline viz screenshot would be great to understand if there is some back pressure. Does your parallelism applies to everything? I mean, do you have a different set of parallelism per operator? Because you may use
disableChaining
if the backpressure is happening not in the reading. Something else I'd also check is the amount of operators you have in your pipeline. Each operator will have to serialize and deserialize the record to forward it.
t
Which deserializer are you using? POJO or Kryo?
Pojo
Does your parallelism applies to everything?
Yes, I don't use
disableChaining
p
I'd use it to be sure your pipeline can go higher than 4 parallelism. Where you have 4 for source only and distribute the rest. If you have a single blend in the operator it's impossible to check for backpressure
t
This is the job, so you can see that everything is chained together.
I'm just thinking, if I disable chaining, I'll be able to probably consume more data, but the job will spent time shuffling the data and it could add up to the latency, no?
p
The consumption throughput should be enough to decrease the latency. I think it will not be an issue at all. After some discussion we realized the pipeline was relying on Kryo and this could be a bottleneck[1]. What is your checkpoint frequency, @Tudor Plugaru? A high checkpoint may linger the offsets to be committed but this does not necessarily mean the lag is there. It only means the offset was not committed yet.
t
Checkpointing is every 30s, I'll reduce it to every 10 sec.
My main concern is that looking at
sourceIdleTime
metric, AVG is ~200ms 😕 so it's like it to have this one lower, I know for sure that there are records in the topic
p
Have you tried having fewer consumers? Just to be sure sure about idleness
t
So, just as an update, the things I did: 1. Changed from Kryo to Pojo serialization 2. Tweaked consumer and producers configs as in the initial message 3. Tried both with chaining enabled and without 4. I did enabled the end to end latency tracking that just showed that a record spends around ~300ms from the source to the sink. That's how I understood the usage of the metric. 5. More CPU and Memory 😅 💸 Currently, I'm out of ideas, only to add more than 4 partitions, but I don't think it'll solve it. I'm open to new ideas and suggestions on how to tweak Flink for sub 100ms 🤞 latency from source to sink.
p
Do the e2e show where more time is spent? Serialization? Any external resources? This is quite high time indeed. I'm wondering if the service consumes anything outside. Schema registry or so
t
Do the e2e show where more time is spent?
It's basically a metric per sink -> operator latency. On our end, we have 4.
And no, just data mapping and json serde, that's it.