Hey guys , I wanted to discuss our current system ...
# troubleshooting
n
Hey guys , I wanted to discuss our current system setup and see if Apache Flink would be a good fit to replace one of our consumer groups. Right now, we have two Kafka consumer groups: the raw-consumer group, which is scalable and handles the input from vehicles, and the parsed-consumer group, which handles the post-processing tasks like decoding and filtering. The issue we’re facing is that while the raw-consumer group can scale effectively with the number of vehicles, the parsed-consumer group starts to slow down when we increase the load, especially as the tasks it performs are quite CPU-intensive. Currently, we’re running the system with 20 vehicles, and here’s the resource breakdown: We have 20 pods, each allocated 150 millicores (0.15 vCPU) and 400 MB of memory. That totals 3 vCPUs and 8 GB of memory. However, when scaling to 10,000 vehicles, we’re expecting a significant increase in resource usage. Specifically, we’ll need 250 pods, which translates to 37.5 vCPUs and 100 GB of memory. Given this, we’re wondering if replacing the parsed-consumer group with Apache Flink would be a good solution. Flink is known for handling heavy processing tasks like decoding and filtering efficiently, and we believe it could scale better to meet the needs of our growing vehicle fleet. Would Flink be able to handle this load more effectively than our current parsed-consumer group? And if we were to make this transition, would it provide the scalability we need without running into performance issues or excessive resource consumption? I dont getting any similar usecase done by anycompany so any help is appreciated.
s
Flink should do what you are looking for, but it can be complicated to set up and manage. I don't want to do a product pitch, but DeltaStream makes this pretty simple using SQL. It also supports materialized views so you can do even more with it. But the idea of filtering and enriching multiple kafka streams to land it somewhere else is exactly what it does, the Flink implementation is all abstracted from you so you don't have to deal with it at all.
p
Based on this, some questions pop up to my head. • Which language is being used on this kafka consumer? • How many partitions does your source topic(s) has? • Is the CPU more intensive on more vehicles because which kind of operation? • Do you have any sort of state on the "parsed-consumer"? • Did you think about more robust pods (more resources) reading more partitions/threads? Flink can definitely handle intensive load applications and we'd be happy to help you achieve this. But we need to understand how and why the resources are changing like this?
n
I am using the Python language. Partitions are 20 for topic ( raw-decompressed-can) Yeah as more vehicles send more signals which we need to parse . What do you mean by sort of state ? I just want to make our resource utilise the minimum so that it will reduce the cost.
p
If you got 20 partitions per topic, you will not be able to achieve
250
pods. Because each one will subscribe to a partition and the others will probably be stale.
What do you mean by sort of state ?
Do the
parsed-consumer
hold state to make some processing or each record of Kafka is enough?
I just want to make our resource utilise the minimum so that it will reduce the cost.
Depending on how we sort this, we could achieve a lower cost with Flink for sure. It will depend on how your infra is set and so on. If you want to, I can spare some 15 minutes for us to speak a bit about the problem and try to guide you through some possible solutions.
n
See I lack some devops aspect things how much they allocated to it We are running around 20 pods for 20 vehicles and each pod is given around 150 millicore , 400 Mb memory We thought this system can scale up to 1000 vehicles with this configuration as mentioned by tech lead . But when we go to 10,000 vehicles we are saying let's say it takes around 250 pods in place that we can run 80 flink pods or less ?
p
It's hard to it could be ran with 80 flink pods or less from outside of what this consumer actually does. But for reference, I managed billions of messages daily with a cheaper footprint than other solutions were doing before.
n
So what pointer should I consider , I am in discussion with the team how flink can solve it , then I need to see how I can implement this.
p
Points I would consider: • What is the piece making your consumer spend more resources? (cpu and mem) • How much does your KC costs today? • Do you need state for this scenario? • Will it be able to handle Flink deployments on your infra? I'd start there
n
what piece means which part of function you wnat to say,I dont have any metric to calculate individual stage metric but its first stage i.e decoding one. Cost is around = 10,000 (20 vehicles ) for normal estimation like for 1000 vehicles it will go around (5,00,000) I wnat to scale the system upto 10,00,000 I dont have clue regarding flink deployement of infra but it should be na? What things i need to consider
p
Can we schedule a 15 min call? I think the communication may be better 😄
n
yes