Hi. All , If my kafka topic has 32 partitions, can we control pinot consuming threads self
m
Mayank
03/26/2021, 12:43 AM
What do you mean by controlling the consuming threads?
c
Charles
03/26/2021, 1:01 AM
@Mayank I means, when kafka topic has 32 partitions, pinot will start 32 threads to consuming this topic, can we control this consuming threads such like i can config to 12?
m
Mayank
03/26/2021, 1:02 AM
Not at the moment. You could play with number of servers though. For example if you add 4 servers then each will read 8 partitions, and if you add 8 then each will read 4 partitions.
Mayank
03/26/2021, 1:02 AM
Curious what's the intention here? Is it because you have less number of cores and also don't want to add more servers?
c
Charles
03/26/2021, 1:04 AM
Year, I think if one topic have 32 partitions but don’t have too much logs, I think we can config less threads to consuming this topic
m
Mayank
03/26/2021, 1:05 AM
Any chance you can reduce number of partitions? At any rate, this does seems like a legit ask. Mind filing an issue (with reasoning above) so we can see if this makes sense to support in future?
c
Charles
03/26/2021, 1:06 AM
Thx @Mayank another question, Can we delete or truncate one day’s data in pinot by commends like mysql does?
m
Mayank
03/26/2021, 1:07 AM
Deletion is supported at segment level, not at record or time-boundary. If you can find segments for the day, threre is api's to delete (check swagger api)