hi, is there some sort of a rate limiting that can...
# general
k
hi, is there some sort of a rate limiting that can be applied on supervisor for log ingestion via kafka? The purpose would be to consume at a rate so as to not overwhelm the consumer and also control the resources memory/cpu usage.
j
Hi Khajjiar, I only know of rate limiting in the Kinesis supervisor, not Kafka. As far as I know Kafka and the Druid Kafka Supervisor have been engineering for high throughput, not the opposite. There is a
pollTimeout
parameter, don't know if you increase that if it will fetch less frequently. Otherwise my only thought is to keep the taskCount small ... in theory that should create lag, which may be what you want .
k
thanks John, will look into the parameter and taskCount ideas.
k
It seems more like a kafka problem instead of a druid problem. TBH consumer rarely overwhelm kafka.
k
hi Karan, i am concerned about a lot of data being pushed into Kafka. we have multiple topics with a high log rate. now for a new feature we have added another topic to karka with a high log rate. now these logs are being consumed by Druid supervisor. with the additional consumption load, to be on a safer side so as not to consume too many resources on druid. Following what John suggested. say i have taskCount as 1, i need figure out the resources that can probably be consumed - cpu/memory
k
@khajjiar trip Druid never pushes data from kafka but reads from it. Druid resources consumed would be • Ingestions tasks ( controlled via task count) • Segment Storage Storage ( Controlled via load rules) • Minor increase in number of segments in the matadata db. (Controlled via task count, task duration and look back period and kill task threshold) So these are the tools we have to control resources on the druid side.
I would start from having a low task count plus a aggressive load rule which removes data older than 5 days.
Also please set compaction from day one.