This message was deleted.

This data source is being pushed into “cold” storage, while tranquility is being pushed into “hot”, could this impact the numbers?

Could you please share how large the records are? You could increase recordBufferSize and maxRecordsPerPoll as well if the records aren't too large

How would I get the record size? I am seeing for segments: 269.44 MB and 2,894,139 rows but this seems a little low at less then a KB per record

It might be worth trying an increase in recordBufferSize and maxRecordsPerPoll. Also, if the records are this small, could you also please set maxRowsInMemory back to the default value of 1,000,000? 75k seems low given that the records are small and could be leading to unnecessary persists to disk

Thanks I will try that 👍

https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#kinesissupervisortuningconfig The default value is 1/6th of the JVM memory. I don't think maxBytesInMemory is the problem

whenever performance is a question, i like to start with profiling. a flame graph of one of your kinesis tasks would illuminate where it is spending its time: https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs

Thanks @Gian Merlino