This message was deleted.
# troubleshootings
Slackbot
12/08/2022, 11:17 PMThis message was deleted.
b
Brandon Tan
12/09/2022, 2:10 AMYes, taskCount = # of shards per Kafka / Kinesis as a best practice
j
Jonathan Du
12/09/2022, 2:50 AMThank you, Brandon
š 1
g
Gian Merlino
12/09/2022, 7:15 AMhmm. even with 10 shards -> 1 task, a GB heap is intended to be enough. it would just go slower than 10 tasks; the idea is you should be scaling up your tasks for perf reasons not memory reasons
Gian Merlino
12/09/2022, 7:15 AMdo you know what is in the 10 GB (do you have any heap dumps, for example)?
Gian Merlino
12/09/2022, 7:15 AMcould indicate something we need to improve
j
Jonathan Du
12/09/2022, 7:19 AMI donāt have any heap dumps, though I suspect these would be very useful. Iāll look into this into the future. Though just recently, I discovered that reducing the recordsPerFetch from default 4000 -> 10 and recordBufferSize from 10000 -> 100 allowed for a 1GB heap
Jonathan Du
12/09/2022, 7:20 AMWe use KPL aggregation on our records and Iām lead to believe the internal buffer for records between the fetch threads and the indexer thread is causing the OOMs
g
Gian Merlino
12/09/2022, 7:22 AMah, maybe that's it! those, i bet, are per-shard
Gian Merlino
12/09/2022, 7:23 AMoh, and you're also saying something else, that your messages are huge because of aggregation
j
Jonathan Du
12/09/2022, 7:23 AMBy āthoseā you mean the internal record buffers?
Jonathan Du
12/09/2022, 7:23 AMMessages are max 1MB; they still abide to Kinesis shard limits
Jonathan Du
12/09/2022, 7:24 AMThough the average size of our messages are 500KB
g
Gian Merlino
12/09/2022, 7:24 AMare you using the
deaggregatej
Jonathan Du
12/09/2022, 7:24 AMyep
g
Gian Merlino
12/09/2022, 7:27 AMnvm, those settings are not per-shard. just checked. so i think you're right that it's deaggregation related
j
Jonathan Du
12/09/2022, 7:30 AMOH I see what you mean
Jonathan Du
12/09/2022, 7:32 AMYeah, my earlier read through the source-code revealed the recordBufferSize counts an aggregated record as 1 count, rather than counting the individual exploded records. Which makes it consistent with recordsPerFetch I suppose, but it was counter to my initial thoughts
g
Gian Merlino
12/09/2022, 9:44 AMraised a patch to set the defaults better, and clarify the docs: https://github.com/apache/druid/pull/13539
j
Jonathan Du
12/09/2022, 11:29 AMYou legend! Thanks, Gian