This message was deleted.

This is a discussion on how Kinesis works and how to achieve best throughput: https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#determining-fetch-settings Another interesting point is whether you are packing many rows into each message and using the deaggregation function at ingestion time. See here: https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#determining-fetch-settings.

Sounds good, thanks @Sergio Ferragut.

No. Unless I am missing something, deaggregate takes multiple rows packed into a single message and processes them as individual rows during ingestion. There is no mechanism to store a packed set of rows at ingestion time as a single row they should always be processed as individual rows at ingestion.

BTW... I haven't used it, but there is a suggestion in the docs that you use one fetchThread per shard, so you might want to tweak that too.

Is the fetchThreads the total threads for all the tasks or the total number of threads per task? I am assuming its the total number of threads for all the tasks?

I made the same assumption, but it’s a good question.