This message was deleted.
# generals
Slackbot
12/05/2022, 6:49 AMThis message was deleted.
v
Vijay Narayanan
12/05/2022, 6:53 AMwhat is it that you are trying to do? Only one of the task replicas will commit to deep storage. so adding more replicas will double the reads but won’t double the ingestion throughput.
v
Vadim
12/05/2022, 7:01 AMadding task replicas would increase Kinesis reads proportionally to the number of replicas you add
Vadim
12/05/2022, 7:01 AMit only doubles if you double the replicas
j
James Von Kaenel
12/05/2022, 7:02 AMGot it. Thanks 👍
James Von Kaenel
12/05/2022, 7:04 AM@Vijay Narayanan I have 17 tasks for native kinesis ingestion, but am thinking of adding a replica for each task.
v
Vijay Narayanan
12/05/2022, 7:04 AMwhat is the reason for adding the replica?
Vijay Narayanan
12/05/2022, 7:09 AMif you are looking to boost ingestion throughput then the approach would be to increase number of tasks (and reshard kinesis if you already have 17 shards). Adding replicas will add fault tolerance but will not increase ingestion throughput
👍 1
r
Renato Santos
12/05/2022, 6:18 PMfault tolerance + query throughput for the live segments
t
Tijo Thomas
12/06/2022, 5:37 AMFor query throughput increasing the processing.numThread also helps . Default is 2 , you could try increasing to 4 .
Tijo Thomas
12/06/2022, 5:37 AMMay I know what is the query pattern and concurrency.
j
James Von Kaenel
12/12/2022, 7:12 AMThank you all for your help. Follow question here if anyone knows.. would tranquility replication also double the reads of the kinesis stream?
I know it would double the number of druid tasks, but not sure if that means tranquility would read the kinesis stream twice or the druid tasks are getting data from tranquility that has only read the kinesis stream once.
r
Renato Santos
12/12/2022, 12:14 PMAFAIK, it will double the reads, at last on kafka that's the way it's
Renato Santos
12/12/2022, 12:15 PMbut that 'should' not be a problem, unless kinesis is priced based on that (that could be the case, given aws way of doing biz)
j
James Von Kaenel
12/12/2022, 6:31 PMthanks Renato
James Von Kaenel
12/12/2022, 6:36 PMhowever, here Rohit is saying that the read is only done once for tranquility:
Copy code
User decide the replication factor while using tranquility via a configuration.Tranquility will read the data once and will send the same data to multiple indexing tasks(equal to number of replication) hence ensuring that each replica gets exactly the same set of <http://data.In|data.In> case of realtime nodes ensuring all replicas get same set of data was not possible.r
Renato Santos
12/12/2022, 7:40 PMhm, I don't know for sure if this feature was merged back into apache druid
Renato Santos
12/12/2022, 7:42 PMfor what I understand, tranquility acts as man(manager)-in-the-middle of the ingestion, so it manager it's own tasks and how they should publish,
but I don't think this feature exists in the apache druid or imply