This message was deleted.
# generals
Slackbot
11/01/2023, 1:39 PMThis message was deleted.
p
Piotr Dołowy
11/01/2023, 4:11 PM2 of existing VMs for brokers
2 of existing VMs for overlord+coordinator + router (for HA)
all of additional VMs for historicals.
You didnt mention current middlemanagers
g
Giwrgos Gkοlfopoulos
11/02/2023, 9:10 AMHello
now we have
2 datanode that have middlemanager and historical
1 master that have coordinator-overlord
1 query that have broker and router
Giwrgos Gkοlfopoulos
11/02/2023, 9:11 AMif in new cluster we have:
1 master node with coordinator-overlord
2 query nodes with brokers and routers
6 datanodes with middlemanager and historical?
p
Piotr Dołowy
11/02/2023, 9:13 AMdont mix historicals with middle
Piotr Dołowy
11/02/2023, 9:21 AMEach query needs to process X segments from historicals(and peons if realtime), one segment can be processed only by one cpu.
g
Giwrgos Gkοlfopoulos
11/02/2023, 9:24 AMhmm ok
lets see what will be the best scenario for 10 VMs, each one with 18GB memory, 4 cores and 300GB storage (All changes accepted).
2 VMs for brokers
2 VMs for coordinator-overlord and routers
2 VMs for middlemanager
4 VMs for historical
??
p
Piotr Dołowy
11/02/2023, 9:30 AMcoordinator-overlord and routers are not resource-hungry components, personally i would try sqeeze them maybe with brokers, what would give me 2 more historicals
Piotr Dołowy
11/02/2023, 9:31 AMis 300GB storage is dedicated or shared?
g
Giwrgos Gkοlfopoulos
11/02/2023, 9:32 AMdedicated!
we use a platform for visualization that query a lot the druid data every day.
so we want cache and we want response time to be as fast as can.
Giwrgos Gkοlfopoulos
11/02/2023, 9:33 AMalso we will have 1 more VM for deep storage (HDFS) as we have now in as-is
Giwrgos Gkοlfopoulos
11/03/2023, 9:13 AMIn conclusion, I have decided to proceed as follows and do a first test:
2 VMs for query node that will have brokers and routers 32GB RAM , 8 cores , 50GB storage
1 VM for master node coordinator-overlord 18GB RAM , 4 cores, 50GB storage
2 VMs for datanode only with middlemanager 18GB RAM , 4 cores, 300 GB storage
6 VMs for datanode only with historical 32GB RAM , 8 cores, 300 GB storage
What's your opinion?
p
Piotr Dołowy
11/03/2023, 12:36 PMsounds good.