This message was deleted.
# troubleshootings
Slackbot
03/30/2023, 5:55 PMThis message was deleted.
v
Vijay Narayanan
03/31/2023, 1:03 AMcan you paste the overlord dynamic config here?
s
Sanjay
03/31/2023, 4:11 AM Copy code
{
"selectStrategy": {
"type": "fillCapacity"
},
"autoScaler": {
"envConfig": {
"numInstances": 1,
"projectId": "gcp-druid-dev",
"zoneName": "us-east4-c",
"managedInstanceGroupName": "druid-middlemanagers"
},
"maxNumWorkers": 2,
"minNumWorkers": 1,
"type": "gce"
}
}Sanjay
03/31/2023, 8:15 AM Copy code
druid.indexer.autoscale.strategy="gce"
druid.indexer.autoscale.doAutoscale=true
druid.indexer.autoscale.workerVersion=1
druid.indexer.autoscale.originTime=2023-03-30T15:00:00Z
druid.indexer.autoscale.workerIdleTimeout=PT30MSanjay
03/31/2023, 8:15 AMoverlord runtime.
t
Thomas
03/31/2023, 8:20 AMIs your MIG a zonal or regional MIG?
s
Sanjay
03/31/2023, 8:20 AMzonal. scope is limited to zonal.
t
Thomas
03/31/2023, 8:21 AMI had an issue where the extension was not able to delete instance because it was running in a regional MIG
s
Sanjay
03/31/2023, 8:22 AMit is trying to delete the instance based on hostname, instead of name.
Sanjay
03/31/2023, 8:22 AMwhich causing that 400.
Sanjay
03/31/2023, 8:25 AMAlso this extension is keep on deleting/creating instances every few secs. I have configured the min instance to 1.
v
Vijay Narayanan
04/03/2023, 12:07 PMwhat are druid.indexer.autoscale.provisionPeriod and druid.indexer.autoscale.terminatePeriod? Depending on pending tasks and these configs middle managers will be created and terminated
s
Sanjay
04/03/2023, 12:10 PMThese are default which is PM1M and PT5M respectively. however cluster don’t have any tasks, even an idle cluster, auto scalar will continuously adding and removing nodes.
v
Vijay Narayanan
04/03/2023, 12:11 PMDo you see any error in the middle manager logs in those nodes? I am wondering whether the issue is with middle managers not coming up properly
s
Sanjay
04/03/2023, 12:13 PMI don’t see anything unusual, middlemanagers are getting registered and shows up in console also when task is submitted, it is able to run it. however, I do see a problem after sometime, tasks goes into pending state which only runs when new node is added back to cluster by auto scalar.