https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, I am creating an autoscaler for minion-ba...
# troubleshooting
r
Hi team, I am creating an autoscaler for minion-based batch ingestions. To scale up and down, I need the number of tasks that are waiting and the number of tasks that are running. I checked the Pinot metrics and found these two: 
pinot_controller_numMinionSubtasksWaiting_Value
and 
pinot_controller_numMinionSubtasksRunning_Value
. However, for each task type, they always show a value of 0 even when tasks are running. Am I using the wrong metrics? Which metrics should I use to build a custom autoscaler for minions?
@Mayank @Xiang Fu can you please help
i need waiting and running count for taskType 
SegmentGenerationAndPushTask
x
is there a metrics for the helix task queue size and running tasks count?
r
pinot_controller_numMinionSubtasksWaiting_Value
and 
pinot_controller_numMinionSubtasksRunning_Value
these are the metrics but its value is always 0 for 
SegmentGenerationAndPushTask
Also, I’m not sure what the “Tasks in Queue” value shown in the UI represents, as I don’t see any tasks actually waiting — all tasks seem to be completed.
x
Let me check the api to fetch the number
r
yes in Ui its coming from api only but it doesn’t seem to be accurate — it always shows some tasks in the queue even when there are none, as you can see in the shared screenshot.
is there any reliable prometheus metric to scale up and down?
++ @Shounak Kulkarni
s
Also, I’m not sure what the “Tasks in Queue” value shown in the UI represents, as I don’t see any tasks actually waiting — all tasks seem to be completed.
That's just the task entries which are still there in zk. It may be completed, ongoing or yet to be picked by minions
r
okay. Can you tell me what metrics i can use to scale up and down minion nodes
want to create a HPA for minions
s
the metrics are exposed by TaskMetricsEmitter controller periodic task which has a default frequency to run every 5 mins. You can reduce it using 
controller.minion.task.metrics.emitter.frequencyPeriod
if you want early feedback on the incomplete tasks
r
Shounak, the problem here isn’t the metric collection frequency — the issue is that Pinot isn’t exposing the correct metric. As you can see in the attached screenshot, Prometheus shows 0 running tasks and 0 waiting tasks for that hour, even though there were 4 ingestion tasks running in Pinot at that time.
image.png
s
can you check values for 
pinot_minion_numberOfTasks_Value
at the same time?
r
Its also same
s
then there are no active tasks on minion. Can you shift to the graph representation?
r
image.png
i do get the count for type upsertCompaction but SegmentGenerationAndPushTask is missing
there is always 0 for SegmentGenerationAndPushTask
s
looks like the SegmentGenerationAndPushTask tasks are being completed within seconds so maybe the promethues scraping is not able to capture the active execution period?
what's the scrape interval ?
r
Its default currently and yes subtask gets complete within seconds
need to add this conf in controller ? 
controller.minion. task.metrics.emitter.frequencyPeriod
x
promethues scraping is usually at 30secs or 1 min interval for the value, so technically there has to be long enough queue time to detect this
you can run some controller APIs to get the task counts:
Copy code
for t in $(curl -s <http://localhost:9000/tasks/tasktypes> | jq -r '.[]'); do
  pending=$(curl -s <http://localhost:9000/tasks/$t/taskstates> | jq 'to_entries | map(select(.value=="NOT_STARTED")) | length')
  running=$(curl -s <http://localhost:9000/tasks/$t/taskstates> | jq 'to_entries | map(select(.value=="IN_PROGRESS")) | length')
  runSubs=$(curl -s <http://localhost:9000/tasks/$t/taskcounts> | jq 'values | map(.running) | add // 0')
  waitSubs=$(curl -s <http://localhost:9000/tasks/$t/taskcounts> | jq 'values | map(.waiting) | add // 0')
  echo "$t pending=$pending running=$running runningSubtasks=$runSubs waitingSubtasks=$waitSubs"
done
Open in Slack
PreviousNext
Powered by