https://pinot.apache.org/ logo
Join Slack
Powered by
just curious, why the pool size for HelixTaskExecu...
# pinot-dev
j
just curious, why the pool size for HelixTaskExecutor is 1 for your cluster?
t
It seems to be the default for STATE_TRANSITION.
is this configurable?
j
I think so. E.g. this is a sample log from one of the pinot servers in production:
Copy code
2020/09/02 17:53:02.586 INFO [HelixTaskExecutor] [<http://ZkClient-EventThread-73-zk-lva1-pinot.prod.linkedin.com:12913/pinot-cluster|ZkClient-EventThread-73-zk-lva1-pinot.prod.linkedin.com:12913/pinot-cluster>] [pinot-server] [] Submit task: 7cfb2b9b-78e2-4a01-a03a-7c8ff38fddd5 to pool: java.util.concurrent.ThreadPoolExecutor@3d1c933[Running, pool size = 20, active threads = 1, queued tasks = 0, completed tasks = 5416]
t
we always used 1 in prod before. adding threads to pool can avoid the deadlock because the other task can be executed. Would you mind to share the Pinot server config related to pool size?
j
checking the code to see where it should be set now
Found it:
Copy code
"STATE_TRANSITION.maxThreads" : "20",
This should be set in 
CONFIGS/CLUSTER/pinot
ZNode.
So the content of that ZNode should be like:
Copy code
{
  "id": "pinot",
  "simpleFields": {
    "STATE_TRANSITION.maxThreads": "20",
    "allowParticipantAutoJoin": "true"
  },
  "listFields": {},
  "mapFields": {}
}
Once this is set, keep in mind that controller be restarted first, then followed by the rest pinot component to pick up this setting
t
thanks and let me try it out.
đź‘Ť 1
I also assign the issue to you and please add details on the issue too.
j
And I deep dived into the Helix code, the default max thread shouldn’t be just 1. It’s 40.
t
I checked our zk config for STATE_TRANSITION.maxThreads and it is set to 1.
j
Oh ok. Yeah it should be set to a higher number
t
Let me find out why it is the case.
We upgraded the cluster from pinot 0.1. Given the config is in zk, it could be stored there by Pinot controllers. I wonder why we can not make it an explicit cluster config.
j
This is a Helix setting and once it’s set, it’ll rarely be changed. I think you can just remove that setting in the cluster, since 40 is a fair number for pinot-server to do realtime consumption.
t
hmmm... this is the log before I applied the Zk manual override
2020-09-01 22:14:05.208 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: STATE_TRANSITION, poolSize: 1, factory: org.apache.helix.participant.HelixStateMachineEngine@625e134e, pool: java.util.concurrent.ThreadPoolExecutor@72bd06ca[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
as you can see the pool size is 1.
for other message type, the pool size is 40
2020-09-01 22:14:07.062 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: TASK_REPLY, poolSize: 40, factory: org.apache.helix.messaging.handling.AsyncCallbackService@517bd097, pool: java.util.concurrent.ThreadPoolExecutor@142eef62[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
j
have you restarted all the components sequentially?
t
no. this is the log yesterday before I made any change.
I just want to make sure the 40 default value is applied to the STATE_TRANSITION message type.
j
Okay sure, just set `"STATE_TRANSITION.maxThreads" : "40"`or remove that setting, and restart all the pinot components
t
for our current problem, your solution is good. I am thinking about two main scenarios:
(1) Users start with an empty Zk node and deploy the latest Pinot 4.0.
(2) Users migrate from an older version of Pinot (with old version of Helix), do they need to sth manual zk value override.
We were doing (2) and the STATE_TRANSITION.maxThreads was 1 in our zk node.
j
(1) is easy to validate. You can spin up a fresh cluster and check whether the setting is there or not. For (2) most of the cases we don’t need to do any extra change. But since you’re upgrading from 1.0 to 4.0 which is like a huge version bump, there may have some changes we need to take care of. Fortunately the thread info is logged in the server log. You can firstly do the upgrade on a smaller cluster, validate the performance and then roll out to larger clusters.
t
We were doing (2) in a small test cluster and found the issue.
It looks like in 0.1 the maxThreads was set to be 1.
j
please explicitly set this config to the rest clusters before upgrading to 4.0 everywhere.
I remember in 1.0 we had a commit that the default value is set to 1 in the cluster if there is no value for it. So please update the value of 
STATE_TRANSITION.maxThreads
before you do the upgrade.
t
thanks and would do that.
Open in Slack
PreviousNext
Powered by