https://pinot.apache.org/ logo
#pinot-dev
Title
# pinot-dev
j

Jack

09/02/2020, 6:04 PM
just curious, why the pool size for HelixTaskExecutor is 1 for your cluster?
t

Ting Chen

09/02/2020, 6:05 PM
It seems to be the default for STATE_TRANSITION.
is this configurable?
j

Jack

09/02/2020, 6:12 PM
I think so. E.g. this is a sample log from one of the pinot servers in production:
Copy code
2020/09/02 17:53:02.586 INFO [HelixTaskExecutor] [<http://ZkClient-EventThread-73-zk-lva1-pinot.prod.linkedin.com:12913/pinot-cluster|ZkClient-EventThread-73-zk-lva1-pinot.prod.linkedin.com:12913/pinot-cluster>] [pinot-server] [] Submit task: 7cfb2b9b-78e2-4a01-a03a-7c8ff38fddd5 to pool: java.util.concurrent.ThreadPoolExecutor@3d1c933[Running, pool size = 20, active threads = 1, queued tasks = 0, completed tasks = 5416]
t

Ting Chen

09/02/2020, 6:16 PM
we always used 1 in prod before. adding threads to pool can avoid the deadlock because the other task can be executed. Would you mind to share the Pinot server config related to pool size?
j

Jack

09/02/2020, 6:24 PM
checking the code to see where it should be set now
Found it:
Copy code
"STATE_TRANSITION.maxThreads" : "20",
This should be set in
CONFIGS/CLUSTER/pinot
ZNode.
So the content of that ZNode should be like:
Copy code
{
  "id": "pinot",
  "simpleFields": {
    "STATE_TRANSITION.maxThreads": "20",
    "allowParticipantAutoJoin": "true"
  },
  "listFields": {},
  "mapFields": {}
}
Once this is set, keep in mind that controller be restarted first, then followed by the rest pinot component to pick up this setting
t

Ting Chen

09/02/2020, 6:33 PM
thanks and let me try it out.
đź‘Ť 1
I also assign the issue to you and please add details on the issue too.
j

Jack

09/02/2020, 6:37 PM
And I deep dived into the Helix code, the default max thread shouldn’t be just 1. It’s 40.
t

Ting Chen

09/02/2020, 6:38 PM
I checked our zk config for STATE_TRANSITION.maxThreads and it is set to 1.
j

Jack

09/02/2020, 6:38 PM
Oh ok. Yeah it should be set to a higher number
t

Ting Chen

09/02/2020, 6:39 PM
Let me find out why it is the case.
We upgraded the cluster from pinot 0.1. Given the config is in zk, it could be stored there by Pinot controllers. I wonder why we can not make it an explicit cluster config.
j

Jack

09/02/2020, 8:10 PM
This is a Helix setting and once it’s set, it’ll rarely be changed. I think you can just remove that setting in the cluster, since 40 is a fair number for pinot-server to do realtime consumption.
t

Ting Chen

09/02/2020, 8:57 PM
hmmm... this is the log before I applied the Zk manual override
2020-09-01 22:14:05.208 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: STATE_TRANSITION, poolSize: 1, factory: org.apache.helix.participant.HelixStateMachineEngine@625e134e, pool: java.util.concurrent.ThreadPoolExecutor@72bd06ca[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
as you can see the pool size is 1.
for other message type, the pool size is 40
2020-09-01 22:14:07.062 [main] INFO org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: TASK_REPLY, poolSize: 40, factory: org.apache.helix.messaging.handling.AsyncCallbackService@517bd097, pool: java.util.concurrent.ThreadPoolExecutor@142eef62[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
j

Jack

09/02/2020, 8:59 PM
have you restarted all the components sequentially?
t

Ting Chen

09/02/2020, 9:00 PM
no. this is the log yesterday before I made any change.
I just want to make sure the 40 default value is applied to the STATE_TRANSITION message type.
j

Jack

09/02/2020, 9:03 PM
Okay sure, just set `"STATE_TRANSITION.maxThreads" : "40"`or remove that setting, and restart all the pinot components
t

Ting Chen

09/02/2020, 9:04 PM
for our current problem, your solution is good. I am thinking about two main scenarios:
(1) Users start with an empty Zk node and deploy the latest Pinot 4.0.
(2) Users migrate from an older version of Pinot (with old version of Helix), do they need to sth manual zk value override.
We were doing (2) and the STATE_TRANSITION.maxThreads was 1 in our zk node.
j

Jack

09/02/2020, 9:16 PM
(1) is easy to validate. You can spin up a fresh cluster and check whether the setting is there or not. For (2) most of the cases we don’t need to do any extra change. But since you’re upgrading from 1.0 to 4.0 which is like a huge version bump, there may have some changes we need to take care of. Fortunately the thread info is logged in the server log. You can firstly do the upgrade on a smaller cluster, validate the performance and then roll out to larger clusters.
t

Ting Chen

09/02/2020, 9:21 PM
We were doing (2) in a small test cluster and found the issue.
It looks like in 0.1 the maxThreads was set to be 1.
j

Jack

09/02/2020, 9:24 PM
please explicitly set this config to the rest clusters before upgrading to 4.0 everywhere.
I remember in 1.0 we had a commit that the default value is set to 1 in the cluster if there is no value for it. So please update the value of
STATE_TRANSITION.maxThreads
before you do the upgrade.
t

Ting Chen

09/02/2020, 9:35 PM
thanks and would do that.