https://pinot.apache.org/ logo
Join Slack
Powered by
<@UDRJ7G85T> Could I know the zookeeper cluster de...
# troubleshooting
t
@Kishore G Could I know the zookeeper cluster deployment suggestion? we will have 50W segments for 4 tables. thanks.
k
what is 50W?
t
500K segments
we are estimating how many zookeeper nodes should be there to support such number of segements. thanks.
I notice the comment below, but Could you please share more info about the how many zookeeper nodes and the configurations to support such scale? thanks. @Kishore G
Copy code
At LinkedIn, we do run a cluster with several million segments (and thousands of tables), and 100s of servers. Over time, we have made improvements to pinot that helps us handle this type of load. The load is on zookeeper. Increasing bandwidth on zookeeper, separating the Helix and Pinot controller instances are things you can do.
m
50k segment per table or across all tables?
For production folks usually run 5 ZK nodes (to ensure consensus). if 50k segments across all servers, it is not that big.
t
Thanks @Mayank, It is not 50K, it is 500K segments and the biggest table is estimated to have 200K segments. so I am not sure how we can support such case.
And we have tried rollup segments which seems a nightmare for team. it would take long long time to rollup and proved a resource consuming task.
m
500k overall is still small. What’s the segment size for the table with 200k segments
t
Here is a segment example, for now, we set the segment threshold is : 15minutes or 500M
Copy code
{
  "segment.realtime.endOffset": "140557099793",
  "segment.start.time": "1660055340000",
  "segment.time.unit": "MILLISECONDS",
  "segment.flush.threshold.size": "1966448",
  "segment.realtime.startOffset": "140555344567",
  "segment.end.time": "1660056581000",
  "segment.total.docs": "1966933",
  "segment.realtime.numReplicas": "1",
  "segment.creation.time": "1660055859719",
  "segment.index.version": "v3",
  "segment.crc": "1224605237",
  "segment.realtime.status": "DONE",
  "segment.download.url": "<s3://xxxxxx>"
}
m
What’s the on disk size for this segment (it seems to have 1.9M rows).
t
It is about 56MB per segment
m
Ok, you can definitely increase the 15 minutes to 1 - 2 hours. That will reduce number of segments by a factor of 4-8
t
the segment size 56MB is the size showed in s3.
How many ZK nodes I should use? currently we use 3. it seems a little slow even to get the table meta infos.
m
For these, ZK is not going to be the bottleneck. What’s the cpu/mem on each node
Also for controllers
t
Here are some configurations:
Copy code
zookeeper:
  ## Replicas
  replicaCount: 3

  autopurge:
    ## The time interval in hours for which the purge task has to be triggered. Set to a positive integer (1 and above) to enable the auto purging.
    ##
    purgeInterval: 1
  resources:
    requests:
      memory: 8Gi
      cpu: 2
    limits:
      cpu: 2
      memory: 8Gi
Copy code
controller:
  replicaCount: 3
  persistence:
    size: "600Gi"
    mountPath: /var/pinot/server/data

  resources:
    requests:
      memory: "24Gi"
      cpu: "6"
    limits:
      cpu: "6"
      memory: "24Gi"
We setup in AWS EKS.
m
It might help to vertically size controllers first
t
Any advice for the controller size? 6
m
Increase to 16 core 64 GB or around that range first
t
For controller?
And any suggestion for Broker and Server?
m
That depends on your workload
t
I know brokers and servers are scalable. but is there any resource configuration in vertically suggestion?
m
Open in Slack
PreviousNext
Powered by