https://pinot.apache.org/ logo
Join Slack
Powered by
If I see the table status as “bad” in the pinot ui...
# troubleshooting
k
If I see the table status as “bad” in the pinot ui (hostname:9000/#/tables), what’s the right way to figure out what’s wrong?
And yes, I believe it’s bad as I don’t get any results when querying the table.
k
whats happens when you click on that table
it should show the status of segments
table status is bad when one or more segments reach ERROR state
k
OK, segments were “below the fold”. I see segments are all ERROR. But when I click on any of the segments, I don’t see anything that says what’s wrong.
k
check in the logs of server, you will see exception
k
PinotFS for scheme: s3 has not been initialized
x
so plugin dir is not set?
k
This is using the Pinot helm chart to install Pinot in EKS/AWS.
I see in logs - “2020/12/09 224846.135 INFO [PluginManager] [main] Successfully loaded plugin [pinot-s3] from jar file [/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.7.0-SNAPSHOT-shaded.jar]”
x
can you check if 
.Values.server.pluginsDir
are set?
hmm
oh
you need to put configs in pinot server conf
Copy code
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
k
So does this mean I need to log into each server in my EKS cluster and manually fix things up?
I guess I assumed an EKS example would be configured to work with S3…
x
you can update the pinot-server config map
and restart pinot servers
same for pinot controller
Copy code
kubectl edit configmap controller-config
kubectl edit configmap server-config
k
Error from server (NotFound): configmaps “server-config” not found
x
namespace?
k
nope - set that
kubectl get pods -n ${EKS_CLUSTER_NAME} works
x
can you check 
kubectl get configmap
k
kubectl edit configmap server-config -n ${EKS_CLUSTER_NAME} fails, so I assume it’s not a namespace issue
NAME           DATA  AGE pinot-broker-config    1   86m pinot-controller-config  1   86m pinot-server-config    1   86m pinot-zookeeper      3   86m
x
what’s the output of 
kubectl get configmap -n ${EKS_CLUSTER_NAME}
ah
k
pinot-server-config
x
it’s 
pinot-server-config
and 
pinot-controller-config
k
I’ve edited, so now…what’s the right way to do the restart (of both controller and servers, I assume)
x
controller then server
k
What’s the best way to do that restart? I can scale the pod to 0, and then back to 1…or something else?
x
ohohoh
just delete the pod should work
Copy code
kubectl -n $NAMESPACE delete $(kubectl -n $NAMESPACE get pods -o name | grep "pinot-controller") || true
similar for server
k
That worked, thanks! Though now I see in the server logs “model.S3Exception: Access Denied”
x
I think you can set ENV_VARIABLES
k
Looks like I can set the accesskey/secret key in the configs
x
yes, that works as well
pinot uses default aws credential mechnism, so you can set it by config file/env variable or directly set the config in the pinot fs conf
k
can we please update the docs for this?
👌 1
k
@Xiang Fu - so for the controller config, does it need to be pinot.server.storage.factory.class.s3, or should it be pinot.controller.blah? I assume the latter
x
the latter
k
I’m closer - out of 32 segments, 9 are still showing as bad.
x
any logs on server?
k
Yes - I see a few warnings (e.g. on server 4). OK, looks like an out of disk space issue for copying/untarring segments. Which means I probably need to figure out how to get the pods to access all of the drives on the EC2 instances…sigh
k
Why all drives? It’s more about the capacity of persistent disk columns rt
x
can you increase ebs volume size?
in k8s it’s pvc size
k
Looking at the pods, they currently are configured with /var/pinot/server/data pointing at a 4GB volume.
I’ve got 4 1.7TB SSDs that aren’t mounted.
x
you mean the ec2 machine?
k
Right - but I also see that when logged into the pod
lsblk shows:
nvme0n1 259:0   0 1.7T 0 disk  nvme1n1 259:1   0 1.7T 0 disk  nvme3n1 259:2   0 1.7T 0 disk  nvme2n1 259:3   0 1.7T 0 disk
Can i configure the Pinot server to use multiple drives when downloading segments?
x
i don’t think that is supported
I think you can increase the pvc size
kubectl get pvc
will show you the disk mount to each pod
k
If it’s EBS, then that’s not going to have nearly as good performance as SSD - wouldn’t that impact Pinot performance?
x
ebs also has ssd support, but you are right, it’s not local ssd perf
another thing you can try it to mount local disk to pinot volume
then set server data directory to that
k
Yes, though only one of the drives would be in use
But better than nothing
x
ic
k
I guess I’d need to delete all segments first, then do that remapping
x
no need
pinot will download them from s3
if it found the data are not in data dir
k
Hmm - I think I can map /var/pinot/server/data/segment/ to one drive, and /var/pinot/server/data/index/ to another drive. So high water mark during segment downloading will be using both drives.
x
ok
Copy code
/var/pinot/server/data/segment/
this is just used as segment download temp dir
once segment is untar’ed and loaded, they will be loaded into
Copy code
/var/pinot/server/data/index/
k
Yes, thanks - that’s what I see when poking around the filesystem.
x
for multiple drives, I think you can make them a raid then mount
try lvm2
k
Right now I’m having some issues trying to get the drive formatting/mounted in the pod. Which is probably not a good idea in any case, but figuring out how to update everything to auto-mount the drive and set up /var/pinot/server/data/ appropriately is a big bite to take
x
right, that’s why we recommend to use ebs. For using local disk mount, if it’s not the original machine to be scheduled on, then disk is gone and we need to download all the segments again from s3
k
OK - many thanks for all your help @Xiang Fu, I would definitely still be stuck without it!
c
I'll work with Xiang to update the docs
Open in Slack
PreviousNext
Powered by