https://pinot.apache.org/ logo
k

Ken Krugler

12/09/2020, 11:07 PM
If I see the table status as “bad” in the pinot ui (hostname:9000/#/tables), what’s the right way to figure out what’s wrong?
And yes, I believe it’s bad as I don’t get any results when querying the table.
k

Kishore G

12/09/2020, 11:47 PM
whats happens when you click on that table
it should show the status of segments
table status is bad when one or more segments reach ERROR state
k

Ken Krugler

12/09/2020, 11:49 PM
OK, segments were “below the fold”. I see segments are all ERROR. But when I click on any of the segments, I don’t see anything that says what’s wrong.
k

Kishore G

12/09/2020, 11:50 PM
check in the logs of server, you will see exception
k

Ken Krugler

12/10/2020, 12:01 AM
PinotFS for scheme: s3 has not been initialized
x

Xiang Fu

12/10/2020, 12:03 AM
so plugin dir is not set?
k

Ken Krugler

12/10/2020, 12:03 AM
This is using the Pinot helm chart to install Pinot in EKS/AWS.
I see in logs - “2020/12/09 224846.135 INFO [PluginManager] [main] Successfully loaded plugin [pinot-s3] from jar file [/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.7.0-SNAPSHOT-shaded.jar]”
x

Xiang Fu

12/10/2020, 12:05 AM
can you check if
.Values.server.pluginsDir
are set?
hmm
oh
you need to put configs in pinot server conf
Copy code
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
k

Ken Krugler

12/10/2020, 12:07 AM
So does this mean I need to log into each server in my EKS cluster and manually fix things up?
I guess I assumed an EKS example would be configured to work with S3…
x

Xiang Fu

12/10/2020, 12:07 AM
you can update the pinot-server config map
and restart pinot servers
same for pinot controller
Copy code
kubectl edit configmap controller-config
kubectl edit configmap server-config
k

Ken Krugler

12/10/2020, 12:13 AM
Error from server (NotFound): configmaps “server-config” not found
x

Xiang Fu

12/10/2020, 12:13 AM
namespace?
k

Ken Krugler

12/10/2020, 12:13 AM
nope - set that
kubectl get pods -n ${EKS_CLUSTER_NAME} works
x

Xiang Fu

12/10/2020, 12:13 AM
can you check
kubectl get configmap
k

Ken Krugler

12/10/2020, 12:13 AM
kubectl edit configmap server-config -n ${EKS_CLUSTER_NAME} fails, so I assume it’s not a namespace issue
NAME           DATA  AGE pinot-broker-config    1   86m pinot-controller-config  1   86m pinot-server-config    1   86m pinot-zookeeper      3   86m
x

Xiang Fu

12/10/2020, 12:14 AM
what’s the output of
kubectl get configmap -n ${EKS_CLUSTER_NAME}
ah
k

Ken Krugler

12/10/2020, 12:14 AM
pinot-server-config
x

Xiang Fu

12/10/2020, 12:14 AM
it’s
pinot-server-config
and
pinot-controller-config
k

Ken Krugler

12/10/2020, 12:19 AM
I’ve edited, so now…what’s the right way to do the restart (of both controller and servers, I assume)
x

Xiang Fu

12/10/2020, 12:19 AM
controller then server
k

Ken Krugler

12/10/2020, 12:23 AM
What’s the best way to do that restart? I can scale the pod to 0, and then back to 1…or something else?
x

Xiang Fu

12/10/2020, 12:23 AM
ohohoh
just delete the pod should work
Copy code
kubectl -n $NAMESPACE delete $(kubectl -n $NAMESPACE get pods -o name | grep "pinot-controller") || true
similar for server
k

Ken Krugler

12/10/2020, 12:29 AM
That worked, thanks! Though now I see in the server logs “model.S3Exception: Access Denied”
x

Xiang Fu

12/10/2020, 12:30 AM
I think you can set ENV_VARIABLES
k

Ken Krugler

12/10/2020, 12:31 AM
Looks like I can set the accesskey/secret key in the configs
x

Xiang Fu

12/10/2020, 12:31 AM
yes, that works as well
pinot uses default aws credential mechnism, so you can set it by config file/env variable or directly set the config in the pinot fs conf
k

Kishore G

12/10/2020, 12:34 AM
can we please update the docs for this?
👌 1
k

Ken Krugler

12/10/2020, 12:36 AM
@Xiang Fu - so for the controller config, does it need to be pinot.server.storage.factory.class.s3, or should it be pinot.controller.blah? I assume the latter
x

Xiang Fu

12/10/2020, 12:37 AM
the latter
k

Ken Krugler

12/10/2020, 12:48 AM
I’m closer - out of 32 segments, 9 are still showing as bad.
x

Xiang Fu

12/10/2020, 12:49 AM
any logs on server?
k

Ken Krugler

12/10/2020, 12:52 AM
Yes - I see a few warnings (e.g. on server 4). OK, looks like an out of disk space issue for copying/untarring segments. Which means I probably need to figure out how to get the pods to access all of the drives on the EC2 instances…sigh
k

Kishore G

12/10/2020, 12:55 AM
Why all drives? It’s more about the capacity of persistent disk columns rt
x

Xiang Fu

12/10/2020, 12:57 AM
can you increase ebs volume size?
in k8s it’s pvc size
k

Ken Krugler

12/10/2020, 12:58 AM
Looking at the pods, they currently are configured with /var/pinot/server/data pointing at a 4GB volume.
I’ve got 4 1.7TB SSDs that aren’t mounted.
x

Xiang Fu

12/10/2020, 12:59 AM
you mean the ec2 machine?
k

Ken Krugler

12/10/2020, 1:00 AM
Right - but I also see that when logged into the pod
lsblk shows:
nvme0n1 259:0   0 1.7T 0 disk  nvme1n1 259:1   0 1.7T 0 disk  nvme3n1 259:2   0 1.7T 0 disk  nvme2n1 259:3   0 1.7T 0 disk
Can i configure the Pinot server to use multiple drives when downloading segments?
x

Xiang Fu

12/10/2020, 1:02 AM
i don’t think that is supported
I think you can increase the pvc size
kubectl get pvc
will show you the disk mount to each pod
k

Ken Krugler

12/10/2020, 1:05 AM
If it’s EBS, then that’s not going to have nearly as good performance as SSD - wouldn’t that impact Pinot performance?
x

Xiang Fu

12/10/2020, 1:08 AM
ebs also has ssd support, but you are right, it’s not local ssd perf
another thing you can try it to mount local disk to pinot volume
then set server data directory to that
k

Ken Krugler

12/10/2020, 1:09 AM
Yes, though only one of the drives would be in use
But better than nothing
x

Xiang Fu

12/10/2020, 1:09 AM
ic
k

Ken Krugler

12/10/2020, 1:09 AM
I guess I’d need to delete all segments first, then do that remapping
x

Xiang Fu

12/10/2020, 1:09 AM
no need
pinot will download them from s3
if it found the data are not in data dir
k

Ken Krugler

12/10/2020, 1:12 AM
Hmm - I think I can map /var/pinot/server/data/segment/ to one drive, and /var/pinot/server/data/index/ to another drive. So high water mark during segment downloading will be using both drives.
x

Xiang Fu

12/10/2020, 1:12 AM
ok
Copy code
/var/pinot/server/data/segment/
this is just used as segment download temp dir
once segment is untar’ed and loaded, they will be loaded into
Copy code
/var/pinot/server/data/index/
k

Ken Krugler

12/10/2020, 1:14 AM
Yes, thanks - that’s what I see when poking around the filesystem.
x

Xiang Fu

12/10/2020, 1:19 AM
for multiple drives, I think you can make them a raid then mount
try lvm2
k

Ken Krugler

12/10/2020, 1:21 AM
Right now I’m having some issues trying to get the drive formatting/mounted in the pod. Which is probably not a good idea in any case, but figuring out how to update everything to auto-mount the drive and set up /var/pinot/server/data/ appropriately is a big bite to take
x

Xiang Fu

12/10/2020, 1:31 AM
right, that’s why we recommend to use ebs. For using local disk mount, if it’s not the original machine to be scheduled on, then disk is gone and we need to download all the segments again from s3
k

Ken Krugler

12/10/2020, 1:39 AM
OK - many thanks for all your help @Xiang Fu, I would definitely still be stuck without it!
c

Chinmay Soman

12/10/2020, 7:42 PM
I'll work with Xiang to update the docs