If I see the table status as bad in the pinot ui hostname 90

Question

If I see the table status as bad in the pinot ui hostname 9000 tables what s the right way to figure out what s wrong

Ken Krugler · Answer

And yes I believe it s bad as I don t get any results when querying the table

Kishore G · Answer

whats happens when you click on that table

Kishore G · Answer

it should show the status of segments

Kishore G · Answer

table status is bad when one or more segments reach ERROR state

Ken Krugler · Answer

OK segments were below the fold I see segments are all ERROR But when I click on any of the segments I don t see anything that says what s wrong

Kishore G · Answer

check in the logs of server you will see exception

Ken Krugler · Answer

PinotFS for scheme s3 has not been initialized

Xiang Fu · Answer

so plugin dir is not set

Ken Krugler · Answer

This is using the Pinot helm chart to install Pinot in EKS AWS

Ken Krugler · Answer

I see in logs 2020 12 09 22 48 46 135 INFO PluginManager main Successfully loaded plugin pinot s3 from jar file opt pinot plugins pinot file system pinot s3 pinot s3 0 7 0 SNAPSHOT shaded jar

Xiang Fu · Answer

can you check if ` Values server pluginsDir` are set

Xiang Fu · Answer

hmm

Xiang Fu · Answer

oh

Xiang Fu · Answer

you need to put configs in pinot server conf

Xiang Fu · Answer

Xiang Fu · Answer

```pinot server storage factory class s3=org apache pinot plugin filesystem S3PinotFS pinot server storage factory s3 region=us west 2 pinot server segment fetcher protocols=file http s3 pinot server segment fetcher s3 class=org apache pinot common utils fetcher PinotFSSegmentFetcher```

Ken Krugler · Answer

So does this mean I need to log into each server in my EKS cluster and manually fix things up

Ken Krugler · Answer

I guess I assumed an EKS example would be configured to work with S3

Xiang Fu · Answer

you can update the pinot server config map

Xiang Fu · Answer

and restart pinot servers

Xiang Fu · Answer

same for pinot controller

Xiang Fu · Answer

```kubectl edit configmap controller config kubectl edit configmap server config```

Ken Krugler · Answer

Error from server NotFound configmaps server config not found

Xiang Fu · Answer

namespace

Ken Krugler · Answer

nope set that

Ken Krugler · Answer

kubectl get pods n $ EKS CLUSTER NAME works

Xiang Fu · Answer

can you check `kubectl get configmap`

Ken Krugler · Answer

kubectl edit configmap server config n $ EKS CLUSTER NAME fails so I assume it s not a namespace issue

Ken Krugler · Answer

NAME DATA AGE pinot broker config 1 86m pinot controller config 1 86m pinot server config 1 86m pinot zookeeper 3 86m

Xiang Fu · Answer

what s the output of `kubectl get configmap n $ EKS CLUSTER NAME `

Xiang Fu · Answer

ah

Ken Krugler · Answer

pinot server config

Xiang Fu · Answer

it s `pinot server config` and `pinot controller config`

Ken Krugler · Answer

I ve edited so now what s the right way to do the restart of both controller and servers I assume

Xiang Fu · Answer

controller then server

Ken Krugler · Answer

What s the best way to do that restart I can scale the pod to 0 and then back to 1 or something else

Xiang Fu · Answer

ohohoh

Xiang Fu · Answer

just delete the pod should work

Xiang Fu · Answer

```kubectl n $NAMESPACE delete $ kubectl n $NAMESPACE get pods o name | grep pinot controller || true```

Xiang Fu · Answer

similar for server

Ken Krugler · Answer

That worked thanks Though now I see in the server logs model S3Exception Access Denied

Xiang Fu · Answer

I think you can set ENV VARIABLES

Ken Krugler · Answer

Looks like I can set the accesskey secret key in the configs

Xiang Fu · Answer

yes that works as well

Xiang Fu · Answer

pinot uses default aws credential mechnism so you can set it by config file env variable or directly set the config in the pinot fs conf

Kishore G · Answer

can we please update the docs for this

Ken Krugler · Answer

< Xiang Fu> so for the controller config does it need to be pinot server storage factory class s3 or should it be pinot controller blah I assume the latter

Xiang Fu · Answer

the latter

Xiang Fu · Answer

sample conf <https docs pinot apache org users tutorials use s3 as deep store for pinot start controller>

Ken Krugler · Answer

I m closer out of 32 segments 9 are still showing as bad

Xiang Fu · Answer

any logs on server

Ken Krugler · Answer

Yes I see a few warnings e g on server 4 OK looks like an out of disk space issue for copying untarring segments Which means I probably need to figure out how to get the pods to access all of the drives on the EC2 instances sigh

Kishore G · Answer

Why all drives It s more about the capacity of persistent disk columns rt

Xiang Fu · Answer

can you increase ebs volume size

Xiang Fu · Answer

in k8s it s pvc size

Ken Krugler · Answer

Looking at the pods they currently are configured with var pinot server data pointing at a 4GB volume

Ken Krugler · Answer

I ve got 4 1 7TB SSDs that aren t mounted

Xiang Fu · Answer

you mean the ec2 machine

Ken Krugler · Answer

Right but I also see that when logged into the pod

Ken Krugler · Answer

lsblk shows

Ken Krugler · Answer

nvme0n1 259 0 0 1 7T 0 disk nvme1n1 259 1 0 1 7T 0 disk nvme3n1 259 2 0 1 7T 0 disk nvme2n1 259 3 0 1 7T 0 disk

Ken Krugler · Answer

Can i configure the Pinot server to use multiple drives when downloading segments

Xiang Fu · Answer

i don t think that is supported

Xiang Fu · Answer

I think you can increase the pvc size

Xiang Fu · Answer

`kubectl get pvc` will show you the disk mount to each pod

Ken Krugler · Answer

If it s EBS then that s not going to have nearly as good performance as SSD wouldn t that impact Pinot performance

Xiang Fu · Answer

ebs also has ssd support but you are right it s not local ssd perf

Xiang Fu · Answer

another thing you can try it to mount local disk to pinot volume

Xiang Fu · Answer

then set server data directory to that

Ken Krugler · Answer

Yes though only one of the drives would be in use

Ken Krugler · Answer

But better than nothing

Xiang Fu · Answer

ic

Ken Krugler · Answer

I guess I d need to delete all segments first then do that remapping

Xiang Fu · Answer

no need

Xiang Fu · Answer

pinot will download them from s3

Xiang Fu · Answer

if it found the data are not in data dir

Ken Krugler · Answer

Hmm I think I can map var pinot server data segment to one drive and var pinot server data index to another drive So high water mark during segment downloading will be using both drives

Xiang Fu · Answer

ok

Xiang Fu · Answer

``` var pinot server data segment ``` this is just used as segment download temp dir

Xiang Fu · Answer

once segment is untar ed and loaded they will be loaded into ``` var pinot server data index ```

Ken Krugler · Answer

Yes thanks that s what I see when poking around the filesystem

Xiang Fu · Answer

for multiple drives I think you can make them a raid then mount

Xiang Fu · Answer

try lvm2

Ken Krugler · Answer

Right now I m having some issues trying to get the drive formatting mounted in the pod Which is probably not a good idea in any case but figuring out how to update everything to auto mount the drive and set up var pinot server data appropriately is a big bite to take

Xiang Fu · Answer

right that s why we recommend to use ebs For using local disk mount if it s not the original machine to be scheduled on then disk is gone and we need to download all the segments again from s3

Ken Krugler · Answer

OK many thanks for all your help < Xiang Fu> I would definitely still be stuck without it

Chinmay Soman · Answer

I ll work with Xiang to update the docs