https://pinot.apache.org/ logo
Join Slack
Powered by
hello team, I am wondering how I can transition m...
# general
u
hello team, I am wondering how I can transition my prod cluster to use deep storage. currently, i have a prod pinot cluster running on aws eks and i want to transition it to use s3 backed deep storage without losing data from current cluster. is that possible? what are the options?
m
what do you use for deep storage at the moment - eks?
u
we didnt specify any deep storage so i assume it is default. i do remember that it is being saved into the containers of my server pods running in eks. to some path @Mark Needham
but now we want to enable deep storage with s3
any ideas? @Mark Needham
m
I think the segments would be written to 
/var/pinot/controller/data
- can you see if you have some files there?
u
@Mark Needham - correct.
what do you suggest?
that i manually copy these files from the container to s3 bucketr?
m
yeh that’s what I was thinking - are they tar.gz files?
u
Copy code
bash-4.2# ls -al var/pinot/controller/data/pinot_data_action/
total 22544
drwxr-sr-x.  2 root users    4096 Apr 20 07:56 .
drwxrwsr-x. 10 root users    4096 Apr 20 08:39 ..
-rw-r--r--.  1 root users 2647719 Apr 19 07:56 pinot_data_action__0__0__20230419T0756Z
-rw-r--r--.  1 root users 3664986 Apr 19 07:56 pinot_data_action__0__1__20230419T0756Z
-rw-r--r--.  1 root users 1361136 Apr 20 07:56 pinot_data_action__0__2__20230419T0756Z
-rw-r--r--.  1 root users 2646995 Apr 19 07:56 pinot_data_action__1__0__20230419T0756Z
-rw-r--r--.  1 root users 3659841 Apr 19 07:56 pinot_data_action__1__1__20230419T0756Z
-rw-r--r--.  1 root users 1372008 Apr 20 07:56 pinot_data_action__1__2__20230419T0756Z
-rw-r--r--.  1 root users 2650171 Apr 19 07:56 pinot_data_action__2__0__20230419T0756Z
-rw-r--r--.  1 root users 3657582 Apr 19 07:56 pinot_data_action__2__1__20230419T0756Z
-rw-r--r--.  1 root users 1398606 Apr 20 07:56 pinot_data_action__2__2__20230419T0756Z
bash-4.2#
this is what i see
m
should be possible to do it. This is how to configure Deep store on S3 - https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot I’ve been playing with this config today and I’ve got it working if I put those config in my values.yaml file under 
extra.config
And then I guess we’d need to update the meta data in zookeeper to tell it that the segments are now elsewhere. I need to figure out exactly what to do there so I don’t tell you the wrong thing
u
@Mark Needham - i was successfully able to implement deep storage
but i have another question for you which pops up in my head now.
there are actually couple of them and here is the first one: consider below scenario: today is 24th april i have a kafka topic with retention policy of 30 days. in this particular scenario, i have deep stored all my data upto 24th april in s3 bucket. i have data in kafka topic from 24th march - 24th april. suddenly 2 days later my pinot cluster went dead, i deployed a new cluster and created realtime tables. what will pinot do in this case? will it process the whole data in kafka topic again and duplicate all the values or will it only process new data from kafka topic and wont duplicate my s3 bucket? @Mark Needham
m
how did you do it?
👀 1
the segments contain kafka offset data, so if the new cluster uses the same deep store, I think it’ll continue from the latest offset. I’m not sure if there’s anything special you need to do to make that happen. Probably @Mayank will know for sure
u
@Mark Needham just applied the attached custom values file to the cluster and it started storing data in s3.
pinot-custom-values_yaml.yaml
however, i have an issue. whenever i try to upgrade the non-s3 deep storage cluster with above settings, the cluster just goes into this race condition. how can i solve that?
m
did you copy across the segments that had already been created?
u
if you wish, i can share the logs of the race condition i suspect.
m
yeh, go for it
u
Copy code
did you copy across the segments that had already been created?
regarding this question, i couldnt since when i upgraded the cluster with s3 deep storage settings, it went into race conditions. allow me few mins and ill upload the logs
hey @Mark Needham, here are the logs i get whenever i upgrade the helm chart with the custom file which has s3 deep storage configs:
Copy code
Failed to start a Pinot [BROKER] at 3.212 since launch
org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot-quickstart
	at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1385) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.helix.manager.zk.ZKHelixManager.createClient(ZKHelixManager.java:782) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.helix.manager.zk.ZKHelixManager.connect(ZKHelixManager.java:819) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.pinot.broker.broker.helix.BaseBrokerStarter.start(BaseBrokerStarter.java:238) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.pinot.tools.service.PinotServiceManager.startBroker(PinotServiceManager.java:143) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:92) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:278) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:278) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-96d34fd280b154c9d6f8435ac787b7bebd1e631e]
Shutting down Pinot Service Manager with all running Pinot instances...
Shutting down Pinot Service Manager admin application...
Deregistering service status handler
i assume this is a race condition
now, broker and minion and server continues to restart forever:
Copy code
Every 2.0s: kubectl get pods -n pinot-ds                                                ip-192-168-0-102.ap-south-1.compute.internal: Tue Apr 25 16:06:29 2023

NAME                                      READY   STATUS             RESTARTS      AGE
pinot-broker-0                            0/1     CrashLoopBackOff   4 (53s ago)   2m50s
pinot-controller-0                        1/1     Running            0             68m
pinot-minion-stateless-6776dbb549-9jsn2   0/1     CrashLoopBackOff   4 (54s ago)   3m1s
pinot-minion-stateless-6fbf74bfd9-khmds   1/1     Running            0             68m
pinot-server-0                            0/1     CrashLoopBackOff   4 (27s ago)   2m46s
pinot-zookeeper-0                         1/1     Running            0             68m
m
I mean you are right that it’s a race condition
but I’m kinda curious why updating the deep store config makes it happen
u
helm upgrade -f work/helm/pinot/pinot-custom-values.yaml pinot pinot/pinot -n pinot-ds
m
u
this is the command i used
m
this is the race condition
image.png
your controller is running though! Anything in the logs?
u
looks like it is running:
Copy code
Closing reporter org.apache.pinot.shaded.org.apache.kafka.common.metrics.JmxReporter
Metrics reporters closed
App info kafka.consumer for PartitionGroupMetadataFetcher-data_action_REALTIME-pinot_data_action unregistered
Reading segment sizes from 2 servers for table: data_action_REALTIME with timeout: 30000ms
Caught 'java.net.UnknownHostException: pinot-server-0.pinot-server-headless.pinot-ds.svc.cluster.local' while executing: GET on URL: <http://pinot-server-0.pinot-server-headless.pinot-ds.svc.cluster.local:8097/table/data_action_REALTIME/size>
Caught 'java.net.UnknownHostException: pinot-server-1.pinot-server-headless.pinot-ds.svc.cluster.local' while executing: GET on URL: <http://pinot-server-1.pinot-server-headless.pinot-ds.svc.cluster.local:8097/table/data_action_REALTIME/size>
Connection error in 'get segment size info from servers'. Details: java.net.UnknownHostException: pinot-server-0.pinot-server-headless.pinot-ds.svc.cluster.local
Connection error in 'get segment size info from servers'. Details: java.net.UnknownHostException: pinot-server-1.pinot-server-headless.pinot-ds.svc.cluster.local
Finished reading information for table: data_action_REALTIME with 0/2 server responses
Failed to get size report for all 9 segments of table: data_action_REALTIME
Finish processing 1/1 tables in task: SegmentStatusChecker
[TaskRequestId: auto] Finish running task: SegmentStatusChecker in 272ms
Getting Helix leader: pinot-controller-0.pinot-controller-headless.pinot-ds.svc.cluster.local_9000, Helix version: 1.0.4, mtime: 1682414936368
Starting TaskMetricsEmitter with running frequency of 300 seconds.
[TaskRequestId: auto] Start running task: TaskMetricsEmitter
[TaskRequestId: auto] Finish running task: TaskMetricsEmitter in 1ms
Getting Helix leader: pinot-controller-0.pinot-controller-headless.pinot-ds.svc.cluster.local_9000, Helix version: 1.0.4, mtime: 1682414936368
Getting Helix leader: pinot-controller-0.pinot-controller-headless.pinot-ds.svc.cluster.local_9000, Helix version: 1.0.4, mtime: 1682414936368
controller logs
error is generated when it is trying to connect with server
m
yeh
b/c the server didn’t stasrt
u
since server is down, it is throwing error
m
can you show me the contents of ‘work/helm/pinot/pinot-custom-values.yaml’ (delete any credentials first)
u
it is the same file i shared with you
m
oh lemme check
did we change the name of the cluster by mistake
don’t think we did
u
i dont think
basically; before applying upgrade: pinot cluster saving segments in eks after applying upgrade: it should store segments in s3 but that doesnt happen as it cannot get to even work in the first place.
there is another way. tear down the whole cluster but it is not possible since ill lose the data
m
and if you revert that deepstore config
it starts happily?
u
you mean if i use 
helm rollback
command?
m
oh I dunno how you’d do it.
So basically since you applied the deep store config it’s stuck?
u
when i "upgrade" the cluster with deep storage deployment gets stuck. yes.
btw, i just rolled back and it works now
but again, it is storing data in eks right now
m
yes
do you see any lines like this in the logs:
Copy code
Invalid cluster setup for cluster
Copy code
missing znode path:
u
in which logs? broker?
m
yeh broker
u
correct
Copy code
Invalid cluster setup for cluster: pinot-quickstart, missing znode path: /pinot-quickstart/IDEALSTATES
means zookeeper is out of sync?
m
wanna screenshare and I’ll have a look?
u
sure
ill share a link
2 Views
Open in Slack
PreviousNext
Powered by