Does anyone know , how to migrate apache pinot? I ...
# troubleshooting
t
Does anyone know , how to migrate apache pinot? I have started a new apache pinot cluster , recreated all tables , copied the ZK Metadata from IDEALSTATES and PROPERTYSTORE and I have changed the deepstore location too but my segments remain in "Updating" , none of them seem to load. Any ideas?
x
are server instances names same for old/new clusters?
can you check if new server instances are trying to download any segment or even doing anything?
t
@Xiang Fu the server names are different, but I of course changed that too
x
then you need to update the Idealstates mapping from old servers to new servers
hopefully you have 1:1 mapping from old/new servers
or if you have deepstore, just run segment metadata push from your deep store to new cluster
t
I have deepstore of course , migrated that from s3 to gcs bucket
how would i do a metadata push from deepstore?
If you dont mind sharing
because i have also migrated IDEALSTATE , but all of them return in ERROR state after 2 minutes
if this is relevant , my tables are REALTIME not OFFLINE
x
hmmm
if it’s realtime table
then you need to read old idealstate, update all the segment to instance mapping to the new instance name
t
Should i edit anything in the idealstate from the old cluster? because theyre ONLINE in the old cluster but in the new one they will not be present they will need to be re-downloaded
x
Don’t touch anything in the old cluster. copy it from old cluster, update instance id then set it in the new cluster
t
Okay , i'll update with results
x
you also need to update the segment metadata downloadUrl when you copy segments from s3 to gcs
t
Yeah thats already been done in the PROPERTYSTORE
x
so new instances in the new cluster will be able to download them from gcs
cool
then just update the IdealState mapping should be good
t
I shouldnt touch EXTERNALVIEW , right?
x
don’t touch EXTERNALVIEW, it should auto updated based on the IDEALSTATE+ state transitions
t
Yeah I didnt touch it , but its been stuck at UPDATING for 3hours now
x
do you see any segment loaded in new servers?
t
unfortunately no
x
then likely your idealstate is still wrong
t
shall i send the idealstate?
Its currently stuck at 0/264 segments , UPDATING
but i also dont see any logs of the server attempting to download the segments
x
yes
please send your old and new Idealstates
and your old, new server instances id
t
These are the idealstates , respectively
x
can you go to pinot UI and check the server instance names
t
Copy code
Server_pinot-prod-server-1.pinot-prod-server-headless.pinot.svc.cluster.local_8098
Server_pinot-prod-server-0.pinot-prod-server-headless.pinot.svc.cluster.local_8098
x
what’s the server log when you restarted the server-0
try to login to the container and fetch the log file
t
it started repopulating segments now!
Thanks a lot Xiang , much much appreciated!
x
👍