https://pinot.apache.org/ logo
Join Slack
Powered by
<@UDQU92KBK> <@UGRJA9TEH> we recently introduced d...
# troubleshooting
s
@Mayank @Xiang Fu we recently introduced deepstore in our cluster but we see only newly compelted segments are only reaching deepstore . we did repair and reload segments , and force commit but doesnt seem to get the old segments into deepstore. are we missing anything? i dont see documentations around these details aswell apart form what we have already tried
x
I think that's the expected behavior, you need to manually copy the old segments to deep store and update the segment metadata to point to the deep store.
You can write a script to do this
s
ok are there any apis. we are unsure if the script would copy all required metadata
"POST /segments/{tableNameWithType}/uploadFromServerToDeepstore&forceMode=true"
Copy code
we tried this api but its only getting the completed ones not the older ones
x
are old segments tar gz files still in controller local? If so, you can copy that segment file to deep store then use zk PUT api to update the segment metadata for downloadUrl field.
s
so we need to hit these apis for each segment in the script thst was the problem . if there is any other best alternative let us know otherwise we are ok with this solution
x
Yes, those are per segment ops
s
Hi @Xiang Fu we are tying to download old segmetns and upload to minio s3 , but now the download url in segment shows controller url where as the download api from swagger shows its trying to download form s3 . whats the way forward?
so our scripts are failing due to this
x
Change the segment metadata from http path to s3 path
s
old segments havent reached to s3 yet right? once the download script is successfull we will have to upload and then update the metadata to s3. the download itslef isnt happening.
download api is poiting to s3 url where as the actual segment download url is controller url itslef. these haven't moved to s3 yet . these are segments prior to deepstore.
x
I see, then it means the deep store is s3 so that all http url will also be redirect to the s3 url already.
s
yeah
any options to load these segments to deepstore ?
x
S3 copy
I think those segment tar files are still on your old controller deep store location
Which should be local to your controller?
s
yes its the controller url we have inhouse kuber netis cluster vms . so data is in those physical controller and servers . our cluster is one year old but minio deepstore is of 2 weeks old.
apis right now wrongly point to s3 instead of those controller urls. we are on 1.5 version of pinot.
any help here? is it some bug or something? we need a way to download the segments
x
Do you have access to controller local disk? Those segments should be there
In old data deep store location
If not, you can try to tar gz the each segment directory from server data dir and then upload it
s
yeah so we arent allowed to touch infra machines. so checking for the api way.
x
then maybe get a infra ticket to access those, or just write a script to do so, I suppose you run it in k8s, then just put it in the initContainer script
s
sure thanks
will this work for disaster recovery if i manually upload these segments ?will they get downloaded
x
yes, manual should work
the main purpose for deepstore is disater recovery
2 Views
Open in Slack
PreviousNext
Powered by