Hey, quick question: we have realtime segment marked as completed and we would like to move it to a offline table however the endpoint to download the segment (

get /segments/{tableName}/{segmentName}

)is trying to fetch it from the deep store. I was just thinking of downloading it and upload it on the offline table directly, how could i achieve this ? Thanks

Yeah i saw that but i don't really want the minion to rebuild the segment, i just want to move them as-is

I’m not sure - the tricky part is that the offline table might not have the same partitioning / sorting / indexing as the realtime table, and the

RealtimeToOfflineSegmentsTask

handles that generically for you - by simply moving the segments as-is, you are kind of shoehorning yourself into never diverging the offline table.

Well i configured the realtime table the exact same as the offline so i should be fine right ?

For the short-term, yes

What would be the problem in the long term ? I mean if i have an issue i can just re-index the segment and re-upload it ?

It’s not an approach I would put into production, though

From my comprehension the only difference between realtime and offline when a segment is completed would be that the realtime stores it locally but the offline does it in the deepstore

Realtime tables also push segments to the deep store

Hmmm, thats surely something i missed

https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=103090634#BypassingdeepstorerequirementforRealtimesegmentcompletion-Principlesofabettersolution:

Thanks, something that i haven't mentionned is that my stream are high level. I'm seeing in the task code that it only works with low level

See this as well: https://docs.pinot.apache.org/basics/components/table#segments

Thanks anyway, you were very helpful

We don't recommend or maintain high level any more. Any particular reason you are using high level?

@Neha Pawar We are currently using GCP's pubsub system for pinot and it doesnt have any "partition" system