Is there a quick way we can convert realtime segme...
# troubleshootinge
Elon
08/19/2020, 10:38 PMIs there a quick way we can convert realtime segments to offline segments? Is there any benefit to doing that since the realtime segment is created with star tree, inverted, sorted and text indexes?
n
Neha Pawar
08/19/2020, 10:41 PM
What’s the motivation for wanting that?
We have this project ongoing to move realtime segments to offline table: https://docs.google.com/document/d/1-e_9aHQB4HXS38ONtofdxNvMsGmAoYfSnc2LP88MbIc/edit#
Neha Pawar
08/19/2020, 10:41 PM
would be helpful if you could take a look at the doc and make sure this would cover what you are looking for
Neha Pawar
08/19/2020, 10:42 PM
as of now, there’s no way. you’d have to download segments, convert to raw data, upload to offline
e
Elon
08/19/2020, 11:03 PMSounds good. Thanks for the link!
Elon
08/19/2020, 11:04 PMWhat do you recommend is the optimal segment size or rowcount?
n
Neha Pawar
08/19/2020, 11:07 PM
in realtime we recommend 150M as optimal segment size
Neha Pawar
08/19/2020, 11:07 PM
if you’re uploading back to same table’s OFFLINE component, you need to be careful about the time boundary for the hybrid table
e
Elon
08/19/2020, 11:08 PMDo you mean be careful If there is overlap in the time intervals, it could delete records in one of the realtime or offline segments?
n
Neha Pawar
08/19/2020, 11:12 PM
yes if there is overlap. be cognizant that the query will be split as such: https://docs.pinot.apache.org/basics/components/broker
Neha Pawar
08/19/2020, 11:12 PM
what are you trying to achieve?
Neha Pawar
08/19/2020, 11:13 PM
nothing will be deleted.
e
Elon
08/20/2020, 7:26 AMJust want to get a better understanding, we want to replace realtime segments with sanitized offline segments, or just upload segments. I think I have enough to go on now, thanks for the info!