https://pinot.apache.org/ logo
Join Slack
Powered by
Hello everyone , I am looking into segmentMergeCon...
# troubleshooting
d
Hello everyone , I am looking into segmentMergeConfig and following this doc :- https://docs.google.com/document/d/1zoklHjbli-HIy0JAiBITABNsBOuC2jFthcjwGqsFVFQ/edit#heading=h.3eldj09ucqrn and found that i am not able to update realtime config with this ‘segmentMergeConfig’ parameter. Can anyone help on this issue?
l
Segment merge works only for OFFLINE tables
d
ok
l
As you are already using RealtimeToOffline task, that should take care of compaction and merge as well.
Can you explain your use-case/requirement.
d
if there are multiple number of segment generated in realtime table within the period .Can we not merge segment as well?
when we are using realtimeToOffline flow and the kafka push data with multiple segments before moving into offline table. Can we merge multiple segments into one in realtime flow?
l
Can we not merge segment as well?
Yes. this is handled in RealtimeToOffline task. RealtimeToOffline task compacts/merges 
m
REALTIME segments into 
n
OFFLINE segments. `n`Depends on max records, bucket period, buffer period.
Can we merge multiple segments into one in realtime flow?
No. Not in realtime flow.
Realtime segments are immutable so far. However, there work in progress make them mutable
d
ok
l
image.png
j
Curious where do you find the link to this doc? Here is the up-to-date doc for this feature, and the implementation is still WIP
The end goal is to support merge for both offline and realtime table
👍 1
Open in Slack
PreviousNext
Powered by