Hello, is using the `startReplaceSegments` and `en...
# troubleshooting
e
Hello, is using the
startReplaceSegments
and
endReplaceSegments
endpoints the correct and most robust approach for doing atomic transactions for offline tables? For example, if my table currently contains segments s1, s2, and s3, and I need to do the following in an all or nothing transaction: • Insert new segments s4 and s5. • Replace existing segment s3 with a new segment s6.
m
@Manish ^^
m
Yes the 2 apis can be used to atomically switch one set of segments with another set of segments. Eg: [s3] -> [s4, s5, s6] -> This will atomically switch s3 to s4, s5, s6 startReplace gives a lineageId (marks the beginning of transaction) and endReplace uses that lineage to close the transaction. Curious about your usecase and from where you'll be using these apis
e
Yeah so we have a daily batch process that moves data from bigquery to offline pinot tables. This is an incremental process, but sometimes data for a previously inserted day can be mutated. If that is the case, then that segment must be entirely replaced with a new segment containing the updated data for that day
Was also wondering about the consistent data push option that seems to have been merged here: https://github.com/apache/pinot/pull/14139 Is this a valid approach as well?
m
Hey Emerson, yes the pr adds consistentPush support, but thats to replace the entire table atomically and not specific segments that you want to, here. Yes using the 2 apis is a valid approach to replace specific list of segments with new segments.
👍 1
e
Ah, thanks. So if we call
startReplaceSegments
to swap s1 and s2 with s3 and s4, and something fails during the ingestion of s3 or s4, do we need to manually call
revertReplaceSegments
?
@Manish Hey just to cover all our bases, do any of the minion ingestion jobs support the atomic swapping of specific segments for APPEND tables? From what I can see tell they do not since they append a UUID to ingested segment names. Thanks again!
m
Sorry about the delay. 1. Regarding the failure - "So if we call
startReplaceSegments
to swap s1 and s2 with s3 and s4" - We will do the ingestion of s3, s4 and the metadata push will happen after startReplaceSegments is called. Could you look at the forceCleanUp up param of ``startReplaceSegments` . That'll auto clean up failed lineages incase the segment replacement fails, when
startReplaceSegments
is called subsequently. 2. For consistent push, we do need unique segment names (unique uuid). Could you look at this
SegmentMetadataPushJobRunner
(pr reference), that gives an example of how stand alone job uses consistent push feature.