done

Hey folks, During a recent discussion on upsert merge-compact task we had found that we are using segment's creation_meta in a bunch of places today. I feel that's an anti-pattern since that value is independently computed by servers. Instead we should consider using segment zk metadata's creation_time which is unique per logical segment. @Tarun Mavani: will share more details on this soon. Related issue: https://github.com/apache/pinot/issues/15846

For real-time segment, is the creation time the

creation_meta

from server, or when it is created from the controller side?

Agree we should not rely on server side timestamp

The

shouldReplaceOnComparisonTie

method currently accesses segment creation time through the local segment metadata. Currently we don't have access to ZKMetadata but I think that should be easy to inject into the

BasePartitionUpsertMetadataManager

and use the creation time from controller. Using the creation.meta from ZK will make upserts more reliable. I looked at the usage of server creation_meta and it seems

shouldReplaceOnComparisonTie

is the main usage. It is used in some of the minion task as well but that might be okay. During the development, I can reassess if we should change that to controller creation.meta or not. If we agree on changing the

BasePartitionUpsertMetadataManager.shouldReplaceOnComparisonTie

impl to use creation.meta from controller, I can pick this up and share the PR in coming week.

hi @Jackie @Xiaobing I have updated the PR (#16034) based on recent discussions here(#15846). please take a look at the same when you can to help us get unblocked for MergeCompactTask.