Hello All. Does pinot have any automatic cleanup of orphan records? Usecase: I have a realtime table with delete enabled, however my stream can contain documentIds that are not available in pinot and these tombstone records are coming as new records when I query the table. Ideally I would like to skip these records altogether instead of adding one more filter to our queries. PS: For these delete records we are just setting the record primary key and deleted flag as true. Thanks.

Is this in the context of upsert feature? Or are you tombstoning yourself as your footnote says? If latter, no way for Pinot to know your intention there.

Its in context of upsert, but the stream has both valid and invalid records ( by invalid I meant those records are not there in pinot). For the valid records its working as expected and records are getting deleted.

Yeah, but how would Pinot know those are invalid? The only way is if Pinot finds a new record for same PK but newer timestamp.

Thanks @Mayank. Just one more follow up question, if we follow this segment compaction, won't that result in smaller segments and reduce query throughput? Also, is there any plans of compacting smaller segments to a larger segment for realtime upsert table? Thanks.

It depends on what’s your upsert ratio. Also, there’s segment merge and rollup for merging smaller segments into larger ones.

If you are referring to https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task. It says its not supported for real time table with upsert. ?

Ah yes. We have discussed cross segment compaction but it is a lot more invovled. cc: @Seunghyun

Got you. I will be on lookout for this feature. Thanks for clarifying on my questions. For my original question, I will go ahead by adding filter to my queries. Regards.