Hi Pinot community, I’m trying to clarify the expected behavior of

UpsertCompactMergeTask

with

validDocIdsType=SNAPSHOT

vs

IN_MEMORY

. We have a REALTIME upsert table on Pinot 1.4.0 with:

"upsertConfig": {
  "mode": "FULL",
  "snapshot": "ENABLE",
  "enableSnapshot": true
}

At the beginning of May, we updated this table config to add

UpsertCompactMergeTask

. Since then, the task has worked partially: we reduced the table from around 20K segments to around 3K segments. However, a lot of old completed segments still cannot be merged. From what we see, the task is not able to retrieve their

validDocIds

using

SNAPSHOT

. When triggering the task manually, the controller logs show something like:

Retrieved validDocIds metadata for 15 segments from 96 server requests
Finished generating 0 tasks configs

But we still have many old completed segments from 2023 to 2025 that look like they should be compactable. I tested the

validDocIdsMetadata

endpoint on one old segment:

validDocIdsType=SNAPSHOT -> []
validDocIdsType=IN_MEMORY -> returns metadata correctly
validDocIdsType=IN_MEMORY_WITH_DELETE -> []

Example with `IN_MEMORY`:

{
  "segmentName": "daily_activity__2__0__20230427T1137Z",
  "totalValidDocs": 4,
  "totalInvalidDocs": 0,
  "totalDocs": 4,
  "segmentCrc": "2205091866",
  "validDocIdsType": "IN_MEMORY",
  "serverStatus": "GOOD"
}

So it looks like the upsert state exists in memory on the realtime servers, but many historical segments do not have

validDocIds

snapshots. From the docs, I understand that: •

SNAPSHOT

is the default

validDocIdsType

for

UpsertCompactMergeTask

•

SNAPSHOT

requires

enableSnapshot=true

• snapshots are created/refreshed during the normal upsert / segment commit lifecycle • there does not seem to be a documented command/API to backfill

validDocIds

snapshots for all already-committed historical segments My questions are: 1. Is this expected behavior for old completed segments that existed before we added the compaction task / before snapshots were generated for them? 2. Is there any supported way to force Pinot to generate

validDocIds

snapshots for all existing completed segments? 3. If not, is the recommended approach to run UpsertCompactMergeTask temporarily with:"validDocIdsType": "IN_MEMORY" to compact the remaining historical backlog, and then switch back to

SNAPSHOT

for steady state? 4. Are there any risks or caveats with using

IN_MEMORY

for this remaining backlog, assuming the servers are stable and the table is not being restarted/rebalanced? Thanks!

@Kartik Khare ^^

@Subbu Subramaniam @Xiang Fu @Jackie any idea ?

1. Is this expected behavior for old completed segments that existed before we added the compaction task / before snapshots were generated for them? As far as i know, older exiting segments , are still eligible for compaction , At-least thats what this Example part of Pinot Doc imply , For this to work you defiantly need to have Snapshot present for these segments (if

validDocIdsType=SNAPSHOT

), You need to check why snapshots are missing for these old segments 2. Is there any supported way to force Pinot to generate

validDocIds

snapshots for all existing completed segments? Currently no. There is a specific Minion Job for this in the StarTree version. However, in Open Source Pinot, if snapshots are enabled, snapshots for existing segments (including older ones) are supposed to be taken or refreshed automatically when new consuming segments are created. 3. If not, is the recommended approach to run UpsertCompactMergeTask temporarily with:"validDocIdsType": "IN_MEMORY" to compact the remaining historical backlog, and then switch back to

SNAPSHOT

for steady state? I don't know if there is a recommended approach , But I do not see any issue with this approach. 4, Are there any risks or caveats with using

IN_MEMORY

for this remaining backlog, assuming the servers are stable and the table is not being restarted/rebalanced? There are no caveats, provided the servers remain perfectly stable (no restarts, crashes, or partition rebalances) while the Minion task is actively reading the in-memory state. At least this is what I know based on my understanding. If there are any misleading points, please correct !

1. Age alone should not make them ineligible. If validDocIdsType=SNAPSHOT, then the task needs snapshot files for those old segments. If SNAPSHOT returns [] but IN_MEMORY returns metadata, the real question is why those snapshots were never created or refreshed. 2. No support 3.

validDocIdsType

is wired for

UpsertCompactionTask

, but

UpsertCompactMergeTask

hardcodes

SNAPSHOT

in both task generation and execution.Setting this

"validDocIdsType": "IN_MEMORY"

can affect UpsertCompactionTask, but not compact-merge. 4. There are caveats with

IN_MEMORY

, but they are operational rather than semantic. It is generally OK only if servers stay stable while the task runs: no restarts, crashes, reloads, rebalances, or segment replacement, and server status/CRC are good. Otherwise the task can read inconsistent or unavailable in-memory validDocIds.

Thanks @Badhusha Muhammed @Xiang Fu. Based on your answer, we understand that: 1.

UpsertCompactMergeTask

requires

SNAPSHOT

and does not honor

validDocIdsType=IN_MEMORY

. 2. There is no supported way to backfill/generate

validDocIds

snapshots for already committed historical segments. 3. If

SNAPSHOT

returns empty for a segment,

UpsertCompactMergeTask

cannot process it, even if

IN_MEMORY

returns valid metadata. 4.

IN_MEMORY

could only be used with

UpsertCompactionTask

, with the operational caveats you mentioned. We checked a few of the remaining segments that are not being compacted. They look healthy from ZK/deepstore/server metadata:

segment.realtime.status = DONE
segment.download.url = present
segment.crc = matches the IN_MEMORY segmentCrc
serverStatus = GOOD

Examples:

daily_activity__2__0__20230427T1137Z
SNAPSHOT -> []
IN_MEMORY -> totalValidDocs=4, totalInvalidDocs=0, totalDocs=4, CRC matches ZK, serverStatus=GOOD

daily_activity__2__1001__20240710T2124Z
SNAPSHOT -> []
IN_MEMORY -> totalValidDocs=1, totalInvalidDocs=0, totalDocs=1, CRC matches ZK, serverStatus=GOOD

daily_activity__9__1002__20240710T1853Z
SNAPSHOT -> []
IN_MEMORY -> totalValidDocs=1, totalInvalidDocs=0, totalDocs=1, CRC matches ZK, serverStatus=GOOD

One pattern we noticed is that these segments have

totalInvalidDocs=0

. Do you have any recommendation on how we should diagnose why the snapshots were never created or refreshed for these segments? More specifically: 1. Is it expected that segments with

totalInvalidDocs=0

may not have

validDocIds

snapshot files? 2. Are there specific server-side files, directories, metrics, or logs we should inspect to confirm whether the snapshots were skipped, never created, deleted, or failed to load? Thanks again for the clarification.

if

totalInvalidDocs=0

then it means all the rows in the segments are valid, so compaction tasks shouldn't touch them and

validDocIds

should always match all

@Xiang Fu The

UpsertCompactMergeTask

Can pick up this segments, As this might be eligible for merging right .?

I checked the

UpsertCompactMergeTask

docs again, and I understand that for

UpsertCompactMergeTask

,

totalInvalidDocs=0

should not exclude a segment from merging. The task can still merge multiple small segments and compact them during the merge, as long as the grouping is feasible and the total valid records stay within the configured limits. So I think our remaining issue is not that these segments have

totalInvalidDocs=0

.

correct, you need to check the merge logic for segment selection

Thanks. Just to clarify, I think the merge selection logic itself is working. The minions have already merged a large number of segments on this table. After enabling

UpsertCompactMergeTask

at the beginning of May, the segment count went down from around 20K to around 3K. The remaining issue I’m trying to understand is more specific: many of the remaining completed segments do not have

validDocIds

available through

SNAPSHOT

, but they do have valid metadata through

IN_MEMORY

. For example:

daily_activity__2__0__20230427T1137Z
DONE, download.url present, CRC matches
SNAPSHOT -> []
IN_MEMORY -> totalValidDocs=4, totalInvalidDocs=0, totalDocs=4, serverStatus=GOOD

daily_activity__2__1001__20240710T2124Z
DONE, download.url present, CRC matches
SNAPSHOT -> []
IN_MEMORY -> totalValidDocs=1, totalInvalidDocs=0, totalDocs=1, serverStatus=GOOD

daily_activity__9__1002__20240710T1853Z
DONE, download.url present, CRC matches
SNAPSHOT -> []
IN_MEMORY -> totalValidDocs=1, totalInvalidDocs=0, totalDocs=1, serverStatus=GOOD

So I don’t think the issue is that these segments are too old, too large, not continuous, or not mergeable by the merge selection logic. The issue seems to be that

UpsertCompactMergeTask

requires

SNAPSHOT

, and these segments only have

IN_MEMORY

validDocIds. What I’m trying to diagnose is specifically why the validDocIds snapshots were never created/refreshed for those completed segments, even though: •

enableSnapshot=true

• segments are DONE • download URL is present • CRC matches • serverStatus is GOOD • IN_MEMORY validDocIds are available Is there a recommended way to diagnose why a segment has validDocIds in memory but no validDocIds snapshot? For example, should we inspect specific files on the server local segment directory, specific logs, or specific metrics related to upsert snapshot creation/loading?

The likely underlying reasons are: 1.

enableSnapshot

was enabled in table config, but the server table/upsert manager was not restarted/reinitialized. Pinot copies that setting into

UpsertContext

at table manager init. You may need to disable/enable the table or just restart all servers 2. No snapshot cycle ran after those immutable segments were loaded. Snapshot is triggered from consuming segment lifecycle, especially when a new consuming segment starts, not by a public “snapshot all now” controller job. 3. The snapshot cycle skipped the segment because it could not acquire the segment lock or hit an exception. Logs would show:

Could not get segmentLock to take snapshot
Caught exception while taking snapshot
Missed taking snapshot

4. The segment was not in

_updatedSegmentsSinceLastSnapshot

when the snapshot pass ran. The snapshot code only writes segments marked as changed/added/replaced since the previous snapshot pass. 5. Snapshot was previously disabled on that server; in that mode Pinot deletes snapshot files during segment add.

Hey there! The most probable reasons why the

SNAPSHOT

is taking fewer segments in latest upgrade could be because all the replicas didn't have snapshots and didn't follow the consensus which we didn't check before. So we made bunch of fixes recently to reduce the contention and for all the replicas to have snapshots by fixing it in this PR: github.com/apache/pinot/pull/19028 But this is not yet included in the Apache Pinot 1.5.1