Hi Team We are using realtime tables with upsert in Pinot Ho Apache Pinot #troubleshooting

Hi Team, We are using realtime tables with upsert ...

Rahul Sharma

10/22/2025, 7:56 PM

Hi Team, We are using realtime tables with upsert in Pinot. However, since Pinot does not actually delete old records, we need to schedule Minion compaction tasks to handle this. I added the configuration below to my table. Now, in the Task Manager, the

upsertCompactionTask

is visible, but its task configuration is empty. As a result, compaction is not working, and the number of records in my table remains the same. Can anyone please help? Conf:

Copy code

"task": {
      "taskTypeConfigsMap": {
        "UpsertCompactionTask": {
          "schedule": "0 */5 * ? * *",
          "bufferTimePeriod": "0d",
          "invalidRecordsThresholdPercent": "10",
          "invalidRecordsThresholdCount": "1000"
        }
      }
    },

Rahul Sharma

10/22/2025, 7:59 PM

@Shounak Kulkarni @Mayank @Xiang Fu can you please help here

Xiang Fu

10/22/2025, 8:42 PM

check controller logs if the UpsertCompactionTaskGenerator gives empty task output

Rahul Sharma

10/23/2025, 3:22 AM

Yes these are the logs

Copy code

[DefaultQuartzScheduler_Worker-3] Finished generating 0 tasks configs for table: customer_requests_REALTIME

Xiang Fu

10/23/2025, 8:03 AM

ok, then it means no segment meets the criteria set in the table config for the compaction task

Rahul Sharma

10/23/2025, 8:52 AM

I am performing upserts on an ID, and there are a total of 400 IDs, but the total number of documents has grown to 2.5 million. How is it possible that this doesn’t meet the criteria?

Rahul Sharma

10/23/2025, 3:46 PM

It wasn't working because

validDocIdsType

was set to SNAPSHOT and i enabled snapshot today so no old segment would have snapshot. I changed it to IN_MEMORY and it worked. Now have again changed to SNAPSHOT as it should work for new segments now.

👍 1

Xiang Fu

10/23/2025, 5:19 PM

got it. have you enabled the

enableSnapshot

in the table upsert config?

Xiang Fu

10/23/2025, 5:20 PM

Copy code

validDocIdsType (Optional) Specifies the source of validDocIds to fetch when running the data compaction. The valid types are SNAPSHOT, IN_MEMORY, IN_MEMORY_WITH_DELETE

SNAPSHOT: Default validDocIds type. This indicates that the validDocIds bitmap is loaded from the snapshot from the Pinot segment. UpsertConfig's enableSnapshot must be enabled for this type.

IN_MEMORY: This indicates that the validDocIds bitmap is loaded from the real-time server's in-memory. 

IN_MEMORY_WITH_DELETE: This indicates that the validDocIds bitmap is read from the real-time server's in-memory. The valid document ids here does take account into the deleted records. UpsertConfig's deleteRecordColumn must be provided for this type.

Rahul Sharma

10/23/2025, 6:33 PM

Yes, i have enabled it now.

👍 1

Open in Slack

Previous Next