Hi Team we are using realtime tables with upserts ...
# troubleshootingu
Utsav Jain
10/29/2025, 5:15 AMHi Team we are using realtime tables with upserts enabled and the ttl window is 12 hrs
To purge any stale record coming after ttl window and to avoid runnig queries with distinct we enabled segment compaction job
But we are seeing issues while running it with full accuracy like few of the older segments are never considered for compaction which is resulting in inaccurate numbers while running the query
Can you please help in understanding the causes of such cases ?
Utsav Jain
10/29/2025, 6:28 AMcc: @Abhigyan Bharati @Mohd Sarfaraz
Utsav Jain
10/29/2025, 8:42 AMUpsert job config
Copy code
"taskTypeConfigsMap": {
"UpsertCompactionTask": {
"schedule": "0 0 * ? * *",
"bufferTimePeriod": "0h",
"invalidRecordsThresholdPercent": "1",
"tableMaxNumTasks": "40",
"validDocIdsType": "SNAPSHOT"
}
}Utsav Jain
10/30/2025, 3:17 PM@Xiang Fu @Mayank can you please help here
x
Xiang Fu
10/30/2025, 8:00 PM
with upsert why you still need distinct?
Xiang Fu
10/30/2025, 8:02 PM
did you configure SNAPSHOT for the upsert table
Xiang Fu
10/30/2025, 8:02 PM
that is required for the UpsertCompactionTask
u
Utsav Jain
10/31/2025, 5:42 AMenableSnapshot property right ? where it needs to be configured exactly
FYI we are able to run UpsertCompactionTask for most of the segments but i guess few of the segments it is not running
Utsav Jain
10/31/2025, 5:44 AMAlso we dont want to run distinct with upserts enabled as compaction should ideally remove all duplicate entries, but because it is not working and skipping segments for us we have to run using distinct
That is why we wanted some help as to how we can tune the config further to make it effective