https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Pinot team, I’m seeing another UpsertCompactMer...
# troubleshooting
h
Hi Pinot team, I’m seeing another UpsertCompactMergeTask failure on Pinot 1.4.0 The task fails during the upload step with this stacktrace:
Copy code
org.apache.pinot.spi.utils.retry.RetriableOperationException:
org.apache.pinot.common.exception.HttpErrorStatusException:
Got error status code: 400 (Bad Request) with reason: ""
while sending request:
/segments?enableParallelPushProtection=true&tableName=success_board&tableType=REALTIME
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63)
	at org.apache.pinot.plugin.minion.tasks.SegmentConversionUtils.uploadSegment(SegmentConversionUtils.java:140)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.pushSegment(BaseMultipleSegmentsConversionExecutor.java:538)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:316)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:84)
This looks similar to another UpsertCompactMergeTask failure I had before: • task generation succeeds • source segments seem valid • failure happens when uploading the merged segment back to the controller • the request includes enableParallelPushProtection=true Questions: 1. What are the common reasons for the controller to return 400 Bad Request on /segments?enableParallelPushProtection=true... for UpsertCompactMergeTask ? 2. Does this usually indicate: • parallel push conflict, • segment lineage changed while the task was running, • or another controller-side validation failure ? 3. Is there a recommended way to debug the exact controller-side reason when the response body is empty ? 4. Are there recommended limits for maxNumSegmentsPerTask / maxNumRecordsPerTask to reduce the chance of this happening? Thanks!
s
Do you see logs in the controller for the segment around the time of failure? That should indicate something.
h
@Mathieu Alexandre
m
👋 i've just found this :
Copy code
2026/04/13 13:29:39.811 WARN [TablesResource] [grizzly-http-server-8] Found that validDocIds is missing for segments [our_table__8__26__20230605T0837Z,...] 
while processing validDocIdsMetadata for table our_table_REALTIME while reading the validDocIds with validDocIdType SNAPSHOT.
h
table config
Copy code
"task": {
      "taskTypeConfigsMap": {
        "UpsertCompactMergeTask": {
          "schedule": "0 0 * ? * *",
          "bufferTimePeriod": "0d",
          "maxNumRecordsPerSegment": "5000000",
          "maxNumRecordsPerTask": "20000000",
          "maxNumSegmentsPerTask": "300",
          "outputSegmentMaxSize": "200MB",
          "numSegmentsBatchPerServerRequest": "500"
        }
      }
    },

    "upsertConfig": {
      "consistencyMode": "NONE",
      "allowPartialUpsertConsumptionDuringCommit": false,
      "snapshot": "ENABLE",
      "mode": "FULL",
      "hashFunction": "NONE",
      "defaultPartialUpsertStrategy": "OVERWRITE",
      "dropOutOfOrderRecord": false,
      "preload": "DEFAULT",
      "metadataTTL": 0,
      "deletedKeysTTL": 0,
      "enableDeletedKeysCompactionConsistency": false,
      "upsertViewRefreshIntervalMs": 3000,
      "newSegmentTrackingTimeMs": 10000,
      "enableSnapshot": true,
      "enablePreload": false
    },
Controller with
Copy code
controller.task.scheduler.enabled=true
We tried to : • restart server and controller • pause/resume consumption Nothing changed
image.png,image.png
Just to add more context from our side: we have the same overall setup and configuration working on another environment with a much smaller data volume. So: • same kind of REALTIME upsert table • same enableSnapshot=true • same UpsertCompactMergeTask setup • same scheduling model • same general Pinot architecture
s
Look around for any more logs that have 
grizzly-http-server-8
(this is the thread that processed the rest request to the controller at the time), then start looking around the source code in controller. start with the 
TableResource
class.
h
Just to close the loop on this, we did not isolate the exact controller-side root cause of the 400 Bad Request, but the issue was effectively resolved by making the UpsertCompactMergeTask configuration less aggressive. In our case, the original settings were too aggressive (large number of source segments / too much work per task). After reducing the task parameters, the merges started working correctly. So the practical takeaway for us is: • the issue does not seem to come from snapshots or task generation • it happens during the upload/push of the merged segment • and lowering the task aggressiveness fixes it We still do not know the exact controller-side validation that was causing the 400, but the mitigation is clear on our side.
3 Views
Open in Slack
PreviousNext
Powered by