Hi team, i have 3 minion instances running and below are the configs:

"task": {
      "taskTypeConfigsMap": {
        "UpsertCompactionTask": {
          "schedule": "0 0 */4 ? * *",
          "bufferTimePeriod": "1h",
          "invalidRecordsThresholdPercent": "0",
          "invalidRecordsThresholdCount": "1",
          "validDocIdsType": "SNAPSHOT"
        }
      }
    },

Its taking too much time how can i optimise it?

Here's a few options: 1. reduce the number of sub tasks by reducing the number of segments. this can be achieved by using larger segments. 2. increase the number of minions in your cluster. this will improve task parallelism. 3. try the UpsertCompactMergeTask. it's like UpsertCompactionTask but also merges smaller segments. 4. try commit time compaction. it not only improve segment build times, but also reduces the workload for the minions

UpsertCompactMergeTask will it also compact every segment in table or only merges? or do i need to use both tasks UpsertCompactionTask for compacting each segment and UpsertCompactMergeTask to merge those small segments

UpsertCompactMergeTask compacts and merges. No, you don't need to use both.

UpsertCompactMergeTask how this task chooses which segments to choose for merging and currently i have 1600 segments use below configs and only 1 sub task is created

"taskTypeConfigsMap": {
        "UpsertCompactMergeTask": {
          "schedule": "0 */5 * ? * *",
          "bufferTimePeriod": "1m",
          "outputSegmentMaxSize": "200MB"
        }
      }

and in tasks i can see only 2 segments are considered for merging and its output segment has 75 mb but my limit was 200 mb and segments docs around 1Million but default is 5Million

"maxNumRecordsPerSegment": "5000000",
    "crc": "591908278,1164424178",
    "segmentName": "compacted__iot_advance_session_upsert__17__1762953010500__0,iot_advance_session_upsert__17__122__20251110T1235Z",
    "TASK_ID": "Task_UpsertCompactMergeTask_acc8e121-edc7-403a-abd4-49d466b9d3a0_1762961400458_0",
    "triggeredBy": "CRON_TRIGGER",
    "tableName": "iot_advance_session_upsert_REALTIME",

Let's suppose UpsertCompactMergeTask merged 10 segments and added a compacted segment and then my server went in dead state due to any reason. So after restart is it possible to get segment unavailable error for merged segments?

44 segments unavailable, sampling 10: [iot_last_session_events_test__14__7__20251114T0235Z, iot_last_session_events_test__20__10__20251114T0541Z, iot_last_session_events_test__8__10__20251114T0555Z, iot_last_session_events_test__28__2__20251113T1940Z, iot_last_session_events_test__10__6__20251114T0116Z, iot_last_session_events_test__30__10__20251114T0541Z, iot_last_session_events_test__4__7__20251114T0233Z, iot_last_session_events_test__2__11__20251114T0652Z, iot_last_session_events_test__10__12__20251114T0759Z, iot_last_session_events_test__30__2__20251113T1937Z], with routing policy: strictReplicaGroup [realtime]

UpsertCompactMergeTask doesn't compact it only merges. Any compaction or merging shouldn't affect segment availability as that work is being performed by minions and not servers. If a server dies then upon recovery the segment replacement protocol will complete to replace any older uncompacts/smaller segments.