This message was deleted.

buffer size cannot be increased as the is 2 GB. You can try reducing max rows in memory and also the number of subtasks. What is the segment granularity? Can you paste the ingestion spec here?

Thanks for reply! I tried to reduce maxRowsInMemory 150000->100000 but didn’t work. below is tuningConfig

"tuningConfig": {
      "type": "hadoop",
      "workingPath": null,
      "partitionsSpec": {
        "type": "hashed",
        "numShards": null,
        "partitionDimensions": [],
        "partitionFunction": "murmur3_32_abs",
        "maxRowsPerSegment": 3000000
      },
      "shardSpecs": {},
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "appendableIndexSpec": {
        "type": "onheap"
      },
      "maxRowsInMemory": 100000,
      "maxBytesInMemory": 0,
      "leaveIntermediate": false,
      "cleanupOnFailure": true,
      "overwriteFiles": false,
      "ignoreInvalidRows": false,
      "jobProperties": {
        "mapreduce.job.queuename": "tiara",
        "mapreduce.job.classloader": "true",
        "mapreduce.map.memory.mb": "2048",
        "mapreduce.map.java.opts": "-Xmx1638m -Djava.net.preferIPv4Stack=true -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps",
        "mapreduce.reduce.cpu.vcores": "1",
        "mapreduce.reduce.memory.mb": "20480",
        "mapreduce.reduce.java.opts": "-Xmx12288m -XX:MaxDirectMemorySize=6g -Djava.net.preferIPv4Stack=true -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
      },
      "combineText": false,
      "useCombiner": false,
      "numBackgroundPersistThreads": 0,
      "forceExtendableShardSpecs": false,
      "useExplicitVersion": false,
      "allowedHadoopPrefix": [],
      "logParseExceptions": false,
      "maxParseExceptions": 0,
      "useYarnRMJobStatusFallback": true,
      "awaitSegmentAvailabilityTimeoutMillis": 0
    }
  },
  "hadoopDependencyCoordinates": null,
  "classpathPrefix": null,
  "context": {
    "forceTimeChunkLock": true,
    "useLineageBasedSegmentAllocation": true
  }
}

try reducing max rows per segment. Are indexing all the string fields?

@Vijay Narayanan hey I tried to reduce

maxRowsPerSegment

and succeeded! Seems like each segments were getting bigger and bigger. you saved my life thanks!

👍remember to turn on auto compaction so that you don't end up with too many small segments.

glad you got it figured out!

@Gian Merlino @Vijay Narayanan we are running into a similar issue. Your clarification here is helpful. One question though- our tasks are eventually succeeding. Why would it happen if the columns size is the issue?

You mean you see some attempt failures due to

Asked to add buffers[XYZ] larger than configured max[2,147,483,647]

which then eventually succeed after a retry?