RAM - 128 GB - currently using it in single server with medium default configuration

I want to see the configs that show up when the service restarts

After our chat I tried the sketch on 25.0. I got the error you did and added amx on disk storage to both broker and historical and am able to run the query.

which flavor and version of linux are you on?

& instance type m5.8xlarge

2023-01-16T183406,064 DEBUG [sql[7d3f2546-055a-4327-82da-144d09c92a12]] org.apache.druid.query.groupby.GroupByQueryConfig - Override config for GroupBy query 5234d353-6023-4ee3-9f35-616298fc7766 - GroupByQueryConfig{defaultStrategy='v2', singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=3000000000, maxOnDiskStorage=10000000000, defaultOnDiskStorage=10000000000, forcePushDownLimit=false, forceHashAggregation=false, intermediateCombineDegree=8, numParallelCombineThreads=1, vectorize=true, forcePushDownNestedQuery=false, enableMultiValueUnnesting=true} 2023-01-16T183406,065 DEBUG [ForkJoinPool-1-worker-1] org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence - Computed parallel tasks: [1]; ForkJoinPool details - sequence parallelism: [16] active threads: [0] running threads: [1] queued submissions: [0] queued tasks: [0] pool parallelism: [24] pool size: [24] steal count: [48] 2023-01-16T183406,065 DEBUG [ForkJoinPool-1-worker-1] org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence - Input sequence count (2) or available parallel merge task count (1) too small to perform parallel merge-combine, performing serially with a single merge-combine task 2023-01-16T183409,914 DEBUG [HttpClient-Netty-Worker-47] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://localhost:8101/druid/v2/] messageReceived: DefaultHttpResponse(chunked: false) HTTP/1.1 400 Bad Request Date: Mon, 16 Jan 2023 183406 GMT Content-Type: application/x-jackson-smile Content-Length: 591 2023-01-16T183409,914 DEBUG [HttpClient-Netty-Worker-47] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://localhost:8101/druid/v2/] Got response: 400 Bad Request 2023-01-16T183409,914 DEBUG [HttpClient-Netty-Worker-47] org.apache.druid.client.DirectDruidClient - Initial response from url[http://localhost:8101/druid/v2/] for queryId[5234d353-6023-4ee3-9f35-616298fc7766] 2023-01-16T183409,914 TRACE [HttpClient-Netty-Worker-47] org.apache.druid.client.DirectDruidClient - Got a response from [http://localhost:8101/druid/v2/] for query ID[5234d353-6023-4ee3-9f35-616298fc7766], subquery ID[null] 2023-01-16T183409,915 DEBUG [HttpClient-Netty-Worker-47] org.apache.druid.client.DirectDruidClient - Completed queryId[5234d353-6023-4ee3-9f35-616298fc7766] request to url[http://localhost:8101/druid/v2/] with 591 bytes returned in 3,850 millis [153.506494 b/s]. 2023-01-16T183410,124 DEBUG [HttpClient-Netty-Worker-46] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://localhost:8083/druid/v2/] messageReceived: DefaultHttpResponse(chunked: true) HTTP/1.1 200 OK Date: Mon, 16 Jan 2023 183406 GMT Content-Type: application/x-jackson-smile X-Druid-Query-Id: 5234d353-6023-4ee3-9f35-616298fc7766 X-Druid-Response-Context: {} Vary: Accept-Encoding, User-Agent Transfer-Encoding: chunked 2023-01-16T183410,124 DEBUG [HttpClient-Netty-Worker-46] org.apache.druid.java.util.http.client.NettyHttpClient - [POST http://localhost:8083/druid/v2/] Got response: 200 OK 2023-01-16T183410,124 DEBUG [HttpClient-Netty-Worker-46] org.apache.druid.client.DirectDruidClient - Initial response from url[http://lo

Post enabling the debug logs observing Bad Request - not sure if this causing the following error message later on

2023-01-16T183415,709 WARN [sql[7d3f2546-055a-4327-82da-144d09c92a12]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [5234d353-6023-4ee3-9f35-616298fc7766] org.apache.druid.query.ResourceLimitExceededException: Not enough merge buffer memory to execute this query. Try enabling disk spilling by setting druid.query.groupBy.maxOnDiskStorage to an amount of bytes available on your machine for on-disk scratch files. Or, if you have additional off-heap memory available, consider increasing druid.processing.buffer.sizeBytes. See https://druid.apache.org/docs/latest/querying/groupbyquery.html#memory-tuning-and-resource-limits for details. at org.apache.druid.client.JsonParserIterator.convertException(JsonParserIterator.java:261) ~[druid-server-24.0.2.jar:24.0.2] at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:183) ~[druid-server-24.0.2.jar:24.0.2] at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:93) ~[druid-server-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:70) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:879) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:929) ~[druid-core-24.0.2.jar:24.0.2] at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3118) ~[?:?] at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:918) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:1025) ~[druid-core-24.0.2.jar:24.0.2] at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:732) ~[druid-core-24.0.2.jar:24.0.2] at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) ~[?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) ~[?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) ~[?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) ~[?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) ~[?:?] 2023-01-16T183415,709 DEBUG [qtp1922735165-189] org.eclipse.jetty.server.HttpOutput - write(array HeapByteBuffer@3d648a63[p=0,l=573,c=573,r=573]={<<<{"error":"Resource limit ...host":"localhost:8101"}>>>}) 2023-01-16T183415,710 DEBUG [qtp1922735165-189] org.eclipse.jetty.server.HttpOutput - write(array) s=CLOSING,api=BLOCKED,sc=false,e=null last=true agg=false flush=true async=false, len=573 null 2023-01-16T183415,710 DEBUG [qtp1922735165-189] org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor - org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@1c8bb926 exclude by status 400 2023-01-16T183415,710 DEBUG [qtp1922735165-189] org.eclipse.jetty.server.HttpChannel - sendResponse info=null content=HeapByteBuffer@6065ac09[p=0,l=573,c=573,r=573]={<<<{"error":"Resource limit ...host":"localhost:8101"}>>>} complete=true committing=true callback=Blocker@b793f52{null} 2023-01-16T183415,710 DEBUG [qtp1922735165-189] org.eclipse.jetty.server.HttpChannel - COMMIT for /druid/v2/sql on HttpChannelOverHttp@60958817{s=HttpChannelState@6f3b43{s=HANDLING rs=BLOCKING os=COMMITTED is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=HANDLING,uri=//localhost:8082/druid/v2/sql,age=9686} 4*00 Bad Request HTTP/1.1* Date: Mon, 16 Jan 2023 183406 GMT Content-Length: 573 Content-Type: application/json X-Druid-SQL-Query-Id: 7d3f2546-055a-4327-82da-144d09c92a12

@Vijay Narayanan thanks a lot for ur help.. post making the configuration changes across the broker, middlemanager, historical able to get the response