https://linen.dev logo
Join Slack
Powered by
I recently switched from MM to MM-less in Kubernet...
# troubleshooting
j
I recently switched from MM to MM-less in Kubernetes and I'm seeing a weird spike in Task Slots every hour. We have a job that runs hourly and creates 5 query_controllers each with 9 query_workers for a total of 50 tasks. When we were using MiddleManager, the metric 
/taskSlot/used/count
would report 10 but on MM-less it is reporting 50. I can't tell if this saturated our available slots in the past but it seems to be saturating all of our available slots now. Is this a change in how MM-less works? Our MM instance ran 33.0.0 and our MM-less instance is running 37.0.0. How many k8s pods am I going to need to scale this up?!?
✅ 1
Here's a sample query (I've obfuscated column names but the query is otherwise identical):
Copy code
INSERT INTO
"test"

SELECT
  TIME_FLOOR(__time, 'PT1H') AS __time,
  a, b, c, d, e, f, g, h, i, j, k,
  i + j + k AS total,
  l, m,
  COUNT(*) AS m0,
  COUNT(*) FILTER (WHERE en = 'Foo') AS Foo,
  COUNT(*) FILTER (WHERE en = 'Bar') AS Bar

FROM
  "ds"
WHERE
  __time >= TIMESTAMP '2026-08-21 23:00:00'
  AND __time < TIMESTAMP '2026-08-22 00:00:00'

GROUP BY
  1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15

PARTITIONED BY DAY
CLUSTERED BY d, b, c, a, e, f
Copy code
"context": {
    "__user": "allowAll",
    "appendToExisting": "true",
    "finalize": true,
    "finalizeAggregations": true,
    "maxNumTasks": 10,
    "maxParseExceptions": 0,
    "partitionedBy": "day",
    "queryGranularity": "hour",
    "queryId": "ds_task_scheduled__2026_08_22T00_15_00_00_00",
    "rollup": "true",
    "rowBasedFrameType": 19,
    "sqlInsertSegmentGranularity": "\"DAY\"",
    "sqlQueryId": "ef4de6a3-1765-454a-b79a-fad6f71854c2",
    "startTime": "2026-08-22T01:23:00.378Z",
    "timestampResultField": "d0",
    "timestampResultFieldGranularity": "\"HOUR\"",
    "timestampResultFieldInOriginalDimensions": 0,
    "windowFunctionOperatorTransformation": true
  }
Oddly, the default split is on 1,000 files or 1 GiB. However, this job was only 311 files and 681 MB so I'm a bit surprised it split into 10 workers...
Ah hah!! 
taskAssignment
- Determines how many tasks to use. Possible values include: • max: Default mode. Uses as many tasks as possible, up to maxNumTasks. • auto: When file sizes can be determined through directory listing (for example: local files, S3, GCS, HDFS) uses as few tasks as possible without exceeding 512 MiB or 10,000 files per task, unless exceeding these limits is necessary to stay within maxNumTasks. When calculating the size of files, the weighted size is used, which considers the file format and compression format used if any. When file sizes cannot be determined through directory listing (for example: http), behaves the same as max.
Is the task slot being used while the pod is being torn down by k8s or something?
I also get a lot of these in the logs after the job completes successfully:
Copy code
2026-08-22T01:23:33,097 ERROR [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcheroverlord] org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - Error while watching role[OVERLORD]
java.lang.RuntimeException: IO Exception during hasNext method.
	at io.kubernetes.client.util.Watch.hasNext(Watch.java:183)
	at org.apache.druid.k8s.discovery.DefaultK8sApiClient$2.hasNext(DefaultK8sApiClient.java:200)
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:266)
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:236)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.InterruptedIOException
	at okhttp3.internal.http2.Http2Stream.waitForIo$okhttp(Http2Stream.kt:716)
	at okhttp3.internal.http2.Http2Stream$FramingSource.read(Http2Stream.kt:419)
	at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.kt:346)
	at okio.RealBufferedSource.exhausted(RealBufferedSource.kt:223)
	at io.kubernetes.client.util.Watch.hasNext(Watch.java:181)
	... 8 more
Hmmm.... I've just learned that the 5 queries actually run sequentially and each take about 1.5 minutes. So now I'm thinking that there's simply a long delay in shutting down a pod (i.e. over 5 minutes) thus the task slot takes a long time to free up...??
False alarm. It's actually unrelated to these jobs at all and related to some parallel Kafka jobs that all started at exactly the same time 😢
2 Views
Open in Slack
PreviousNext
Powered by