This the MIDDLE manager config which we are passing But getting some issue in Memory in some pods of MM Please help me to fix this issue

middleManager:
  ## If false, middleManager will not be installed
  ##
  metricsName: metrics
  metricsPort: 9200
  enabled: true
  name: middle-manager
  replicaCount: 4
  port: 8091

  config:
    druid_node_type: 'middleManager'
    druid_worker_capacity: 6
    druid_worker_baseTaskDirs: '["/opt/druid/var/druid/worker_task_baseDir"]'
    druid_worker_baseTaskDirSize: '20000000000'
    DRUID_XMX: 8G
    DRUID_XMS: 8G
    DRUID_MAXDIRECTMEMORYSIZE: 16g
    druid_processing_buffer_sizeBytes: '200000000'
    druid_processing_numMergeBuffers: 3
    druid_processing_numThreads: 6
    druid_server_http_numThreads: 250
    druid_indexer_runner_javaOptsArray: '["-server", "-Xms2g", "-Xmx4g", "-XX:MaxDirectMemorySize=8g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'
    druid_indexer_fork_property_druid_processing_buffer_sizeBytes: '200000000'
    druid_indexer_fork_property_druid_processing_numMergeBuffers: 3
    druid_indexer_fork_property_druid_processing_numThreads: 6
    druid_indexer_fork_property_druid_server_http_numThreads: 70
    #new changes
    druid_indexer_task_baseDir: "/opt/druid/var/druid/task_baseDir"
    druid_indexer_task_baseTaskDir: "/opt/druid/var/druid/baseTaskDir"
    druid_indexer_task_restoreTasksOnRestart: true
    druid_processing_tmpDir: "/opt/druid/var/druid/tmpDir"
    druid_indexer_storage_type: "metadata"
    druid_realtime_cache_useCache: true
    druid_realtime_cache_populateCache: true
    druid_cache_type: caffeine
    #groupby
    druid_query_groupBy_maxMergingDictionarySize: "500000000"
    druid_query_groupBy_maxOnDiskStorage: "10000000000"

    #autoscaling task
    druid_indexer_autoscale_strategy: "ec2"
    druid_indexer_autoscale_doAutoscale: true

    druid_monitoring_monitors: '["org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor"]'
    #druid_monitoring_monitors: '["org.apache.druid.client.cache.CacheMonitor", "org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor", "org.apache.druid.java.util.metrics.JvmThreadsMonitor", "org.apache.druid.server.metrics.EventReceiverFirehoseMonitor"]'

Hi everyone, We have around 150 dimensions in our Druid cluster. Only three columns account for roughly 45% of the total storage volume. After analysing the dimension-level storage usage, we found that these three columns use the hyperUnique data type. Is there a way to reduce the storage consumed by columns with the hyperUnique data type?

Hello, I have Task Logging enable via:

druid.indexer.logs.type=file
druid.indexer.logs.directory=/data/indexing-logs
druid.indexer.logs.kill.enabled=true
druid.indexer.logs.kill.durationToRetain=2592000000

However, I'm noticing that I only have 24 hours of task logs. Am I missing something?

Hi team, I am using Druid 24.0.0 on my local. I have defined a compaction rule on my datasource. Though setting the

Skip offset from latest

as PT0S, druid is not auto compacting the segments. After manual server restart Compaction is happening. My compaction config looks like:

{
  "dataSource": "shivam_test_with_diff_dates",
  "taskPriority": 25,
  "inputSegmentSizeBytes": 100000000000000,
  "maxRowsPerSegment": 5000000,
  "skipOffsetFromLatest": "PT0S",
  "tuningConfig": {
    "maxRowsInMemory": null,
    "appendableIndexSpec": null,
    "maxBytesInMemory": null,
    "maxTotalRows": null,
    "splitHintSpec": null,
    "partitionsSpec": {
      "type": "dynamic",
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null
    },
    "indexSpec": null,
    "indexSpecForIntermediatePersists": null,
    "maxPendingPersists": null,
    "pushTimeout": null,
    "segmentWriteOutMediumFactory": null,
    "maxNumConcurrentSubTasks": null,
    "maxRetry": null,
    "taskStatusCheckPeriodMs": null,
    "chatHandlerTimeout": null,
    "chatHandlerNumRetries": null,
    "maxNumSegmentsToMerge": null,
    "totalNumMergeTasks": null,
    "maxColumnsToMerge": null,
    "type": "index_parallel",
    "forceGuaranteedRollup": false
  },
  "granularitySpec": {
    "segmentGranularity": "DAY",
    "queryGranularity": null,
    "rollup": false              
  "dimensionsSpec": null,
  "metricsSpec": null,
  "transformSpec": null,
  "ioConfig": null,
  "taskContext": null
  }
}

What is wrong. Can someone plz help.

Hi team, I have an existing Druid table and I want to update or delete specific rows/columns. Could anyone share the best approach or examples for doing this?

Hi team, I have an existing Druid datasource and I want to update or delete specific rows/columns from Python code. Could anyone share the best approach or examples for doing this?

This morning I came in to discover that during a deploy trying to turn on ingestion task autoscaling, druid deleted supervisors. Does anyone have a clue why that could happen? afaik taskCountMin wasn't even set to 0 (we do some python pre-templating and have it set more or less to be

math.ceil(int(configuredTaskCount * 0.75))

^ With regard to this, I figured it out. I accidentally had the supervisor spec with taskCountMin > taskCountMax, which resulted in the supervisor being deleted for having an invalid config. That seems like a great way to cause an outage compared to, say, using the last known good config. Would people be open to a ticket for this?

Hi Team, We are running a

GROUP BY

query on our rollup table, but the execution time is currently around 25–30 seconds. Is there any way to optimize this so that we can achieve sub-second or 3–5 second latency, even with 15–20 concurrent queries running in parallel?

1. Would switching to

topN

be the primary recommendation here for speed? 2. Are there specific Broker/Historical memory tuning parameters (processing threads/buffers) that help specifically with concurrency? 3. Does enabling

vectorize: "force"

usually help with Theta Sketches, or should we avoid it?

How is this streaming task auto restart feature suppose to work ? I enabled it but despite having set a graceful termination of 6min (k8s pod termination) the tasks are still marked as failed and the supervisor spawns new tasks on other MMs nodes that re-reading everything from the failed task. I'm running druid v34 on k8s with canonical hostnames enabled.

Untitled

Hi team, I am using Druid 34.0.0. We are using Iceberg, AWS Glue with Druid ingestion spec. The problem we are noticing is ingesting just "1000" records is taking almost 5 mins. We feels it's too expensive and not sure, we are fully utilizing the druid optimization properly or not. If Druid itself is NOT supporting Iceberg, with AWS Glue data load into Druid which is causing this performance issue. We are in the process running heavy load (100 millions) and so worried to use Druid or not with data ingestion process because of the time it takes. Kindly review if anybody has any suggestions or recommendation is highly appreciated. Thank you so much. My Ingestion config looks like:

I seem to be getting a lot of logs like this:

2025-12-15T15:06:29,572 ERROR [Coordinator-Exec-HistoricalManagementDuties-0] org.apache.druid.server.coordinator.ServerHolder - Load queue for server [druid-data13:8083], tier [_default_tier] has [390] segments stuck.: {class=org.apache.druid.server.coordinator.ServerHolder, segments=[REPLICATE{segment=redacted, runsInQueue=182}, REPLICATE{segment=redacted, runsInQueue=77}, ...], maxLifetime=60}

Different data hosts, different datasources. Anybody else know how to resolve this?

Hello Druid Experts, Has anyone performed benchmarks specifically for the “Virtual Storage” feature in Druid version 35? I’m interested in understanding the query latency metrics when Virtual Storage is enabled. Also, since this feature is marked as experimental in version 35, is it expected to be production-ready in the next release?

What is the correct usage of properties to ingest from Us East 1 S3 Bucket When Druid is hosted in Us East 2?

S3 error: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: Z4BTWT1VAVXB3G5C; S3 Extended Request ID: lMp2P3BULzc0sHeFwxOSelDlBv7LduKfg6AfzX80ythBUA0iIW//22XPbGbBTI7udNx9LCB7Im8qeH0xCq0Zgjr7gf8NylkIIoFFFymKF1E=; Proxy: null)

Hey 👋 I’m trying to deploy a new Druid cluster (v36) on EKS using the Druid Operator, but the routers are failing to start. I’m seeing this on startup:

ERROR [main] org.apache.druid.cli.CliRouter - Error when starting up. Failing.
javax.servlet.ServletException: javax.servlet.ServletException: 
java.lang.IllegalStateException: Insufficient configured threads: required=2 < max=2 
for QueuedThreadPool[AsyncManagementForwardingServlet...]

Router runtime properties look like this:

druid.service=druid/router
druid.server.http.numThreads=100
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
druid.router.managementProxy.enabled=true

I’m not entirely sure where to look next - has anyone run into something similar with Druid 36 or the management proxy? Any pointers would be much appreciated 🙏

sure thanks

Hello Team, I have a question about the Kafka -> Druid Ingestion, have a supervisor ingestion spec which creates and always reference to a single data source. Is this TRUE statement? meaning one spec can point to one datasource. I don't see anyway to achieve for multiple spec can point to a single data source? what i have established so far is one spec points to one data source however i couldn't able to achieve multiple spec points to a single data source. so help me if this is working already, any reference, really appreciated? Thank you so much.

Hello 👋. I am trying to ingest parquet data stored in GCS to druid using native parallel ingestion but running into hadoop related errors. Is it not possible to natively ingest parquet from gcs without hadoop dataproc cluster dependency? Or something missing in my config?

2026-02-18T23:40:12,199 WARN [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290) ~[?:?]
	at org.apache.druid.data.input.parquet.ParquetInputFormat.initialize(ParquetInputFormat.java:68) ~[?:?]
	at org.apache.druid.data.input.parquet.ParquetInputFormat.createReader(ParquetInputFormat.java:98) ~[?:?]at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290) ~[?:?]
	at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:432) ~[druid-indexing-service-27.0.0.jar:27.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:173) ~[druid-indexing-service-27.0.0.jar:27.0.0]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
	at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]

From the task logs i am loading all the extensions.

2026-02-18T23:40:08,513 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-google-extensions], jars: druid-google-extensions-27.0.0.jar, google-api-services-storage-v1-rev20190523-1.26.0.jar
2026-02-18T23:40:10,406 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.loadList: ["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-google-extensions", "druid-avro-extensions", "druid-kafka-indexing-service", "druid-parquet-extensions"]

And also in classpath it seems to have hadoop

org.apache.druid.cli.CliPeon - * java.class.path:lib/druid-gcp-common-27.0.0.jar:lib/guice-servlet-4.1.0.jar:lib/aws-java-sdk-ec2-1.12.497.jar:lib/gcs-connector-hadoop3-latest.jar

Here is my sample ingestion spec.

{
    "type": "index_parallel",
    "spec": {
        "dataSchema": {
            "dataSource": "test_parquet_gcs",
            "timestampSpec": {.....},
            "dimensionsSpec": {.....},
            "metricsSpec": [....],
            "granularitySpec": {
                "type": "uniform",
                "segmentGranularity": "HOUR",
                "queryGranularity": "HOUR",
            }
        },
        "ioConfig": {
            "type": "index_parallel",
            "inputSource": {
                "type": "google",
                "prefixes": [
                    "gs://<bucket>/date_key=2026-02-17/hour=22"
                ]
            },
            "inputFormat": {
                "type": "parquet"
            },
            "appendToExisting": false
        },
        "tuningConfig": {
            "type": "index_parallel",
            "maxRowsPerSegment": 1000000,
            "maxRowsInMemory": 1000000,
            "maxNumConcurrentSubTasks": 8,
            "partitionsSpec": {
                "type": "dynamic",
                "partitionDimensions": []
            },
            "forceGuaranteedRollup": false
        }
    }
}

This is kind of "dangerous" because it could mess up replicas if you use them, and idempotency if reloading is needed, etc. But, I think it might be possible now. I'm not sure whether this is "Imply only", I don't think so.

Hi! I have been checking heap dump for Druid Historical process that exited due to OutOfMemory error. It looks

org.apache.druid.client.cache.CaffeineCache

took 32476592624 (~32GB) bytes while my config is set to

druid.cache.sizeInBytes=25769803776

(~25GB). I am running Druid version 31.0.1. What could be the reason of Druid cache going beyond specified limit?

Hi, I am running into an error when submitting supervisor in druid I have a realtime pipeline running using EMR (Apache Flink) and MSK in AWS . The job reads from Apache Kafka topic, processes it, and then writes to the output Kafka topic. Once data lands in Kafka topic (MSK), the data is written to Druid segment through a supervisor created in Druid However, when I try to

Submit supervisor

using

Load Data

option in Druid console, I get an error on the supervisor created

{
  "dataSource": "master_order_event",
  "stream": "realtime-pipelines-pos-db.pos_local.orders",
  "partitions": 0,
  "replicas": 1,
  "durationSeconds": 3600,
  "activeTasks": [],
  "publishingTasks": [],
  "minimumLag": {},
  "aggregateLag": 0,
  "suspended": true,
  "healthy": true,
  "state": "PENDING",
  "detailedState": "PENDING",
  "recentErrors": [
    {
      "timestamp": "2026-03-06T06:35:28.082Z",
      "exceptionClass": "org.apache.kafka.common.config.ConfigException",
      "message": "Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.",
      "streamException": false
    },
    {
      "timestamp": "2026-03-06T06:36:37.461Z",
      "exceptionClass": "org.apache.kafka.common.config.ConfigException",
      "message": "Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.",
      "streamException": false
    },
    {
      "timestamp": "2026-03-06T06:37:49.854Z",
      "exceptionClass": "org.apache.kafka.common.config.ConfigException",
      "message": "Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.",
      "streamException": false
    }
  ]
}

I have tried to troubleshoot this issue with a few steps: I added the

aws-msk-iam-auth-2.2.0-all.jar

to

/opt/druid/extensions/druid-kafka-indexing-service/aws-msk-iam-auth-2.2.0-all.jar

1. Placed aws-msk-iam-auth-2.2.0-all.jar in /opt/druid/extensions/druid-kafka-indexing-service/ Druid Middle Manager docker container inside EC2 instance 2. Restarted Druid Middle Manager docker container After this, I could see the jars are in correct place, the druid-indexing-service is added to common-runtime.properties, the container was restarted but still running into Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found. Then I copied the jar to another path

cp /opt/druid/extensions/druid-kafka-indexing-service/aws-msk-iam-auth-2.2.0-all.jar /opt/druid/lib/

Is this the right solution? I am still running into issues

Hi all, I have an issue regarding native query granularization. Due to issues with PeriodGranularity granularity, we use a custom "QueryPeriodGranularity" granularity that

extends PeriodGranularity

to solve issues with arbitrary `origin`s. The issue is regarding variable-length time units and DST changes. For example, if you increment 3 months from

2024-12-31

using PeriodGranularity, you get a bucket starting with

2025-03-28

instead of

2025-03-31

. This is because the granularity works by adding iteratively instead of multiplying from the origin, so it gets "off track" when it encounters short months. This custom QueryPeriodGranularity worked well enough to fix the "bad" buckets we were getting, but we recently discovered an issue where querying one year w/ a granularity of

P1M

returns different results than querying one month at a time with

all

granularity. The sum of all months is the same, but some data is aggregated into the wrong buckets. When attempting this query with the PeriodGranularity, the query instead fails. We are primarily running on Druid 28, but this is repeatable on Druid 34 (of which we are in the process of upgrading). Query snippet:

{
  "queryType": "groupBy",
  "intervals": [
    "2024-12-31T23:00:00.000Z/2025-12-31T23:00:00.000Z"
  ],
  "granularity": {
    "type": "period",
    "period": "P1M",
    "origin": "2024-12-31T23:00:00.000Z",
    "timeZone": "UTC"
  },
  ...
}

Exception snippet:

{
    "t": "2026-03-09T20:13:40.456Z",
    "message": "Exception with one of the sequences!",
    "logger_name": "org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2",
    "thread_name": "processing-5",
    "level": "ERROR",
    "stack_trace": "java.lang.IllegalArgumentException: The end instant must be greater than the start instant\n\tat org.joda.time.base.AbstractInterval.checkInterval(AbstractInterval.java:63)\n\tat org.joda.time.base.BaseInterval.<init>(BaseInterval.java:94)\n\tat org.joda.time.Interval.<init>(Interval.java:201)\n\tat org.apache.druid.query.vector.VectorCursorGranularizer.create(VectorCursorGranularizer.java:82)\n\tat org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$VectorGroupByEngineIterator.<init>(VectorGroupByEngine.java:286)\n\tat org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$1.make(VectorGroupByEngine.java:192)\n\tat
    ...
}

I'm a bit lost, so here are my questions: 1. Is this a valid point for extension, or are there other places that the existing PeriodGranularity bucket logic may be "hard-coded" that is causing such a discrepancy between the buckets and aggregation? 2. Should I file an issue regarding potential issues with the PeriodGranularity granularity?

Hello all, I am facing an issue where a specific datasource (XYZ) becomes queryable only intermittently and frequently drops into an "Inactive/Unused" state. The Issue: The datasource appears in the "Unused" list in the console. When clicked on datasource and , I receive the following error: Plan validation failed / org.apache.calcite.runtime.CalciteContextException: Object 'XYZ' not found / org.apache.calcite.tools.ValidationException Current Workaround: If I manually "Mark segments as used" in the console, the table becomes visible and queryable again. However, after some time, it reverts to the "Unused" state automatically. Key Context & Environment: Ingestion: Data is ingested via manual batch processes. The most recent ingestion was several weeks ago, and no manual ingestion was running when the segments became unused. Compaction: Auto-compaction is enabled for this datasource with skipOffsetFromLatest: PT0S. Retention: There are no active drop rules or retention policies configured. Deep Storage: S3 Druid Version: 25.0.0 Requesting Help with: Since no manual ingestion was active, could Auto-Compaction with a PT0S offset be marking segments as unused but failing to successfully publish/load the new compacted versions? Are there specific Coordinator or Overlord log signatures I should search for to see why these segments are being deactivated? Are there known issues where PT0S offset causes race conditions that lead to all segments being marked as unused? Any guidance on which logs to check or common pitfalls with this setup would be greatly appreciated!

Hello everybody, Could somebody point me to the root-cause/fix of task starting failure of batch ingestion job that happens very rarely? Still it's annoying because is a back-filling task for historical datasources. You can find attached the full task logs but this is the error stack trace:

2026-04-14T20:05:06,869 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up.  Failing.
java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:390) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.Main.main(Main.java:112) [druid-services-34.0.0.jar:34.0.0]
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at org.apache.commons.io.function.Uncheck.wrap(Uncheck.java:339) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:199) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more
Caused by: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148) ~[?:?]
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:226) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323) ~[?:?]
	at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71) ~[?:?]
	at java.base/java.nio.file.Files.walk(Files.java:3918) ~[?:?]
	at org.apache.commons.io.file.PathUtils.walk(PathUtils.java:1847) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.streamFiles(FileUtils.java:2964) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.lambda$listFiles$10(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:197) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more

A task that re-ingest data from other one datasource (realtime) to another (historical-hour) cannot start because it doesn't find some files of a compaction task shuffled data that is associated with a completely unrelated datasource. The task doesn't even overlap in time, the compaction task finished around 9 minutes before the failed re-indexing task even started. Thank you!

Hey , I had druid setup with 4 historical servers , over time free disk space became smaller and smaller .. till got to 99% full , at this point I added another historical server … , question - is druid know to rebalance the capacity of historical disk space after adding / removing an historical server ?, .as it seems now 4 historical’s still allmost full , and the new historical is allmost empty .. should I do some action to have a better spread between historicals ? this is the current state + (I allready set retention policy)

Hello, I'm trying to go middlemanager-less in k8s with Druid v34 but I'm having issues with Peon startup the job fails with:

2026-04-16T15:13:13,086 INFO [main] org.apache.druid.cli.CliPeon - Task file not found, trying to pull task payload from deep storage
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) [Guice/ErrorInCustomProvider]: IllegalStateException: Optional.get() cannot be called on an absent value
  at CliPeon$1.readTask(CliPeon.java:314)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> CliPeon$1
  at CliPeon$1.getBroadcastDatasourcesToLoad(CliPeon.java:356)
      \_ for 1st parameter
  at CliPeon$1.getBroadcastDatasourcesToLoad(CliPeon.java:356)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> CliPeon$1
  at DataSourceTaskIdHolder.broadcastDatasourceLoadingSpec(DataSourceTaskIdHolder.java:45)
      \_ for field broadcastDatasourceLoadingSpec
  at MetricsModule.configure(MetricsModule.java:94)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> MetricsModule
  while locating DataSourceTaskIdHolder
  at MetricsModule.getMonitorScheduler(MetricsModule.java:114)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> MetricsModule
  while locating MonitorScheduler
  at MetricsModule.configure(MetricsModule.java:99)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> MetricsModule
  while locating MonitorScheduler annotated with @Named("ForTheEagerness")

Learn more:
  <https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER>

1 error

======================
Full classname legend:
======================
CliPeon$1:              "org.apache.druid.cli.CliPeon$1"
DataSourceTaskIdHolder: "org.apache.druid.server.metrics.DataSourceTaskIdHolder"
MetricsModule:          "org.apache.druid.server.metrics.MetricsModule"
Modules$OverrideModule: "com.google.inject.util.Modules$OverrideModule"
MonitorScheduler:       "org.apache.druid.java.util.metrics.MonitorScheduler"
Named:                  "com.google.inject.name.Named"
========================
End of classname legend:
========================

I'm curious why I'm seeing this in my coordinator logs and if it would cause any problems in moving segments (i.e. that the min > max)

2026-04-21T14:23:18,690 INFO [Coordinator-Exec-HistoricalManagementDuties-0] org.apache.druid.server.coordinator.balancer.SegmentToMoveCalculator - Need to move [368] segments in tier[_default_tier] to attain balance. Allowed values are [min=200, max=100].

Druid data sources are not showing the realtime segments , any idea why this happens. Because of that reason we are not able to query live data. We are running Druid on K8s