I seem to be getting a lot of logs like this:

2025-12-15T15:06:29,572 ERROR [Coordinator-Exec-HistoricalManagementDuties-0] org.apache.druid.server.coordinator.ServerHolder - Load queue for server [druid-data13:8083], tier [_default_tier] has [390] segments stuck.: {class=org.apache.druid.server.coordinator.ServerHolder, segments=[REPLICATE{segment=redacted, runsInQueue=182}, REPLICATE{segment=redacted, runsInQueue=77}, ...], maxLifetime=60}

Different data hosts, different datasources. Anybody else know how to resolve this?

Hello Druid Experts, Has anyone performed benchmarks specifically for the “Virtual Storage” feature in Druid version 35? I’m interested in understanding the query latency metrics when Virtual Storage is enabled. Also, since this feature is marked as experimental in version 35, is it expected to be production-ready in the next release?

What is the correct usage of properties to ingest from Us East 1 S3 Bucket When Druid is hosted in Us East 2?

S3 error: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect; Request ID: Z4BTWT1VAVXB3G5C; S3 Extended Request ID: lMp2P3BULzc0sHeFwxOSelDlBv7LduKfg6AfzX80ythBUA0iIW//22XPbGbBTI7udNx9LCB7Im8qeH0xCq0Zgjr7gf8NylkIIoFFFymKF1E=; Proxy: null)

Hey 👋 I’m trying to deploy a new Druid cluster (v36) on EKS using the Druid Operator, but the routers are failing to start. I’m seeing this on startup:

ERROR [main] org.apache.druid.cli.CliRouter - Error when starting up. Failing.
javax.servlet.ServletException: javax.servlet.ServletException: 
java.lang.IllegalStateException: Insufficient configured threads: required=2 < max=2 
for QueuedThreadPool[AsyncManagementForwardingServlet...]

Router runtime properties look like this:

druid.service=druid/router
druid.server.http.numThreads=100
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator
druid.router.managementProxy.enabled=true

I’m not entirely sure where to look next - has anyone run into something similar with Druid 36 or the management proxy? Any pointers would be much appreciated 🙏

sure thanks

Hello Team, I have a question about the Kafka -> Druid Ingestion, have a supervisor ingestion spec which creates and always reference to a single data source. Is this TRUE statement? meaning one spec can point to one datasource. I don't see anyway to achieve for multiple spec can point to a single data source? what i have established so far is one spec points to one data source however i couldn't able to achieve multiple spec points to a single data source. so help me if this is working already, any reference, really appreciated? Thank you so much.

Hello 👋. I am trying to ingest parquet data stored in GCS to druid using native parallel ingestion but running into hadoop related errors. Is it not possible to natively ingest parquet from gcs without hadoop dataproc cluster dependency? Or something missing in my config?

2026-02-18T23:40:12,199 WARN [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290) ~[?:?]
	at org.apache.druid.data.input.parquet.ParquetInputFormat.initialize(ParquetInputFormat.java:68) ~[?:?]
	at org.apache.druid.data.input.parquet.ParquetInputFormat.createReader(ParquetInputFormat.java:98) ~[?:?]at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290) ~[?:?]
	at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:432) ~[druid-indexing-service-27.0.0.jar:27.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:173) ~[druid-indexing-service-27.0.0.jar:27.0.0]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
	at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]

From the task logs i am loading all the extensions.

2026-02-18T23:40:08,513 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-google-extensions], jars: druid-google-extensions-27.0.0.jar, google-api-services-storage-v1-rev20190523-1.26.0.jar
2026-02-18T23:40:10,406 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.loadList: ["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-google-extensions", "druid-avro-extensions", "druid-kafka-indexing-service", "druid-parquet-extensions"]

And also in classpath it seems to have hadoop

org.apache.druid.cli.CliPeon - * java.class.path:lib/druid-gcp-common-27.0.0.jar:lib/guice-servlet-4.1.0.jar:lib/aws-java-sdk-ec2-1.12.497.jar:lib/gcs-connector-hadoop3-latest.jar

Here is my sample ingestion spec.

{
    "type": "index_parallel",
    "spec": {
        "dataSchema": {
            "dataSource": "test_parquet_gcs",
            "timestampSpec": {.....},
            "dimensionsSpec": {.....},
            "metricsSpec": [....],
            "granularitySpec": {
                "type": "uniform",
                "segmentGranularity": "HOUR",
                "queryGranularity": "HOUR",
            }
        },
        "ioConfig": {
            "type": "index_parallel",
            "inputSource": {
                "type": "google",
                "prefixes": [
                    "gs://<bucket>/date_key=2026-02-17/hour=22"
                ]
            },
            "inputFormat": {
                "type": "parquet"
            },
            "appendToExisting": false
        },
        "tuningConfig": {
            "type": "index_parallel",
            "maxRowsPerSegment": 1000000,
            "maxRowsInMemory": 1000000,
            "maxNumConcurrentSubTasks": 8,
            "partitionsSpec": {
                "type": "dynamic",
                "partitionDimensions": []
            },
            "forceGuaranteedRollup": false
        }
    }
}

This is kind of "dangerous" because it could mess up replicas if you use them, and idempotency if reloading is needed, etc. But, I think it might be possible now. I'm not sure whether this is "Imply only", I don't think so.

Hi! I have been checking heap dump for Druid Historical process that exited due to OutOfMemory error. It looks

org.apache.druid.client.cache.CaffeineCache

took 32476592624 (~32GB) bytes while my config is set to

druid.cache.sizeInBytes=25769803776

(~25GB). I am running Druid version 31.0.1. What could be the reason of Druid cache going beyond specified limit?

Hi, I am running into an error when submitting supervisor in druid I have a realtime pipeline running using EMR (Apache Flink) and MSK in AWS . The job reads from Apache Kafka topic, processes it, and then writes to the output Kafka topic. Once data lands in Kafka topic (MSK), the data is written to Druid segment through a supervisor created in Druid However, when I try to

Submit supervisor

using

Load Data

option in Druid console, I get an error on the supervisor created

{
  "dataSource": "master_order_event",
  "stream": "realtime-pipelines-pos-db.pos_local.orders",
  "partitions": 0,
  "replicas": 1,
  "durationSeconds": 3600,
  "activeTasks": [],
  "publishingTasks": [],
  "minimumLag": {},
  "aggregateLag": 0,
  "suspended": true,
  "healthy": true,
  "state": "PENDING",
  "detailedState": "PENDING",
  "recentErrors": [
    {
      "timestamp": "2026-03-06T06:35:28.082Z",
      "exceptionClass": "org.apache.kafka.common.config.ConfigException",
      "message": "Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.",
      "streamException": false
    },
    {
      "timestamp": "2026-03-06T06:36:37.461Z",
      "exceptionClass": "org.apache.kafka.common.config.ConfigException",
      "message": "Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.",
      "streamException": false
    },
    {
      "timestamp": "2026-03-06T06:37:49.854Z",
      "exceptionClass": "org.apache.kafka.common.config.ConfigException",
      "message": "Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.",
      "streamException": false
    }
  ]
}

I have tried to troubleshoot this issue with a few steps: I added the

aws-msk-iam-auth-2.2.0-all.jar

to

/opt/druid/extensions/druid-kafka-indexing-service/aws-msk-iam-auth-2.2.0-all.jar

1. Placed aws-msk-iam-auth-2.2.0-all.jar in /opt/druid/extensions/druid-kafka-indexing-service/ Druid Middle Manager docker container inside EC2 instance 2. Restarted Druid Middle Manager docker container After this, I could see the jars are in correct place, the druid-indexing-service is added to common-runtime.properties, the container was restarted but still running into Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found. Then I copied the jar to another path

cp /opt/druid/extensions/druid-kafka-indexing-service/aws-msk-iam-auth-2.2.0-all.jar /opt/druid/lib/

Is this the right solution? I am still running into issues

Hi all, I have an issue regarding native query granularization. Due to issues with PeriodGranularity granularity, we use a custom "QueryPeriodGranularity" granularity that

extends PeriodGranularity

to solve issues with arbitrary `origin`s. The issue is regarding variable-length time units and DST changes. For example, if you increment 3 months from

2024-12-31

using PeriodGranularity, you get a bucket starting with

2025-03-28

instead of

2025-03-31

. This is because the granularity works by adding iteratively instead of multiplying from the origin, so it gets "off track" when it encounters short months. This custom QueryPeriodGranularity worked well enough to fix the "bad" buckets we were getting, but we recently discovered an issue where querying one year w/ a granularity of

P1M

returns different results than querying one month at a time with

all

granularity. The sum of all months is the same, but some data is aggregated into the wrong buckets. When attempting this query with the PeriodGranularity, the query instead fails. We are primarily running on Druid 28, but this is repeatable on Druid 34 (of which we are in the process of upgrading). Query snippet:

{
  "queryType": "groupBy",
  "intervals": [
    "2024-12-31T23:00:00.000Z/2025-12-31T23:00:00.000Z"
  ],
  "granularity": {
    "type": "period",
    "period": "P1M",
    "origin": "2024-12-31T23:00:00.000Z",
    "timeZone": "UTC"
  },
  ...
}

Exception snippet:

{
    "t": "2026-03-09T20:13:40.456Z",
    "message": "Exception with one of the sequences!",
    "logger_name": "org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2",
    "thread_name": "processing-5",
    "level": "ERROR",
    "stack_trace": "java.lang.IllegalArgumentException: The end instant must be greater than the start instant\n\tat org.joda.time.base.AbstractInterval.checkInterval(AbstractInterval.java:63)\n\tat org.joda.time.base.BaseInterval.<init>(BaseInterval.java:94)\n\tat org.joda.time.Interval.<init>(Interval.java:201)\n\tat org.apache.druid.query.vector.VectorCursorGranularizer.create(VectorCursorGranularizer.java:82)\n\tat org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$VectorGroupByEngineIterator.<init>(VectorGroupByEngine.java:286)\n\tat org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$1.make(VectorGroupByEngine.java:192)\n\tat
    ...
}

I'm a bit lost, so here are my questions: 1. Is this a valid point for extension, or are there other places that the existing PeriodGranularity bucket logic may be "hard-coded" that is causing such a discrepancy between the buckets and aggregation? 2. Should I file an issue regarding potential issues with the PeriodGranularity granularity?

Hello all, I am facing an issue where a specific datasource (XYZ) becomes queryable only intermittently and frequently drops into an "Inactive/Unused" state. The Issue: The datasource appears in the "Unused" list in the console. When clicked on datasource and , I receive the following error: Plan validation failed / org.apache.calcite.runtime.CalciteContextException: Object 'XYZ' not found / org.apache.calcite.tools.ValidationException Current Workaround: If I manually "Mark segments as used" in the console, the table becomes visible and queryable again. However, after some time, it reverts to the "Unused" state automatically. Key Context & Environment: Ingestion: Data is ingested via manual batch processes. The most recent ingestion was several weeks ago, and no manual ingestion was running when the segments became unused. Compaction: Auto-compaction is enabled for this datasource with skipOffsetFromLatest: PT0S. Retention: There are no active drop rules or retention policies configured. Deep Storage: S3 Druid Version: 25.0.0 Requesting Help with: Since no manual ingestion was active, could Auto-Compaction with a PT0S offset be marking segments as unused but failing to successfully publish/load the new compacted versions? Are there specific Coordinator or Overlord log signatures I should search for to see why these segments are being deactivated? Are there known issues where PT0S offset causes race conditions that lead to all segments being marked as unused? Any guidance on which logs to check or common pitfalls with this setup would be greatly appreciated!

Hello everybody, Could somebody point me to the root-cause/fix of task starting failure of batch ingestion job that happens very rarely? Still it's annoying because is a back-filling task for historical datasources. You can find attached the full task logs but this is the error stack trace:

2026-04-14T20:05:06,869 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up.  Failing.
java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:390) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.Main.main(Main.java:112) [druid-services-34.0.0.jar:34.0.0]
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at org.apache.commons.io.function.Uncheck.wrap(Uncheck.java:339) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:199) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more
Caused by: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148) ~[?:?]
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:226) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323) ~[?:?]
	at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71) ~[?:?]
	at java.base/java.nio.file.Files.walk(Files.java:3918) ~[?:?]
	at org.apache.commons.io.file.PathUtils.walk(PathUtils.java:1847) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.streamFiles(FileUtils.java:2964) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.lambda$listFiles$10(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:197) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more

A task that re-ingest data from other one datasource (realtime) to another (historical-hour) cannot start because it doesn't find some files of a compaction task shuffled data that is associated with a completely unrelated datasource. The task doesn't even overlap in time, the compaction task finished around 9 minutes before the failed re-indexing task even started. Thank you!

Hey , I had druid setup with 4 historical servers , over time free disk space became smaller and smaller .. till got to 99% full , at this point I added another historical server … , question - is druid know to rebalance the capacity of historical disk space after adding / removing an historical server ?, .as it seems now 4 historical’s still allmost full , and the new historical is allmost empty .. should I do some action to have a better spread between historicals ? this is the current state + (I allready set retention policy)

Hello, I'm trying to go middlemanager-less in k8s with Druid v34 but I'm having issues with Peon startup the job fails with:

2026-04-16T15:13:13,086 INFO [main] org.apache.druid.cli.CliPeon - Task file not found, trying to pull task payload from deep storage
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) [Guice/ErrorInCustomProvider]: IllegalStateException: Optional.get() cannot be called on an absent value
  at CliPeon$1.readTask(CliPeon.java:314)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> CliPeon$1
  at CliPeon$1.getBroadcastDatasourcesToLoad(CliPeon.java:356)
      \_ for 1st parameter
  at CliPeon$1.getBroadcastDatasourcesToLoad(CliPeon.java:356)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> CliPeon$1
  at DataSourceTaskIdHolder.broadcastDatasourceLoadingSpec(DataSourceTaskIdHolder.java:45)
      \_ for field broadcastDatasourceLoadingSpec
  at MetricsModule.configure(MetricsModule.java:94)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> MetricsModule
  while locating DataSourceTaskIdHolder
  at MetricsModule.getMonitorScheduler(MetricsModule.java:114)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> MetricsModule
  while locating MonitorScheduler
  at MetricsModule.configure(MetricsModule.java:99)
      \_ installed by: Modules$OverrideModule -> Modules$OverrideModule -> MetricsModule
  while locating MonitorScheduler annotated with @Named("ForTheEagerness")

Learn more:
  <https://github.com/google/guice/wiki/ERROR_IN_CUSTOM_PROVIDER>

1 error

======================
Full classname legend:
======================
CliPeon$1:              "org.apache.druid.cli.CliPeon$1"
DataSourceTaskIdHolder: "org.apache.druid.server.metrics.DataSourceTaskIdHolder"
MetricsModule:          "org.apache.druid.server.metrics.MetricsModule"
Modules$OverrideModule: "com.google.inject.util.Modules$OverrideModule"
MonitorScheduler:       "org.apache.druid.java.util.metrics.MonitorScheduler"
Named:                  "com.google.inject.name.Named"
========================
End of classname legend:
========================

I'm curious why I'm seeing this in my coordinator logs and if it would cause any problems in moving segments (i.e. that the min > max)

2026-04-21T14:23:18,690 INFO [Coordinator-Exec-HistoricalManagementDuties-0] org.apache.druid.server.coordinator.balancer.SegmentToMoveCalculator - Need to move [368] segments in tier[_default_tier] to attain balance. Allowed values are [min=200, max=100].

Druid data sources are not showing the realtime segments , any idea why this happens. Because of that reason we are not able to query live data. We are running Druid on K8s

Hi team, few questions on Kafka supervisor behaviour. If anyone can provide pointers on this it will be helpful. Thanks: 1. With

stopTaskCount=2

,

taskCount=4

and

replicas=2

, does

stopTaskCount

stop 2 task groups (all replicas included), or 2 individual tasks(staggering the two replicas)? 2. With same config as above, will both replicas of a task group start at the same time? During

intermediatePersistPeriod

flushes, will both hit persist roughly simultaneously? If one peon is blocked on merge buffer acquisition, can the

broker

use the other replica's peon to serve the query, or does it wait for both?

guys the druid operator helm chart has been disappeared. I can’t find it. Used to be

helm repo add datainfrahq <https://datainfrahq.github.io/druid-operator>

but now there is nowhere to be found. ANY IDEA ANYONE?

Need help Not able to emit the logs to Kafka how to fix it

Hi we are on Druid version 33. Trying to get Multi Region Access Point working on AWS. Seeing the error below. Could someone please help?

2026-05-27T19:28:09,754 INFO [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractTask - Not pushing task logs and reports from task.
2026-05-27T19:28:09,760 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Exception while running task[AbstractTask{id='partial_index_generic_merge_****_icdeldgd_2026-05-27T19:28:03.740Z', groupId='index_parallel_*****_bloobmgo_2026-05-27T17:12:18.920Z', taskResource=TaskResource{availabilityGroup='partial_index_generic_merge_*****_icdeldgd_2026-05-27T19:28:03.740Z', requiredCapacity=1}, dataSource='***', context={forceTimeChunkLock=true, useLineageBasedSegmentAllocation=true}}]
java.lang.RuntimeException: com.amazonaws.AmazonClientException: Unable to complete transfer: region cannot be empty
	at org.apache.druid.storage.s3.S3DataSegmentPusher.pushZip(S3DataSegmentPusher.java:123)
	at org.apache.druid.storage.s3.S3DataSegmentPusher.pushToPath(S3DataSegmentPusher.java:93)
	at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:84)
	at org.apache.druid.indexing.common.task.batch.parallel.PartialSegmentMergeTask.mergeAndPushSegments(PartialSegmentMergeTask.java:299)
	at org.apache.druid.indexing.common.task.batch.parallel.PartialSegmentMergeTask.runTask(PartialSegmentMergeTask.java:188)
	at org.apache.druid.indexing.common.task.batch.parallel.PartialGenericSegmentMergeTask.runTask(PartialGenericSegmentMergeTask.java:46)
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:176)
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478)
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: region cannot be empty
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286)
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265)
	at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.waitForCompletion(AbstractTransfer.java:103)
	at org.apache.druid.storage.s3.ServerSideEncryptingAmazonS3.upload(ServerSideEncryptingAmazonS3.java:195)
	at org.apache.druid.storage.s3.S3Utils.uploadFileIfPossible(S3Utils.java:372)
	at org.apache.druid.storage.s3.S3DataSegmentPusher.lambda$pushZip$0(S3DataSegmentPusher.java:113)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163)
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153)
	at org.apache.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:121)
	at org.apache.druid.storage.s3.S3DataSegmentPusher.pushZip(S3DataSegmentPusher.java:111)
	... 14 more
Caused by: java.lang.IllegalArgumentException: region cannot be empty
	at com.amazonaws.util.ValidationUtils.assertStringNotEmpty(ValidationUtils.java:89)
	at com.amazonaws.services.s3.S3AccessPointResource.<init>(S3AccessPointResource.java:41)
	at com.amazonaws.services.s3.S3AccessPointResource.<init>(S3AccessPointResource.java:27)
	at com.amazonaws.services.s3.S3AccessPointResource$Builder.build(S3AccessPointResource.java:219)
	at com.amazonaws.services.s3.S3ArnConverter.parseS3AccessPointArn(S3ArnConverter.java:169)
	at com.amazonaws.services.s3.S3ArnConverter.convertArn(S3ArnConverter.java:77)
	at com.amazonaws.services.s3.AmazonS3Client.createRequest(AmazonS3Client.java:5096)
	at com.amazonaws.services.s3.AmazonS3Client.createRequest(AmazonS3Client.java:5064)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1823)
	at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:169)
	at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:149)
	at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115)
	at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	... 3 more

Need help Not able to emit the logs to Kafka how to fix it Please help here

Hi we are on Druid 33 version, we are having issues with loading Task page on UI and due to this when we submit tasks using druid task api, its giving 400 Bad Request. When I check the overlord logs during that time it throws the below error. Does anyone faced similar issue or any idea how this can be resolved

2026-06-10T02:05:00,117 WARN [qtp465556527-363] org.apache.druid.indexing.overlord.http.OverlordResource - Failed to perform task action
2026-06-10T02:05:00.118437315Z java.lang.RuntimeException: java.util.concurrent.TimeoutException
2026-06-10T02:05:00.118443086Z 	at org.apache.druid.indexing.common.actions.LocalTaskActionClient.performAction(LocalTaskActionClient.java:72)
2026-06-10T02:05:00.118448737Z 	at org.apache.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:54)
2026-06-10T02:05:00.118454337Z 	at org.apache.druid.indexing.overlord.http.OverlordResource$1.apply(OverlordResource.java:525)
2026-06-10T02:05:00.118459887Z 	at org.apache.druid.indexing.overlord.http.OverlordResource$1.apply(OverlordResource.java:514)
2026-06-10T02:05:00.118488872Z 	at org.apache.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:835)
2026-06-10T02:05:00.118496266Z 	at org.apache.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:511)
2026-06-10T02:05:00.118503098Z 	at jdk.internal.reflect.GeneratedMethodAccessor507.invoke(Unknown Source)
2026-06-10T02:05:00.118526963Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2026-06-10T02:05:00.118535008Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:569)

Hey y'all, I'm implementing a custom aggregator and running into some trouble. I have the module set up, I know the module is being loaded, but when I submit a query which uses the aggregator, what gets returned is the initial state of my aggregation and as far as I can tell the

aggregate()

method is never called (I log when it's called and it never appears.) I also don't see the aggregator as an option if I choose to load data from the console, I'm not sure if that's an indicator of something going wrong? Where should I start looking to track this down?

Is there a 'recommended' way to debug a druid service locally? Still poking at that aggregator and I'm trying to debug it, but when I added the agentlib and all that to the historical jvm args, the process doesn't appear in my debugger. I also tried modifying run-druid to inject them if $1 is "historical" but that didn't seem to do it either? The java invocation I'm seeing in my process table seems totally different to what run-druid is setting up. This is using the 28 release for what it's worth (we're working on upgrading.)

Hi all, We're running a high-rate batch ingestion workload and have hit what appears to be a per-datasource segment allocation bottleneck. I'd appreciate a sanity check on our diagnosis and advice on next steps — and I'd especially like to hear from anyone who has load-tested batch segment allocation tuning at similar rates. Setup - Druid 36.0.0, on EC2 (based on the AWS "scalable analytics" solution) - Metadata store: PostgreSQL (RDS), healthy — sub-ms write latency - Overlord: 24 GB heap, httpRemote runner - ~100 MiddleManager nodes × druid.worker.capacity=2 → ~200 ingestion slots (plus a separate worker category for compaction, via equalDistributionWithCategorySpec) - Workload: ~200 index_parallel tasks/minute, each ingesting one small S3 file, all appending to a single datasource with HOUR segment granularity, dynamic partitioning, maxNumConcurrentSubTasks=1. An upstream consumer submits tasks from S3-event notifications and polls status. Relevant Overlord config druid.indexer.runner.type=httpRemote druid.indexer.runner.pendingTasksRunnerNumThreads=20 druid.indexer.tasklock.batchSegmentAllocation=true druid.indexer.tasklock.batchAllocationWaitTime=500 druid.server.http.numThreads=150 druid.indexer.queue.maxSize=2500 What we've observed 1. Originally on the ZK remote runner, the single rtr-pending-tasks-runner-0 thread was pinned at 100% CPU in ZkWorker.getRunningTasks Jackson deserialization — assignment ceiling ~150–180 tasks/min, below our 200/min arrival, so pending grew without bound. Switching to httpRemote fixed assignment throughput. 2. With assignment fixed, healthy task wall-time is ~12–25 s, but under load it degrades to ~66 s+. Phase breakdown of a sample task (99 s total): ~11 s peon startup, 56 s waiting on segment allocate, ~3 s build/push, 32 s waiting on publish. So ~90% of task time is queued on Overlord metadata operations, not on MM work. 3. The hot hour-interval accumulates ~1,800 partitions (dynamic shards) — one per task — so allocation transactions scan an increasingly large pendingSegments/ segments set as the hour progresses. 4. From reading SegmentAllocationQueue in the 36 source: batches for a single datasource execute strictly sequentially regardless of batchAllocationNumThreads, so a single hot datasource can't be parallelized — the per-datasource batch rate × batch size is the hard ceiling. What we tried that went badly We enabled druid.manager.segments.useIncrementalCache=ifSynced (Overlord + Coordinator) together with batchAllocationWaitTime=3000 and batchAllocationReduceMetadataIO=true. Shortly after the restart we saw a wave of task failures with Cannot allocate segment for timestamp[...] → ISE: Failed to add a row. We've since attributed at least part of that to a restart-induced backlog plus our client's aggressive task-kill policy (kills anything unfinished 5 min after creation, including queue time — a race where killed tasks lose their locks mid-run and allocation returns null). We reverted the cache. Question: is the incremental segment metadata cache considered usable at this scale in 36.0.0, or should it wait for 37 where it's default? Questions for the group 1. Has anyone load-tested batchSegmentAllocation (batchAllocationWaitTime, batch size behavior) at ≥200 allocations/min on a single datasource? What allocation throughput ceiling did you observe, and what settings worked? 2. Are there known strategies for scaling allocation for one hot datasource — beyond reducing task count — given per-datasource batches are serial? (Splitting into multiple datasources is unattractive for query reasons.) 3. Is ~1,800 dynamic partitions in an open hour interval itself a known allocation-cost amplifier, and does anything besides compaction (which can't touch the open hour) mitigate it? 4. Any experience with useIncrementalCache on 36.0.0 in production, positive or negative? Any pointers to prior threads, benchmarks, or design docs are much appreciated. Happy to share more config or metrics. Thanks!

Hello Team, I'm trying to use the LATEST function in the sql query. This is working fine when i run a select query to get the latest record if multiple records for a group by columns. however i'm trying the run this sql in MSQ to create a new datasource then create went through but column data contains "{"lhs":1735689647000,"rhs":"search"}" instead of the actual data so how to get rid of this to retain only actual data. Highly appreciated if any help on this. Thanks druid.apache.org/docs/latest/querying/sql-functions#…

Hello Team, We are currently using Apache Druid 37.0.0 and are exploring whether MSQ or Druid SQL supports querying/joining directly against external catalog tables, specifically Iceberg tables registered in an AWS Glue Catalog. Our goal is to execute a query similar to:

SELECT ...
FROM druid_datasource d
JOIN glue_catalog.some_database.some_iceberg_table i
  ON d.id = i.id

without first ingesting the Iceberg table into a Druid datasource. From the current Druid documentation, it appears that the Iceberg extension supports AWS Glue Catalog, but primarily through the

IcebergInputSource

, where the Iceberg data is read and ingested into Druid. We also see that MSQ supports

JOIN

operations and can read external data through

EXTERN

, but we are not clear whether an Iceberg/Glue Catalog table can be used directly as a queryable table source in an MSQ query. Could you please clarify: 1. Does Druid 37.0.0 support direct SQL/MSQ queries against Iceberg tables registered in AWS Glue Catalog? 2. If not, is this supported in any newer Druid version? 3. Is there currently a way to perform a JOIN between a Druid datasource and an Iceberg/Glue Catalog table without first ingesting the Iceberg data into Druid? 4. If this is not currently supported, is there an alternative approach or planned feature that would enable this type of federated query? Our use case is to avoid an additional ingestion step where possible and directly leverage the existing Iceberg/Glue Catalog data for joins and transformations. Reference: Druid Iceberg Ingest Extension Documentation - druid.apache.org/docs/…/iceberg#…

I recently switched from MM to MM-less in Kubernetes and I'm seeing a weird spike in Task Slots every hour. We have a job that runs hourly and creates 5 query_controllers each with 9 query_workers for a total of 50 tasks. When we were using MiddleManager, the metric

/taskSlot/used/count

would report 10 but on MM-less it is reporting 50. I can't tell if this saturated our available slots in the past but it seems to be saturating all of our available slots now. Is this a change in how MM-less works? Our MM instance ran 33.0.0 and our MM-less instance is running 37.0.0. How many k8s pods am I going to need to scale this up?!?

Hi Druids! We are hosting Druid Office Hours on Thursday, September 24, 2026, from 830–930 a.m. PT. Please use this link to join: meet.google.com/xvj-hxgs-rkf For more details, please see the mailing list thread: lists.apache.org/thread/… Looking forward to seeing some of y’all there! 😄