Apache Druid

On larger clusters, the coordinator and overlord are usually deployed separately. Adding more of them helps for high availability but only one of them is ever the leader.
A couple of ideas on what to check:
• do you have enough capacity to run 121 tasks concurrently, if there are not enough worker slots, the controller process will timeout and fail. 
• Is this on kubernetes? If the coordinator/overlord is failing and there are no errors, perhaps it is exceeding its memory/cpu limit and getting killed by kubernetes. Are the jvm settings in line with the pod resources?
• How busy was the coordinator node in terms of CPU and memory when it failed? Does it have enough resources?

I agree with Sergio that your master nodes do seem underprovisioned for the number of workers you are running.

Hey <@U030K4UM3H7> <@U031A9D19NU> Thanks for helping.
Using a bigger node for master server helped me here.
Btw my druid version is 25.0.0 and its on kubernetes.

how are you running druid ? is ok eks + fargate.

can you check the health of all services, logs of each node.

Screenshot 2023-05-05 at 12.57.25.png

Firstly, i run query node (start-cluster-query-server.sh) with port 8888 and health check  (/). Everything OK

Secondly. I ran master on port 8081. But I am not sure what is heath check i need to set up in target group

In my browser i saw that coordinator use 8888 port for this <http://elb.amazonaws.com:8888/druid/coordinator/v1/lookups|elb.amazonaws.com:8888/druid/coordinator/v1/lookups>

thats router port, not coordinator. router is proxying the request to coordinator

there is a status endpoint that you can point to

I used  different dockerfiles for each node.  Is it ok?

```FROM openjdk:8

ENV DRUID_VERSION 25.0.0
ENV ZOOKEEPER_VERSION 3.4.14

USER root

RUN apt-get update &amp;&amp; \
    apt-get install -y \
    curl \
    build-essential

RUN curl -L <https://cpanmin.us> | perl - --self-upgrade

RUN cpanm FindBin
RUN mkdir -p /tmp \
    &amp;&amp; cd /tmp/ \
    &amp;&amp; curl -fsLS "<https://dlcdn.apache.org/druid/25.0.0/apache-druid-25.0.0-bin.tar.gz>" | tar xvz \
    &amp;&amp; mv apache-druid-$DRUID_VERSION /opt/druid

WORKDIR /opt/druid/

USER root

EXPOSE 8888
EXPOSE 8081
EXPOSE 8082
EXPOSE 8083
EXPOSE 8090
EXPOSE 2181 2888 3888

ENTRYPOINT ./bin/start-cluster-query-server```


I recommend running your configuration on a minikube first to see if basic things like port exposure is working correctly.

Can you check the coordinator logs for the segment id which got pushed to the deep storage
We can get the segment id from the worker log.

image.png

I am not sure, where to find that file, but I checked this log folder, under that checked coordinator-overload, here I see the message as server not found.

Hi, Seems like the historical node might be having some issues

Screenshot 2023-05-08 at 2.42.54 PM.png

If you go to the Druid console -&gt; Services page, do we see the Type `historical`

How is your emitter configured in the docker case? Could this be a docker networking issue? Do you see any errors or warnings in the logs?

Thanks for the response,I have fixed it by moving the emitter config to runtime environment and changing the exporter from docker to binary version

Could you please check your metadata store's `druid_pendingSegments`  table for such an entry?

Also check drui_segments.  Deleting directly off S3 leaves the metadata in the metadata tables.  Better to use kill tasks to clean up data.

<@U0319HD8HEC> <@U0411DE2SS0>, there are not druid_pendingSegments table here?

Screenshot 2023-05-08 at 10.37.45 AM.png

<@U0411DE2SS0>, I did not directly remove from. I did issue an "kill" task and it succeeded.

Oh!  I misunderstood!  And it's not in sys.segments or sys.server_segments?  (Or, for druid_segments and druid_pendingSegments, I meant in the metadata database, usually something like MySQL, Postgres, RDS, etc.)  But strange if it's not in sys.  Restarting coordinators might be worth a try.

I found this <https://support.imply.io/hc/en-us/articles/360009052994-Unable-to-append-new-rows-from-Kafka-indexing-when-segments-already-exist|doc>. That said, this issue disappeared probably due to restarting the coordinator.

Can you please paste the full stack trace?

You could create a new docker image based out of druid:&lt;tag&gt;, and add the MySQL JDBC driver jar file to the new docker image.

Thanks Frank. I already done that and the installation is running now.

If i download the mysql jdbc driver how do i pass it into the setup in the helm chart?

Did you add the mysql in druid system.runtime.properties ?

web console shows many stats around query execution. can you share those stats with us?

Hi Bharat,


for Step #1 you have to create new segment(s) using valid time chunk(s).  If those segments comprise many GB of data then 20 minutes may not be unreasonable to expect to do this operation.

However you do not have to create new segments that are same size as the originals ... if you data is clustered around specific times then you only need re-ingest those times, using valid time partitions ... for example if the data all falls within a couple of one or two hour time chunks on a given day, then you can do a REPLACE/OVERWRITE using just those hours instead of the entire day.

However if your delete is interspersed evenly throughout the data in the segments then you may have no other choice than to do the full replace as you are doing.   It looks like Abhishek is asking for info that would relate to performance tuning here.

Hey <@U031A9D19NU> <@U04C4593VDY> Thanks for looking into the issue.
<@U031A9D19NU> I am using sql API to trigger this query so I don't have any stats from console. Do we have any way to get stats when we trigger via apis?
<@U04C4593VDY> total data size of these segments is ~800mb which is taking 20-30mins. Also, I are having the data on *Day*  level.
Also, i think we are doing the same thing that you are suggesting us. We are not updating whole segment, we are just replacing/overwriting a set of rows with respect to the column name.

<@U03DJ6BJY5A> when using the API to submit queries, you can still use the web console to get stats using the "attach query by ID" feature. It'll pop up a tab in the console with your SQL and a bunch of useful counters

If you get this stuff it'd help us help you :slightly_smiling_face:

Hi <@U056732TEUW>, Do you see any error in the ZK logs? Is there anything else running on the nodes that uses 8080?

If you have backups on your metadata storage, just restore from backup.

No good way, it's important to have backups or HA for metadata DB so it's not a single point of failure.  There was a deprecated <https://druid.apache.org/docs/latest/operations/insert-segment-to-db.html|tool>, but it's deprecated because it's not 100% reliable.  And it won't work at all after 0.15, iiuc.

is this on AWS RDS? You may be able to restore using AWS tools.

It may be possible (albeit difficult) to recover even if you're missing a metadata store backup. The segments on deep storage have all the data but you have a couple of issues if you're missing the metadata store:

• You can't easily tell if some segments were marked unused (manually or by drop rules). However, you can tell what's the newest version for each time chunk, and just load _all_ of them.
• The descriptors (`payload` in metadata store) are missing, so you'd need to generate those and re-insert them into a new metadata store. You may be able to find enough info based on the directory structure in deep storage. The interval, version, datasource should be embedded in there. You may need to set additional fields based on your knowledge of what's supposed to be there

I hope you can find a backup, but if not, the I hope the above clues are helpful!

it likely means that overlord shutdown the task. you should check the overlord logs to see why it shut down that task

checked that, overlord  failed to check the status of task and decided to shut it down.

But interesting part was why the task was failing to respond. something happened on the gcp node where the mm was running. i see some GCP node logs like this. Any one familiar with this ? <@U031A9D19NU> Have you seen something like this ?

```task jbd2/sdb-8:1565 blocked for more than 327 seconds```

we should probably change that message from "Triggering JVM shutdown" to "Triggering JVM shutdown. Check overlord logs to see why the task was shutdown"

Does the overlord log have more detail on why it couldn't communicate with the task? Was it just unable to connect or did it timeout waiting for the communication?

request were timed out, something went wrong on the node, that is all i could figure out, as moving to different resolved the issue.

some config regarding data nodes if it helps:
instance type: `c6gd.4xlarge`
vCPUs: 16
Memory (GiB): 32.0
count: 3 instances

*jvm.config*
```-Xms8g
-Xmx8g
-XX:MaxDirectMemorySize=10g```
*runtime.properties*
```# HTTP server threads
druid.server.http.numThreads=60

# Processing threads and buffers
druid.processing.buffer.sizeBytes=500MiB
## numMergeBuffers should be around numThreads/4
druid.processing.numMergeBuffers=4
## numThreads should be vCPU-1
druid.processing.numThreads=15
druid.processing.tmpDir=var/druid/processing

# Query cache
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=256MiB```


Some thoughts:
What is the query? It may be simple but still use up a significant amount of resources.
Are your data nodes running both MMs and Historicals? Does this coincide with ingestion workloads?  If so, you'll likely need to distribute the CPUs among the `druid.processing.numThreads` for the historical and the `druid.worker.capacity` of the middle managers and consider the memory footprint of each such that all Peons, the MM and the Historical all fit within the 32g of the node.

yep, that is correct. we've got both middle manager and historical running on the same node .. i see, currently we have put all the threads with vcpu-1 for historicals

but the timeouts don't always coincide with the ingestion activities ...

but one thing i forgot to mention was that the qps is also quite high
peak: 300 queries per seconds

if data server CPU only goes to 50–60%, you are doing high QPS and getting some timeouts, and you have already maxed `druid.processing.numThreads` then I suggest spending some time figuring out why CPU isn't at 90–100%

It sounds like your system is at/near max load (based on the timeouts)— ordinarily we want CPU to be maxed when this is the case, not hovering at 50–60%. Could mean some inefficiency somewhere.

I would first try adding more data servers and seeing if that improves your situation with the timeouts. If not then it's likely an inefficiency at the Broker. If it does go up then it's likely an inefficiency at the data servers

Once you know that, then you can poke deeper

i see ..understood. will increase the data node count to 4 and monitor.
wondering if there's any other metrics / stats we should be monitoring

Screenshot 2023-05-11 at 2.29.17 PM.png

Screenshot 2023-05-11 at 2.28.51 PM.png

Screenshot 2023-05-11 at 2.28.39 PM.png

sharing a few which we have been monitoring currently.

also sharing query nodes config (2 r6g.xlarge - 4 vcpu, and 32 gigs of mem)
```## broker jvm.config
# HTTP server settings
druid.server.http.numThreads=60
druid.server.http.maxSubqueryRows=600000

# HTTP client settings
druid.broker.http.numConnections=50
druid.broker.http.maxQueuedBytes=10MiB

# Processing threads and buffers
druid.processing.buffer.sizeBytes=500MiB
druid.processing.numMergeBuffers=6
druid.processing.tmpDir=var/druid/processing

# Query cache disabled -- push down caching and merging instead
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false```
and broker jvm.config
```-server
-Xms18g
-Xmx18g
-XX:MaxDirectMemorySize=6g```

&gt; i see ..understood. will increase the data node count to 4 and monitor.
looking good so far! thanks <@U0344FW86DD>, <@U030K4UM3H7>


&gt; I would first try adding more data servers and seeing if that improves your situation with the timeouts. If not then it's likely an inefficiency at the Broker. If it does go up then it's likely an inefficiency at the data servers
will continue monitoring, but i guess early signs points toward "inefficiency at the data servers" :thinking_face:

:face_holding_back_tears: i spoke too soon. got a spike again .. so something to do with query nodes then

bumping this up ^ <@U0344FW86DD> / <@U030K4UM3H7>

Is there a particular query that does this? Is there a large join or high cardinality group by occurring in such queries? How many query nodes do you have? Are the <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#connection-pool-sizing|http threads in the historicals high enough to accommodate all broker numConnections>?

I went back and read your configs...According to that recommendation, your historicals should have about 110 for httpNumThreads. It is possible that your data nodes are running out of connection threads when both brokers are dealing with high concurrency.

&gt; Is there a particular query that does this
not that i can tell :thinking_face:  the most frequent one looks like a simple query with no join
```select avg(ratings) from ratings where user = '' and __time in last month```


for the number of threads, actually we're a bit confused since there are 2 sets of documentations around this .. one on <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html>
```For Historicals, druid.server.http.numThreads should be set to a value slightly higher than the sum of druid.broker.http.numConnections across all the Brokers in the cluster.```
```On the Brokers, please ensure that the sum of druid.broker.http.numConnections across all the Brokers is slightly lower than the value of druid.server.http.numThreads on your Historicals and Tasks.```
and on <https://druid.apache.org/docs/latest/configuration/index.html>
the doc says to use this formula for `max(10, (Number of cores * 17) / 16 + 2) + 30` for historical, indexer and broker
so since our
• historical has 16 vCPUs it comes out to be 49 and for
• query nodes has 4 vCPUs it comes out to be 40

The historical cpu based formula is for its processing threads, the other calc is for http threads. 

Screenshot 2023-05-17 at 10.47.59 AM.png

Screenshot 2023-05-17 at 10.47.44 AM.png

i thought for processing threads it's just vcpus-1

Right, for Historicals, processing threads v CPU’s -1
Http threads = sum of broker num connections +10 or so for internal communication. 

Screenshot 2023-06-12 at 10.19.35 AM.png

Screenshot 2023-06-12 at 10.19.59 AM.png

hi, <@U030K4UM3H7> in order to further investigate we think that the culprit is a query which is scanning many segments (30 days with daily granularity) and for row level data instead of aggregated data.
we've also captured more metrics which might be helpful. during the issue, the query wait time, segment time and segment scan pending  -- all of these metrics on historical have a spike. also all the requests are failing on historical with timeout error and not of broker nodes.

Glad you found the culprit.  Saturated Historicals can have this behavior because the processing threads are busy and cannot handle the simple small requests. Some implementations use <https://druid.apache.org/docs/latest/configuration/index.html#query-prioritization-and-laning|Query Laning >and/or <https://druid.apache.org/docs/latest/operations/mixed-workloads.html#historical-tiering|Historical tiering> to separate workloads in order to avoid this kind of problem. The number of segments might be the problem, so <https://druid.apache.org/docs/latest/operations/segment-optimization.html|segment size optimization> is also good to look at. Also, like you mentioned, creating an aggregate table for other use cases might make a lot of sense since it would allow you to aggregate once and then query it many times using much less resources over all vs. aggregating the detail across broad timeframes whenever the long-term queries are executed.

You can <https://druid.apache.org/docs/latest/configuration/index.html#enabling-metrics|enable metrics emitter >and add  <https://druid.apache.org/docs/latest/operations/metrics.html#jvm|JVMMonitor> in the `druid.monitoring.monitors`. There are a few memory related metrics that are emitted for each JVM.

Thanks <@U030K4UM3H7>
to be more specific in query how can I check each running task for particular data source(index task) memory consumption?
I mean which metrics I can use for that here?

I'll have to test to see the actual values, but these metrics will have dimensions for `service` and `host`  which should at least identify the `Peon` and the `host ip or host name` it is running on. I don't think you can do this by taskId or by dataSource.

Okay <@U030K4UM3H7>!
but if I want to check any particular if task is running how much memory that particular task consuming can atleast these details be collected?

That would definitely be an interesting dimension for Peon JVM monitor to produce. According to the docs it does not have it. But it sounds like a great improvement to the metrics. Would you like to<https://github.com/apache/druid/issues| create a github issue> with the improvement description?

<@U030K4UM3H7>
I have created an issue for this. Thaks for this.
<https://github.com/apache/druid/issues/14253>

Screen Shot 2023-05-09 at 12.19.06.png

Screen Shot 2023-05-09 at 12.20.48.png

For anyone else who sees this problem; we did a rolling restarted of our Zookeeper nodes which seemed to fix the issue.

Hmmm... what version are you on?
Have you tried using `druid.coordinator.loadqueuepeon.type=http`
to avoid the dependency on zookeeper for load/drop instructions?

We are on 0.23.0.  I was not aware of this config option; we are using the default value which I see was changed from `curator` to `http` in the latest version. Interesting. Thank you.

<@U030K4UM3H7> - regarding your answer is druid can work without zookeeper when setting
druid.coordinator.loadqueuepeon.type=http ?

There is work being done to fully remove zookeeper. But this parameter  avoids zk as the communication mechanism between the coordinator and the historical nodes. 

Please check this issue for the currently planed features for release: <https://github.com/apache/druid/issues/14064>

The window functions need some additional work prior to release.

Window functions have an experimental tag and won’t be on by default. There’s still some more work we have to do

one more thing: is there a documented api endpoint to set the workerCategorySpec? The docs only mention doing it via UI, but I like to script it.

`druid.server.tier` is more for segment assignment so I wouldn't expect that to stick here

`druid.worker.category` OTOH I would expect to be honored… I see the same code in Indexer and MM for setting it from the props file

the docs for `workerCategorySpec` are at <https://druid.apache.org/docs/latest/configuration/index.html#equal-distribution-with-category-spec>

you'd use the select strategy type `equalDistributionWithCategorySpec`

(otherwise `workerCategorySpec` is ignored)

Thanks Gian. I really miss a “default” tier for everything and then listing exceptions. And I tried both, the indexer stays in the `_default_tier`. which is ok, as long as I could pin all `index_kafka` to it and everything else to a default tier with a different name. Is there a comprehensive list of tasks, if I have to spell it out?

hmm do you have a "repro" for "Indexer stays in `_default_tier`"? like, some steps to follow on a fresh download that exhibit things not working as you'd expect?

I ask b/c I haven't tried personally but it seems like the code is there for it to work…

If it's a bug simplest thing is to just fix it

On your other question "Is there a comprehensive list of tasks" — not that I'm aware of in the docs, but I think you can get one from the Overlord itself by submitting a task with an invalid `type`. IIRC it'll list out all the valid types in the error message

regarding reproducable:
```          {% if item.tier is defined %}
          druid_server_tier={{ item.tier }}
          druid_worker_category={{ item.tier }}
          {% endif %}```
I feed that into `apache/druid:25.0.0` docker image. `middleManger`  and `historical` respect it, indexer doesn't.

```$ curl -s -X POST druid-router.service.dw2.consul:8888/druid/indexer/v1/task -d'{"type":"reflection"' -H "Content-Type: application/json"|grep -o -E "\[.*?\]"|head -1|grep -o -E '\w+'|sort|jq --raw-input . | jq --slurp .
[
  "archive",
  "compact",
  "index",
  "index_hadoop",
  "index_kafka",
  "index_parallel",
  "index_realtime",
  "index_realtime_appenderator",
  "index_sub",
  "kill",
  "move",
  "noop",
  "partial_dimension_cardinality",
  "partial_dimension_distribution",
  "partial_index_generate",
  "partial_index_generic_merge",
  "partial_range_index_generate",
  "query_controller",
  "query_worker",
  "restore",
  "single_phase_sub_task"
]```
maybe it's late, but you don't make it simple. How about an api call that just gives me this output?

follow up: looks like `indexer` is observing category, but not showing in router overview.

&gt; maybe it's late, but you don't make it simple. How about an api call that just gives me this output?
ouch, I guess we deserve that :slightly_smiling_face:

yeah, that'd be nice, as opposed to relying on an error message!

&gt; follow up: looks like indexer is observing category, but not showing in router overview.
ah, good to know! that probably means it isn't in `sys.servers` (or it is but the UI isn't showing it)

What is your config for <https://druid.apache.org/docs/latest/configuration/index.html#task-logging|task logging>? and <https://druid.apache.org/docs/latest/configuration/index.html#log-retention-policy|log retention> ?

I'm also wondering which specific directory you mean and what files are still being held?

Check the coordinator log to see what happened, it seems like it isn't available.

It looks like the request that's failing is for an `isObjectInBucket` call

This calls `boolean doesObjectExist(String bucketName, String objectName)` on the S3 client without attaching SSE key

Maybe, for this call, real S3 doesn't need an SSE key but this emulated S3 does

If that's what's going on, this will fix it: <https://github.com/apache/druid/pull/14290>

The issue also affect real S3, I created an issue yesterday on github with more details and reproduction steps: <https://github.com/apache/druid/issues/14284>. I was able to reproduce the issue with a AWS S3 bucket.

ok, thanks! I just commented with a link to <https://github.com/apache/druid/pull/14290>

Hi all, I am getting error from druid for monthly data query. can anyone please help.

Try "2023-01-31T18:00:000Z/2023-03-31T18:00:000Z" ... not sure if this will help, but I noticed your time interval did not match the normal specs (three zeros in front of the 'Z', and the end time is excluded so you state the beginning of the next period)

I bet it's got something to do with the `granularity` + `origin`. Maybe try something similar but using `timeZone` instead of `origin`?

Was it able to process any data? Is this an ingestion from S3? Seems like an S3 access problem. Maybe <https://druid.apache.org/docs/latest/development/extensions-core/s3.html#s3-permissions-settings|permissions on the S3 bucket. >
Can you share the load spec?

```{
  "type": "index_parallel",
  "spec": {
    "dataSchema": {
      "dataSource": "test",
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "!!!_no_such_column_!!!",
            "missingValue": "2010-01-01T00:00:00Z"
          },
          "dimensionsSpec": {
            "dimensions": null
          }
        }
      },
      "metricsSpec": null,
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": {
          "type": "none"
        },
        "rollup": false,
        "intervals": null
      },
      "transformSpec": {
        "filter": {
          "type": "not",
          "field": {
            "type": "in",
            "dimension": "ID",
            "values": ["1", "2"],
            "extractionFn": null
          }
        },
        "transforms": null
      }
    },
    "ioConfig": {
      "type": "index_parallel",
      "firehose": {
        "type": "ingestSegment",
        "dataSource": "test",
        "interval": "2023-04-25T20:00:00.000Z/2023-04-25T21:00:00.000Z",
        "segments": null,
        "filter": null,
        "dimensions": null,
        "metrics": null,
        "maxInputSegmentBytesPerTask": 555157286400
      },
      "appendToExisting": false
    },
    "tuningConfig": {
      "type": "index_parallel",
      "maxRowsPerSegment": null,
      "maxRowsInMemory": 1000000,
      "maxBytesInMemory": 0,
      "maxTotalRows": null,
      "numShards": null,
      "partitionDimensions": [],
      "indexSpec": {
        "bitmap": {
          "type": "concise"
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "maxPendingPersists": 0,
      "maxNumConcurrentSubTasks": 5,
      "buildV9Directly": true,
      "forceGuaranteedRollup": false,
      "reportParseExceptions": false,
      "pushTimeout": 0,
      "segmentWriteOutMediumFactory": null,
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0
    }
  },
  "context": {},
  "dataSource": "test"
}```
Same approach works fine for datasource with less data, so i s3 is accessible , but it might be timeout issue since many segments are getting ingested  continuously for longer duration?

tried with another datasource with many segments for 1 day. could see  below error in logs- any leads to fix this kind of issue?
```2023-05-15T04:39:01,821 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes:
  WARNING: A HTTP GET method, public void org.apache.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type.
2023-05-15T04:39:02,318 ERROR [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Exception when stopping method[public void org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner.stop()] on object[org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner@6a2badb1]
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_232]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_232]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_232]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_232]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.stop(Lifecycle.java:463) [druid-core-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:366) [druid-core-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:367) [druid-services-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.cli.Main.main(Main.java:118) [druid-services-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
Caused by: java.util.concurrent.RejectedExecutionException: Service not started.
	at org.apache.druid.java.util.emitter.core.LoggingEmitter.emit(LoggingEmitter.java:94) ~[druid-core-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.java.util.emitter.core.ComposingEmitter.emit(ComposingEmitter.java:57) ~[druid-core-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:67) ~[druid-core-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.java.util.emitter.service.ServiceEmitter.emit(ServiceEmitter.java:72) ~[druid-core-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner.stop(SingleTaskBackgroundRunner.java:226) ~[druid-indexing-service-0.15.0-incubating-iap10.jar:0.15.0-incubating-iap10]
	... 8 more
Finished peon task```

I just noticed that you are on a very old release. For batch jobs such as this Druid 25.0's <https://druid.apache.org/docs/latest/multi-stage-query/index.html|SQL Based Ingestion> has proved to be more efficient at processing large loads. You can use the <https://druid.apache.org/docs/latest/multi-stage-query/reference.html#extern|EXTERN> table function to access S3 buckets for ingestion directly from SQL.

Seconding the recommendation to upgrade and use SQL ingestion :slightly_smiling_face:

Anyone implemented the column level access security with LDAP integration?  TIA

Perhaps you have the loadList property specified in some other properties file? The service specific properties files override the properties specified in the common properties file.

since i use bin/start-single-server-medium script i have configured loadlist at "druid/apache-druid-25.0.0/conf/druid/single-server/medium/_common/common.runtime.properties" , don't know if this is true

just tested this with the nano config. I updated the `conf/druid/single-server/nano-quickstart/_common/common.runtime.properties` file with
```druid.extensions.loadList=["druid-hdfs-storage", "druid-kinesis-indexing-service", "druid-datasketches", "druid-multi-stage-query", "postgresql-metadata-storage"]```
Here's what the streaming load screen looks like:

also tried with the `medium` deployment script with the same results.

Hi Everyone, I am trying to insert data from postgres to druid using task spec