Apache Druid

what do you mean by "not seeing"?

In terms of the load rule itself, it should be seen immediately

In terms of the tier existing, that would depend on whether Historical servers are online with that tier configured or not

Screenshot 2023-04-20 at 3.01.51 PM.png

Hm… I can no longer create this rule from the UI

Screenshot 2023-04-20 at 3.03.01 PM.png

In Druid 25, I can only do this: which seems different.

yeah, that's certainly different; the top one would load 2 replicas in each tier. the bottom one would load 2 replicas only in `_default_tier` (the lower rule would be skipped since only the first matching rule applies)

<@U030W50CNAC> do you know if something changed in the ui?

No, nothing changed in the UI code. I think what is going on is that the UI is loading the tiers that are reported in the system and only showing the "Add tier" button for a rule if you have unassigned tiers

Hm… POST command to Druid failed as well:
```2023-04-20 16:06:13,279 cleric.lib.druid_api INFO function=post_datasource_rules url=<https://my-running-druid/druid/coordinator/v1/rules/_default>, message="Sending request" payload=[
  {
    tieredReplicants: {
      _default_tier: 2,
      _default_tier_2: 2
    },
    type: "loadForever"
  }
]

urllib.error.HTTPError: HTTP Error 500: Internal Server Error
and empty body string```


I think what happened in the past with us is that one of us perform the update using the API, not the UI. I saw the custom function created in our automation.

But currently that API call is also broken, LMAO

image.png

you can also check how many tiers you have by doing `GET /druid/coordinator/v1/tiers`

the UI will not let you assign replicas to tiers that do not exist and I bet in your case that API will report just one tier. You could, of course, bypass the UI and just use the API to set whatever load rules you want.

ok, I fixed our own tooling, submitting from the API indeed works.

Screenshot 2023-04-20 at 4.42.32 PM.png

And coordinator picked it up almost immediately:
```druid-coordinator-2 druid-coordinator 2023-04-20T23:43:18,621 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier_2] : Assigned 2110 segments among 20 servers
druid-coordinator-2 druid-coordinator 2023-04-20T23:43:18,621 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Assigned 110 segments among 1 servers```

<@U0344FW86DD> is there a catch 22 here? Does a tier not get reported by the `GET /druid/coordinator/v1/tiers` API until it has some segments in it by having some load rule assign segments to it?

yes, there is, the `/druid/coordinator/v1/tiers` API only shows tiers that have data

(otherwise they are essentially invisible to the Coordinator)

IMO a good UI approach would be to have the kind of dropdown that still lets you type stuff into it

Oh man. That UI is totally catch 22ed right now

The reason it is the way that it is (and not a suggestible input box which is more common in the console) is because the underlying model is a key value json object and it means that you can not represent a UI state with it directly. Like imagine `{"tier": 2, "tier2": 2}` and then you start typing and delete the `2` in `tier2` at that moment `{"tier": 2, "tier": 2}` is not a valid state so without a special intermediate UI state the UI will appear to collapse.

I assume there isn't an API which will give the console all the possible tiers (as defined by the historical servers that are configured). What happens if a replication rule assigns data to a tier that does not exist (historical wise)?

&gt; I assume there isn't an API which will give the console all the possible tiers (as defined by the historical servers that are configured)
I think you can get them from `sys.server_segments`. however a suggestible input box would still be good, b/c people may want to configure load rules before they turn on the new servers

&gt; What happens if a replication rule assigns data to a tier that does not exist (historical wise)?
the Coordinator will start logging warnings about how it can't find servers for the tier

I assume you mean `sys.servers` . Also thank you. I have a clear idea on how to improve the UI

Made this PR, that will resolve the catch-22 (although more work should be done to make the UI perfect): <https://github.com/apache/druid/pull/14143>

It could be too many segments.
1. How big is your datasource: (GB and # segments)?
2. What is the # segments scanned metric show?
3.  How many Middle Historicals do you have?

This is all I got from Historicals
```druid_emitted_metrics{datasource="ds",host="druid-historicals-0:8088",id="1ad24d9f-8c26-4778-95f0-405372400dce",metric_name="query-bytes",service="druid-historical"} 2.0327946e+07
druid_emitted_metrics{datasource="ds",host="druid-historicals-0:8088",id="1ad24d9f-8c26-4778-95f0-405372400dce",metric_name="query-cpu-time",service="druid-historical"} 6.799292e+06
druid_emitted_metrics{datasource="ds",host="druid-historicals-0:8088",id="1ad24d9f-8c26-4778-95f0-405372400dce",metric_name="query-segment-time",service="druid-historical"} 2232
druid_emitted_metrics{datasource="ds",host="druid-historicals-0:8088",id="1ad24d9f-8c26-4778-95f0-405372400dce",metric_name="query-segmentAndCache-time",service="druid-historical"} 2233
druid_emitted_metrics{datasource="ds",host="druid-historicals-0:8088",id="1ad24d9f-8c26-4778-95f0-405372400dce",metric_name="query-time",service="druid-historical"} 13049
druid_emitted_metrics{datasource="ds",host="druid-historicals-0:8088",id="1ad24d9f-8c26-4778-95f0-405372400dce",metric_name="query-wait-time",service="druid-historical"} 10118```


I don't know if you have to set up additional emitter parameters to get "segment" metrics ...

Have you run compaction?  350GB / 6000 = ~60MB per segment size on average, which is small.   Is your 98% segment scan time very low?

(query/segment/time)

Maybe I'm missing logging. Have trouble setting that properly on k8s for queries

I have it setup:
```"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"},```
Just can't seem to extract from my emmiter

<https://druid.apache.org/docs/latest/configuration/index.html#metrics-monitors>

Try enabling HistoricalMetricsMonitor and SegmentStatsMonitor

That would help :slightly_smiling_face:  Thanks

(I'm fairly new to this myself ... the <http://learn.imply.io|learn.imply.io> classes are really good hands-on tutorials for learning about Druid at a medium level of technical depth ... I strongly recomend going through them if you have the time :slightly_smiling_face: ... there is a Metrics class in there that I took recently)

these segments are spread around 6 servers

Quantile performance is linked with size of K ?

It’s provides a size-accuracy trade-off

because it need more memory and merging happen on disk ?

Larger `k` means the sketch is bigger footprint, so there's more compute work to do when adding items and merging sketches

In my experience quantile sketches are pretty fast at the default `k` value, although if you crank up `k` then perf will drop accordingly

Lower K value gives a lot of variance (from the actual ) in term of actual answers. So we thought we will bump up the K to get more accurate answer .  On top of that the answer is not consistent it always give a different answer which kind of gives impression to some use cases that something is wrong and answer is not correct.

This is pretty visible in histogram queries (bucket the counts), where  items keeps changing it's buckets. And this changes answer in some bucket +-1k. which is pretty large.

You generally don't want to be cranking up `k` on sketches solely to try to get the result to be consistent from run to run— it's better to educate your users that the computations are approximate and may vary from query to query. Otherwise you run into this issue where high `k` means slow perf (and diminishing returns on accuracy)

If the approximation errors are problematic beyond that, though, then higher `k` is a good way to fix it

I wonder if the ideal `k` for you is somewhere between default and the (presumably much higher) value that causes the results to be "stable"

yeah, that is what we did,  ran the query with different-2 k and choose a reasonable K which gives acceptable answer with in acceptable  time.

Hi Psss - have a look at the examples to see if any of them are appropriate <https://druid.apache.org/docs/latest/multi-stage-query/examples.html#insert-for-reindexing-an-existing-datasource>

thankyou <@U030MBK46BD> i have a question here i want to add data into existing datasource accordin to the doc REPLACE for replacing a specific time segment if that segment doesn’t exist will it create that segment?

That’s right - replace will create if it did not exist <https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#overwrite-data-with-replace>

+1 for what Kyle says ... here's an example you can use on the Wikipedia demo data set, I am inserting a new time chunk for today that doesn't exist:

```REPLACE INTO wikipedia_part2 OVERWRITE
WHERE __time &gt;= TIMESTAMP '2023-04-21 00:00:00' AND __time &lt; TIMESTAMP '2023-04-22 00:00:00'
SELECT   CURRENT_TIMESTAMP "__time",
  "isRobot",
  "channel",
  "flags",
  "isUnpatrolled",
  "page",
  "diffUrl",
  "added",
  "comment",
  "commentLength",
  "isNew",
  "isMinor",
  "delta",
  "isAnonymous",
  "user",
  "deltaBucket",
  "deleted",
  "namespace",
  "cityName",
  "countryName",
  "regionIsoCode",
  "metroCode",
  "countryIsoCode",
  "regionName"
from wikipedia_part2 WHERE __time &gt;= TIMESTAMP '2016-06-27 00:00:00' AND __time &lt; TIMESTAMP '2016-06-28 00:00:00'
PARTITIONED BY HOUR```
Please note -- the __time you are selecting to insert must match the time range you specify in the OVERWRITE clause ... otherwise you will get this error:

&gt; <https://druid.apache.org/docs/latest/multi-stage-query/reference.html#error_InsertTimeOutOfBoundsFault|InsertTimeOutOfBoundsFault>: Query generated time chunk [2023-04-21T11:00:00.000Z/2023-04-21T12:00:00.000Z] out of bounds specified by replaceExistingTimeChunks (Stack trace)
&gt; Failed task ID: query-94220bfb-f69c-430c-90a0-fb05a37684af (on host: localhost:8100)
&gt; Debug: get query detail archive
Though otherwise it does not matter what the source of your data is for the SELECT statement ... in my example I'm just grabbing some random data from the table because I know there are records there ... I could have pulled that data from any datasource, or an EXTERN()

I think I will need to cast 0 to archer type.
But still trying to understand is '' getting converted to integer type and becoming 0 ?

I wonder, are you setting `druid.generic.useDefaultValueForNull = false` in your configs?

If not, I would recommend it, as it makes null handling more SQL compatible

You can read more here <https://druid.apache.org/docs/latest/querying/sql-data-types.html#null-values>

Thanks <@U0344FW86DD>. I read through the documentation just one additional question.
If druid.generic.useDefaultValueForNull=true which is by default then while ingesting a dataset from Hadoop which has 2 dimension column(1 string and 1 int ) and 1 metric column.
In my case I am observing Null values in integer type dimension are getting converted to 0 but the null values in string dimension are still being kept as Null after being ingested into druid. Is this expected behaviour ?
Based on the article my understanding was, null should be converted to 0 for int/long columns and null values in string dimension should be converted to empty string in druid dataset ?

in the default-value mode, Druid treats NULLs and empty strings interchangeably

sometimes you will see `null` and sometimes you will see `""` depending on situation

most common is you will see `null` though (for both `null` string and `""` empty string)

<https://druid.apache.org/docs/latest/operations/metrics.html#query-metrics|Query metrics> will help a lot. You can <https://druid.apache.org/docs/latest/configuration/index.html#enabling-metrics|enable them> and choose an emitter from the emitter extensions:
• <https://druid.apache.org/docs/latest/development/extensions-contrib/ambari-metrics-emitter.html|ambari-metrics-emitter>
• <https://druid.apache.org/docs/latest/development/extensions-contrib/graphite.html|graphite-emitter>
• <https://druid.apache.org/docs/latest/development/extensions-contrib/statsd.html|statsd-emitter>
• <https://druid.apache.org/docs/latest/development/extensions-contrib/kafka-emitter.html|kafka-emitter>
• <https://druid.apache.org/docs/latest/development/extensions-contrib/opentsdb-emitter.html|druid-opentsdb-emitter>
• <https://druid.apache.org/docs/latest/development/extensions-contrib/influxdb-emitter.html|druid-influxdb-emitter>
• <https://druid.apache.org/docs/latest/development/extensions-contrib/prometheus.html|prometheus-emitter>
You can also use <https://druid.apache.org/docs/latest/operations/request-logging.html#configure-request-logging|request logging> and query ids to identify the query and the resources each of them use throughout the cluster.

Also useful are flame graphs so you can see what the CPU is being spent on: <https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs>

I believe you need addition storage drive to the addtional  node or you can add a new disk to existing node to increase the capacity of single server but having new node will be  good in terms of HA  .

They can use the same disk drive as long as they are using separate directories on that disk

(They cannot share a local data directory)

That being said it is generally a good practice to dedicate a drive or filesystem to each server, for perf reasons

Thanks <@U0344FW86DD> - asume I am setting 2 historical each have it's own local disk drive , from druid poiint of view - each historical holding the same segments data , or is each of historical holding 50% of segment data ...
is the historical cluster work as replicated / or sharding ?

if you have two servers, and replicas = 2 (the default), then they will each have a copy of the same data

and when a query hit druid cluster it will be forward to one of the historical ..
or does it will use both to fetch from each part of the query result set ?

In general, Druid will use both servers, if possible

Additionally, even though I only have 9 tasks in the running STATE, summed up usage from Services tab is 12 (out of 12) slots

The slots are definitely bogus, looking at output of `/druid/indexer/v1/workers` I can see same supervisor tasks, different task IDs on multiple workers/indexers

We had all components but indexer + historical (kafka included) doing a rollout restart for node upgrades over the weekend, but that probably shouldn’t end up with overlord getting confused on what tasks are still running (especially given other task is in SUCCESS state)

Overlord seems to realise that something is wrong, reporting
```"2023-04-24T11:07:10,316 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Assigned a task[index_kafka_telemetry-872b103e-293f-428c-90d3-ea4ddb310e8c_82e3bc7c65576a5_mmjcjfhh] that is already pending!"```

```2023-04-24T11:07:10,338 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[192.168.5.4:8091] wrote RUNNING status for task [index_kafka_telemetry-872b103e-293f-428c-90d3-ea4ddb310e8c_82e3bc7c65576a5_mmjcjfhh] on [TaskLocation{host='null', port=-1, tlsPort=-1}]```

And then two hours later:
```2023-04-24T13:07:13,042 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[192.168.5.4:8091] wrote RUNNING status for task [index_kafka_telemetry-872b103e-293f-428c-90d3-ea4ddb310e8c_82e3bc7c65576a5_mmjcjfhh] on [TaskLocation{host='192.168.5.4', port=8091, tlsPort=-1}]```

Is there perhaps some task state held in zookeeper? I’ve noticed before that when I do a rollout restart of my zookeeper cluster, most druid components restart themselves (as IP address of the zookeeper nodes change and they don’t try to resolve it again? something to debug on my side). This time one of the indexer nodes wasn’t restarted, I wonder if it was still having some ghost connection to the old ZK cluster, and that affected something.
I can see that pre-restart none of the indexing tasks are taking over 1 hour (as configured) but post-restart some tasks are taking 3 hours (on the clock) or 4 hours (on the clock) and still succeeding.

The sketch library doesn’t check system endianness though

So if you take both as chararrays or similar, we can assume they translate smoothly?

given your above assertion has been verified

I’ve generally encoded as base64 to transport between systems

but as long as you’re running on the major version of the sketches package, they’re both using the java sketches library

and if you find binary incompatibilities loading then in c++ or python, please file a bug with us (or reach out in <#C04TB8M30DV|> first)

Seconding the recommendation to use base64 for ingest! I believe you just need to do that and set `isInputThetaSketch: true` on the aggregator

5 million is a guideline. So yes, it is okay to have bigger segments. You can monitor the performance difference and tune the size for your workload.

Similar <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1681759613889479|thread>: which suggests lowering `maxRowsPerSegment`
Also if you have a lot of intermediate persists you may want to set `maxColumnsToMerge` (see <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1663951074038569?thread_ts=1663936671.637089&amp;cid=C0309C9L90D|this thread >for that conversation).

datasource size is big(~1tb with out replication) total rows are 76billion. avg segment rows 7Million max is 20Million.

table-level metadata on column cardinalities isn't available, so if you need to know it, then you would need to use a function like this

perhaps you could try caching the result?

In general I also suggest using `APPROX_COUNT_DISTINCT_DS_HLL`, since it's somewhat faster. I don't think it will knock minutes off your query, but it should help a little.

this works when i try for one day.. we have 30 days data there it takes long time.. though for now got the needed data

Hi <@U0344FW86DD>
Any help in this is much appreciated.
we need to find cardinality of each column for for 30 days . is there any way we can find out this number with out bugging down historicals?

As currently running the below query we see high cpu usage in the few historicals and it comes down. datasource contains 30 days data.
```SELECT APPROX_COUNT_DISTINCT_DS_HLL("column1") FROM "datasource1" ```
however for one day its runs fine
```SELECT APPROX_COUNT_DISTINCT_DS_HLL("column1") FROM "datasource1" 
WHERE "__time" &gt; '2023-05-27T02:00:00.000Z' and  "__time" &lt; '2023-05-28T00:00:00.000Z'```


Looks like you’re crunching too much data for your historicals sizing.

Have you considered having a rolledup datasource with data pre-aggregated from the base one?

... and if you are doing "approximate" distinct counts anyway, does your data change so much from hour to hour and day to day that you can't run sample count distincts for several random hourly periods within your 30 day window?

if you enable request logging you can see the query string (SQL or JSON) as well <https://druid.apache.org/docs/latest/operations/request-logging.html>

It seems like you could do that ... however what is the final goal here?  Just to run compaction?  Have you ruled out just running regular auto-compaction instead?

As far as compaction performance itself, I think that is largely based on the size of the segments being compacted.  It can vary  by the type of data in additional to volume ... I have seen typical 500MB segments with 3-5m records taking in the 2-4 minute range.  I have also seen smaller segments take 10-15 seconds.  But with auto-compaction running once every 30 minutes (by default) using only the allocated worker slots, this keeps up with the ingestion and doesn't overuse resources on the cluster.

The most common reason for 143 exit code is the `oom-killer`, which you could confirm by looking at `dmesg`. If that's the case, it's most commonly due to one of the processes being configured to use more memory than is actually available in the pod. So I'd suggest double-checking your memory settings, like `Xmx` and the amount of memory allocated to the pod

In Druid 25+ you can also use the included `bin/start-druid` script with the `-m` parameter to automatically set memory settings to stay within a certain budget

And it finished with :
```2023-04-25 12:33:34,976 [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcheroverlord] level: ERROR category: org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - message: Expection while watching for NodeRole [OVERLORD].
java.lang.RuntimeException: java.lang.InterruptedException
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:144) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.concurrent.LifecycleLock.awaitStarted(LifecycleLock.java:245) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:257) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:237) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367) ~[?:?]
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:139) ~[druid-core-25.0.0.jar:25.0.0]
	... 8 more
2023-04-25 12:33:44,977 [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatchercoordinator] level: ERROR category: org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher - message: Expection while watching for NodeRole [COORDINATOR].
java.lang.RuntimeException: java.lang.InterruptedException
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:144) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.concurrent.LifecycleLock.awaitStarted(LifecycleLock.java:245) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.keepWatching(K8sDruidNodeDiscoveryProvider.java:257) ~[?:?]
	at org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatcher.watch(K8sDruidNodeDiscoveryProvider.java:237) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1367) ~[?:?]
	at org.apache.druid.concurrent.LifecycleLock$Sync.awaitStarted(LifecycleLock.java:139) ~[druid-core-25.0.0.jar:25.0.0]
	... 8 more
Finished peon task```

I am not sure what is going on, it seems like something specific to the k8s announcing functionality. I know that has received a bunch of attention recently. You might get some luck over at the <#C04Q0047B4M|> channel

The issue seems to be related to a malformed request to k8s which I have no time right now to debug.
I ended giving up and moved to zk
I can always restore setting to k8s for testing purposes if needed

Hi Amantha - the console executes REST API requests to the different services.
Since you can see the console, this means the `router` service has started ok, but it looks like the `overlord` and `coordinator` services might be having trouble.

Have a look at the `Services` tab - any hints there?

Any hints in the router log? This could be something fundamental like the ZooKeeper or hostname resolution

requests are failing with 500 so you should check the logs of druid services. Especially coordinator, overlord, broker - in that order

WARN [qtp436328432-84] org.eclipse.jetty.server.HttpChannel - /druid/v2/sql
org.apache.druid.java.util.common.ISE: No default server found!

What version of Druid and ZooKeeper please?

Ssl  14:16   0:35 /usr/bin/java -server -Xms4500m -Xmx4500m -XX:+ExitOnOutOfMemoryError -XX:+UseG1GC -Duser.timezone=UTC -Dfile.encoding=UTF-8 -<http://Djava.io|Djava.io>.tmpdir=var/tmp -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dderby.stream.error.file=var/druid/derby.log -cp /usr/share/apache-druid-0.22.1/conf/druid/single-server/small/coordinator-overlord:/usr/share/apache-druid-0.22.1/conf/druid/single-server/small/_common:/usr/share/apache-druid-0.22.1/conf/druid/single-server/small/_common/hadoop-xml:/usr/share/apache-druid-0.22.1/conf/druid/single-server/small/../_common:/usr/share/apache-druid-0.22.1/conf/druid/single-server/small/../_common/hadoop-xml:/usr/share/apache-druid-0.22.1/bin/../lib/* org.apache.druid.cli.Main server coordinator

issue was fixed, it was a storage issue. thanks all

What is your setup? How many task slots do you have? When is this happening?
Theory: If this is happening at taskDuration, it could be that you don't have enough task slots available to accommodate the new set of tasks. At task duration, a checkpoint is done where all reading tasks transition to a publishing state and a new set of tasks are created to continue consuming. So you need to have enough task slots available to fit 2 sets of tasks (including replicas if you use them) for the transition to go smoothly. 45 minutes is a lot for a task to stay in the pending state, what else is going on in your cluster? Are the task slots busy with other work?

I have 3 indexers, each one with 16 slots (total of 48) and 16 slots are available right now.
That’s not exactly ideal, but our current architecture is a bit specific - out of 32 supervisor tasks that we have, only 4 are actually ingesting any data from kafka, and the other tasks are just idling with nothing to do, and they have no data to publish at the end of their runtime.
As for pending tasks - looking at the metrics, I don’t see an increase in the druid_task_pending_count metric while those messages were being logged.

For our usecase having enough slots to fit 2 sets of tasks just for the publishing transition is a bit cost prohibitive

The first task `index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_cccocake` was shut down just 20 seconds after it was started by the indexer, the previous task for that supervisor `index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_hckkdgha` was started by the indexer here:
```2023-04-25T00:31:37,275 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task[index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_cccocake] started.```
It actually started consuming data from kafka here (45 minutes later):
```2023-04-24T23:31:36,199 INFO [[index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_hckkdgha]-threading-task-runner-executor-7] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]```
and then finished reading data 1 hour later (taskDuration is PT3600S) :
```2023-04-25T00:31:37,077 INFO [[index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_hckkdgha]-threading-task-runner-executor-7] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Finished reading partition[0], up to[0].```
and then it the task received status FAILED 20 seconds later:
```2023-04-25T00:31:57,273 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task [index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_hckkdgha] completed with status [FAILED].```
most likely because indexer was unable to upload task report to GCP bucket:
```2023-04-25T00:31:52,554 WARN [threading-task-runner-executor-7] org.apache.druid.java.util.common.RetryUtils - Retrying (4 of 4) in 4,605ms.
java.io.IOException: insufficient data written
[... lots of traceback]
2023-04-25T00:31:57,216 ERROR [threading-task-runner-executor-7] org.apache.druid.indexing.overlord.ThreadingTaskRunner - Exception caught during execution```
This also seemingly failed the next indexing task that just started:
```2023-04-25T00:31:57,330 INFO [KafkaSupervisor-telemetry-985093ac-671b-4ab5-ba22-27b5958a645f] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_telemetry-985093ac-671b-4ab5-ba22-27b5958a645f_36e21c9330640a2_cccocake] because: [No task in the corresponding pending completion taskGroup[0] succeeded before completion timeout elapsed]```


Do you have some steps to reproduce this? I'd like to try it.

It's a very simple process.
• We created a static lookup using the UI.
• When queriying the data, we sometimes get lookup not found.
I think this happens when data (recent) is served from middle manager, as this is data ingested with Kafka.

I believe ingestion tasks load the lookups at startup, were there any tasks already running when you defined the lookup? Does the problem go away when task duration expires and new tasks are created?

I may need to enable load at startup! Inthink that’s what’s missing 

You mean `druid.lookup.enableLookupSyncOnStartup` ? Could be, but it is `true` by default.

We had set to false... Then again, that wouldn't explain why we get that error

Actually, it would :slightly_smiling_face:

Good to hear :slightly_smiling_face: :+1:

Screenshot 2023-04-26 at 11.58.31 AM.png

what is your balancer strategy?  You could increase the max segments to move and turn on the batch sampler in the coordinator dynamic configs

druid.coordinator.balancer.strategy should be cost.  You can go into the coordinator dynamic configs for the other changes

Any reference in the doc? Let me have a look.

<@U03TAQRMJPQ> druid.coordinator.balancer.strategy is not set in my case.

And here is the json format dynamic config

```{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 100,
  "maxSegmentsToMove": 5,
  "percentOfSegmentsToConsiderPerMove": 100,
  "useBatchedSegmentSampler": false,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 10,
  "balancerComputeThreads": 1,
  "emitBalancingStats": false,
  "killDataSourceWhitelist": [],
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 100,
  "decommissioningNodes": [],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,
  "pauseCoordination": false,
  "replicateAfterLoadTimeout": false,
  "maxNonPrimaryReplicantsToLoad": 2147483647
}```

Balancing isn’t always done based on distribution of data. It is sometimes set based on what maximizes query performance.

Definitely increase the "maxSegmentsToMove" to at least 100. That's how many segments it will rebalance per coordinator cycle, usually every minute.

right now that means 5 segments moves per minute which could be really slow in a large cluster.

<@U0304FW2HPH>, thx for the info. Is there some reference as how the balancing algorithm works here? Or some tuning guide here?

A few things control the overall balancing process, as <@U03TAQRMJPQ> mentioned, the recommended strategy is `cost` . Other parameters that affect it are:
• `druid.segmentCache.numLoadingThreads` determines how many threads Historicals will use to process segment retrievals from Deep Storage. The default is probably fine for most cases, but having more will allow the historicals to process load segment requests from the coordinator faster. But this is not likely your issue since you have empty load/drop queues in your historicals.
• `maxSegmentsInNodeLoadingQueue` (Dynamic Coordinator Config) determines the max load/drop segment operations assigned to each Historical per cycle.
• `balancerComputeThreads` (Dynamic Coordinator Config) higher makes the balancing decision calculation faster.  


The most important to increase the speed of balancing is `maxSegmentsToMove`

<@U030K4UM3H7>, I will tune the maxSegmentsToMove to be higher to speed up this rebalance. One following question, how do I know if the rebalance after adding new server is finished or not?

rebalance is actually never finished, it is always making decisions about which segments to move. But you can leave that setting at 100 and it will keep up with balancing as you load more data. I think that given the low values it hasn't been able to catch up.

This is an important one too, useBatchedSegmentSampler should be true and will help speed up balancing

&gt;  rebalance is actually never finished, it is always making decisions about which segments to move. But you can leave that setting at 100 and it will keep up with balancing as you load more data. I think that given the low values it hasn't been able to catch up.
Let me put it this way, what would be the targeted optimal segment layout if druid.coordinator.balancer.strategy is cost (which is default)? For example, if after sometime, say one day, we still see some server 90% full, some server 10% full. Is this normal or not. If the "cost" criteria took query serving speed etc into consideration, different segment size into consideration,  then it may deem this allocation as optimal. Can we shed some light in this respect as how the cost is calculated?

It is not normal.  If it is balancing it should be more evenly distributing segments.  If you adjust maxsegmentstomove to say 200 or 300, balancer threads to 2 or 3, and turn on the batchsampler, it should optimize the balancing better.

catchingCost.png

&gt; druid.coordinator.balancer.strategy should be cost.
<@U03TAQRMJPQ> how about `cachingCost` strategy? Would that be more suitable if we want to expedite segments rebalancing? Pasting a snapshot of the documentation page that seems to suggest so.

It can be but there are other factors involved such as the size of the cluster, number of nodes, etc....  I believe cost is the standard now and adjusting the other settings will have a greater impact.

We are experimenting with this config change in one of the clusters and see that moving the `maxSegmentsToMove`  from `5` to `30` didn't help. Basically the strategy being used doesn't seem to be finding any candidate segments for moving. Following is a log snippet that we see repeating with each coordinator run.

```2023-04-27T19:32:22,841 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Processing 30 segments for balancing between active servers
2023-04-27T19:32:22,913 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - No good moves found in tier [_default_tier]
2023-04-27T19:32:22,913 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: Segments Moved: [0] Segments Let Alone: [30]
2023-04-27T19:32:22,913 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Assigned 200 segments among 5 servers
2023-04-27T19:32:22,913 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Dropped 0 segments among 5 servers
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Moved 0 segment(s)
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Let alone 30 segment(s)
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.64.198.217:8088, historical, _default_tier] has 100 left to load, 0 left to drop, 56,308,263,256 bytes queued, 1,081,955,67
9,079 bytes served.
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.64.210.149:8088, historical, _default_tier] has 100 left to load, 0 left to drop, 56,592,261,348 bytes queued, 1,081,956,56
0,890 bytes served.
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.64.197.22:8088, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 2,999,999,211,819 bytes serv
ed.
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.64.211.225:8088, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 2,999,999,619,582 bytes ser
ved.
2023-04-27T19:32:22,914 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.64.207.247:8088, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 2,999,999,685,143 bytes ser
ved.```
What might be the reasons behind this? Pasting the coordinator dynamic config from the experimental cluster here-

```{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 100,
  "maxSegmentsToMove": 30,
  "percentOfSegmentsToConsiderPerMove": 100,
  "useBatchedSegmentSampler": false,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 10,
  "balancerComputeThreads": 1,
  "emitBalancingStats": false,
  "killDataSourceWhitelist": [],
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 100,
  "decommissioningNodes": [],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 70,
  "pauseCoordination": false,
  "replicateAfterLoadTimeout": false,
  "maxNonPrimaryReplicantsToLoad": 2147483647
}```

With "maxSegmentsToMove"=30 it does seem to have had an effect
```Processing 30 segments for balancing between active servers```
I don't know why it is deciding to leave them alone:
```[_default_tier]: Segments Moved: [0] Segments Let Alone: [30]```
<@U03TAQRMJPQ> any idea?

You might want to try Ian's other recommendations:
```"maxSegmentsToMove": 300,
"useBatchedSegmentSampler": true,
"balancerComputeThreads": 3```

Thanks <@U030K4UM3H7> and <@U03TAQRMJPQ> This worked except for one cluster. Not sure what's different their. Tried with random and cost strategies. Will get to it later (it's a test cluster so not very critical at the. moment).

Good to hear! Please let us know if when you figure out the test cluster issue. I'm interested to see what the cause is.

May be historical servers are down/not able to load segments.  Check historical and coordinator logs for error details.

Hello There, Did anyone have Okta integration with Druid using pac4j ? Need some inputs

Right, PasswordProvider is not supported for this variable

This would be a really nice feature to contribute, if you're interested :slightly_smiling_face:

You could use `AWSCredentialsConfig` as a model; this is how it's done for S3

Looking at that example, I think all that's needed is replace `String` with `PasswordProvider` in `AzureAccountConfig`

:+1: for someone who might want to contribute this.

There is general support for environment variables in configs: <https://druid.apache.org/docs/latest/configuration/index.html#configuration-interpolation|https://druid.apache.org/docs/latest/configuration/index.html#configuration-interpolation>. Has a slightly different format from PasswordProvider but works for all runtime property configs.

They should be created on startup of the Coordinator

If that isn't working you could try running the `metadata-init` tool: <https://druid.apache.org/docs/latest/operations/metadata-migration.html#create-druid-tables>

Hello There, I am trying to integrate Okta with router services with `pac4j` extenrsion. When I am trying to use `JWT` type which is part of `pac4j` extension I am getting below error while starting services. In git documentation I can see `jwt` will be supported by `pac4j` extension but log shows its not loaded.
```Caused by: java.lang.IllegalArgumentException: Please make sure to load all the necessary extensions and jars with type 'jwt' on 'druid/router' service. Could not resolve type id 'jwt' as a subtype of `org.apache.druid.server.security.Authenticator` known type ids = [allowAll, anonymous, basic, pac4j, trustedDomain]```


It's something that can happen when merging partial segments during ingestion if their columns aren't in the same order

In this case it's going to use your dimensions in lexicographic order to sort segments, rather than the order specified by `dimensionsSpec` (as normal)

Are you using dimension discovery / schemaless dimensions? Perhaps that's related?

hmm interesting, we do have schema defined for the ingestion spec, not sure how this could happen.

worker task failed to start, I am using default 2 tasks  (1 controller + 1 worker).
I have druid.worker.capacity set to 4.

```2023-04-28T01:17:13,065 WARN [task-runner-0-priority-0] org.apache.druid.msq.exec.ControllerImpl - Controller: Work failed; task query-2ed4dc3b-b613-45a4-b627-89825cae5106; host 10.40.90.150:8100: TaskStartTimeout: Unable to launch all the worker tasks in time. There might be insufficient available slots to start all the worker tasks simultaneously. Try lowering 'maxNumTasks' in your query context to lower than [2] tasks, or increasing capacity. (java.lang.RuntimeException: org.apache.druid.msq.indexing.error.MSQException: TaskStartTimeout: Unable to launch all the worker tasks in time. There might be insufficient available slots to start all the worker tasks simultaneously. Try lowering 'maxNumTasks' in your query context to lower than [2] tasks, or increasing capacity.)```

Was something else still running?

If not, then can you share your SQL query?

the pending task moved to failed with above error. I have 6 replicas of MM’s in my cluster, but seems like can spin up only one task at a time

It does seem like a scheduling issue

I wonder if something about your MM configuration prevents it from working properly? Do you see them all show up separately with the expected 4 slots in the "Servers" tab of the web console?

I had to nuke ZK and redeploy, it wasn’t able to detect newly added MMs

Hi <@U0470TPTCUR>, in a native timestamp spec you can define the format as described here: <https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#timestampspec>.

For SQL based ingestion, you can use the `TIME_PARSE` function, which also allows you to define the format.

My bad actually, I was checked in wrong partition

This is usually due to a JSON syntax error

i validate the json, its fine. i checked this with another druid instance with the same data and its working fine.

after click the submit button nothing happens

Can you check if overlord has any issues by seeing the logs?

If the button is grayed out, it means for some reason the web console is not wanting you to submit the JSON. Try asking about it in <#C030K0Z7S1Z|>?

If there is an error with the spec JSON then you will see it if you click outside the json input part of the window (like click anywhere outside):

If that does not reveal an issue (which should how up in the bottom left corner of the JSON input) try refreshing your browser window

you browser will save the spec you created, but just in case you can copy paste it into a text editor

Also, just FYI (for anyone reading!) whatever happens you can always submit a task by pasting the spec into the submit task dialog on the `Ingestion` tab:

<@U030W50CNAC> <@U0344FW86DD> i downgraded druid 25 to druid 22, now its working

i was trying to create user in druid, i followed below steps and didnt work

<https://druid.apache.org/docs/latest/development/extensions-core/druid-basic-security.html>

do you guys know anything about creating users in druid ??

The docs for it are here: <https://druid.apache.org/docs/latest/operations/security-overview.html#authentication-and-authorization>