Apache Druid

Hi Vishal, you may want to consider storing them as <https://imply.io/videos/apache-druid-24-0-new-feature-nested-json/|nested columns>

&gt; I have one more question, is there a way to ingest the ingestion timestamp in druid if there is no timestamp column in the original source event?
You can use the `missingValue` field in the `timeSpec` section

Thanks <@U04CGQ1P68Z>, but missing value is not dynamic, my use case is to have dynamic timestamp.

Hi Team, how to get total size of all loaded lookups in druid?

Hi <@U04FKDX12ET>, That is a normal ingestion log message, what is your specific concern? The kafka ingestion process will use local disk to store data temporarily, and then merge all of that data when it comes time to build and publish segments.

I am running 0.22.1 version of druid on AWS ECS and the storage runs on EFS

in your update did you take the (default) configs from 24.0.1 without copying over your configs? Have a look at the differences in `conf/druid/{config}/_common/log4j2.xml` between your old configs and your new configs

more about that here: <https://druid.apache.org/docs/latest/configuration/logging.html>

Strongly recommend this superb (free) course on logging here: <https://learn.imply.io/apache-druid-logs>

thanks all, i hoped there might be a way to configure this via environment variable, but I guess not. Thanks anyway :slightly_smiling_face:

Hi team,
I am trying to setup a druid cluster in GKE. I am using default helm source from <https://github.com/apache/druid/tree/6b0b1d7af317d1a6b219cf1213139abf1607eaa0/helm/druid|this official repo>
After the helm chart deployment, my cluster is coming up fine but `coordinator` pod is constantly  crashing. Initially it was failing for liveness &amp; readiness probe. Just to start the pod fine, i disabled <https://github.com/apache/druid/blob/6b0b1d7af317d1a6b219cf1213139abf1607eaa0/helm/druid/templates/coordinator/deployment.yaml#L66-L71|the probes> but the container is still restarting constantly.
```druid-dev                druid-coordinator-649ff7fcb-5z2hw                         ●  2/2        164 Running```
There is no error trace or logs for why druid container is crashing constantly (the container comes to ready / running state and then restarts)

Would appreciate any pointers on this. Would be happy to provide any other details if needed. Thanks in advance!

coordinator can fail to start if the metastore connectivty is not established though it should have an error in the log

Hi All,
I'm applying APPROX_COUNT_DISTINCT_DS_THETA(total_unique_visitors) on druid query, It's working fine. When we are displaying data on superset, It's not loading.
Can you please help

these sound like issues with Superset and Metabase, you should reach out to the makers of those tools. Maybe file an issue in their GitHub

My understanding is that you can create metrics using druid sql expressions in superset. I will check this and get back

image.png

Hi Sachidananda, I am the author of that query so I can shed some light on it. The `sys` schema tables are virtual tables that provide a SQL queryable view into Druid's metadata (as cached on the broker). You can read more about them here: <https://druid.apache.org/docs/latest/querying/sql-metadata-tables.html> . The web console uses these tables extensively (incidentally if you have any more questions about the web console please post them in the <#C030K0Z7S1Z|> channel). You can build your own queries on top of the `sys` tables for whatever monitoring you might want to do. Every web console table that is powered by a sys table provides a way for you to "open the query" in the Query view.

Also as mentioned in this thread: <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1669706871042619> there have been massive improvements in the efficiency of the task table in the 24.0 version of Druid. Really worth upgrading for that one improvement alone!

Thanks a lot <@U030W50CNAC> for the insights

are these virtual tables that provide a SQL queryable view are present in memory? in overloard or something? or how does that work

Task info comes from the overload but the SQL processing itself is done on the broker

can this cause overlord to go down if there are too many tasks? And the ui refresh setting is set at 10 seconds?

As i observerd some overlord cpu utilization for overlord went high and leader shifted form one to other while loading tasks page but not sure

As I said above you should update to 24.0

ok sure <@U030W50CNAC> thanks for the inputs

Hi Ashok,

May be we should get the log from deep storage location (<https://s3.console.aws.amazon.com/s3/buckets/bibekdeepclusture?region=us-east-1&amp;prefix=deepstorage/indexing-logs/&amp;showversions=false|indexing-logs> folder) where the tasks logs will get stored and also overlord log should have more information.

Check if you have live connection to kafka and zookeeper.

Hi Sahoo...  we have 11 datasources (Tables) in druid and in middle manager runtime properties.  druid.worker.capacity = 12  .   How to calculate how worker.capacity we can set ?   I belive this leads to 404 error &amp; goes to PENDING / FAILED Task status !?

is there any calculation logic there to arrive worker.capacity to set ?

As per guide, It should be `druid.worker.capacity` “Number of CPUs on the machine - 1”

ok.  so it's not based on the number of data sources also ?

While checking the data-source status log &amp; report  " request failed with status code 404  "

[0.23.0 -&gt; 24.0] Post upgrade none of the peons tasks come up.

```2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[pool-6-thread-1] org.apache.druid.emitter.kafka.KafkaEmitter - Exception while getting record from queue or producer send, Events would not be emitted anymore.: \nat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~ [?:1.8.0_275]\nat java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~ [?:1.8.0_275]\nat java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~ [?:1.8.0_275]\nat org.apache.druid.emitter.kafka.MemoryBoundLinkedBlockingQueue.take(MemoryBoundLinkedBlockingQueue.java:59) ~ [?:?]\nat org.apache.druid.emitter.kafka.KafkaEmitter.sendToKafka(KafkaEmitter.java:156) ~ [?:?]\nat org.apache.druid.emitter.kafka.KafkaEmitter.sendMetricToKafka(KafkaEmitter.java:138) ~ [?:?]\nat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~ [?:1.8.0_275]\nat java.util.concurrent.FutureTask.run(FutureTask.java:266) ~ [?:1.8.0_275]\nat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~ [?:1.8.0_275]\nat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~ [?:1.8.0_275]\nat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~ [?:1.8.0_275]\nat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~ [?:1.8.0_275]\nat java.lang.Thread.run(Thread.java:748) ~ [?:1.8.0_275]\n"}
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting: "}
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[Thread-31] org.apache.zookeeper.ZooKeeper - Session: 0x100103961870005 closed: "}
2022-12-23T00:23:17+00:00	middleManager	{"Time":"1970-01-01T00:00:00Z","Severity":"INFO","Component":"middleManager-ip-172-31-4-225","Message":"[Thread-31] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]: "}```


not sure, but the docs say minimum version of zk is 3.5: <https://druid.apache.org/docs/latest/dependencies/zookeeper.html#minimum-zookeeper-versions>

Although the release notes don't seem to mention a zk change from 0.23 -&gt; 24.0.

I see kafka errors in the MM log, have to run an errand, bbiab

Since 0.23.0 is working fine, it is puzzling what is breaking as the backwards incompatible changes in the release notes have nothing relevant to the issue above.

true...  Did you create new config files, or use the same?

I will try with zookeeper version &gt; 3.5 and update

Are you using 24.0.2, just in case there's a new bug fix that matters?

I have tried 24.0.0, .1 and .2 and same behavior

If it doesn't work w/3.5, a bug report would probably be good, even if we figure it out.

in Druid 0.22 support for ZK 3.4 was dropped &lt; <https://github.com/apache/druid/issues/11657#22-up-dropped-zk-3.4> I guess it would be nice if there was like a console warning message that told you about the bad ZK version. It is curious that stuff just happened to work.

Also, while you should definitely update you ZK to &gt; 3.4 it should also be noted that in Druid 25 (which is currently in release candidate stage! <https://lists.apache.org/thread/ry3w3778xx5lv6tdxtz2gh2fks2f6bn7>) the defaults are changing to http based task announcement (see my response in this thread <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1671473660817489>). I bet that switching away form using ZK for task announcement would help in this case also.

Confirming that zookeeper version is 3.5.9.

coordinator.logs

is this the full log? I don’t see the message which states starting lifecycle

I can see below in your middle manager log
```Node[<http://192.168.5.232:8081>] of role[overlord] went offline.: "}```

<@U03BJAMQ4H5> Yes, above are complete logs.   What else I can do to troubleshooting why overload is shutting down?

what is the Xmx in the jvm config for the overlord. One possibility is that the overlord is running out of resources. You are running coordinator as overlord?

While adding Druid distinct count extension getting error on cluster start extension not found

Checked extension folder can’t see any folder for distinct count 

Hi, Please share:
1. Druid version
2. Any errors in the log files for master nodes ? 

Druid version 24.02
Error 
Extension not found in
Druid/extensions

What’s the name of the extension used/defined in the `common.runtime.properties`?

This is not a core extension. Did you follow the  steps to load it <https://druid.apache.org/docs/latest/development/extensions.html#loading-community-extensions>

This is a WARN and not ERROR, but I understand that can be irritating.  I've seen people improve this by using http-based segment management, easing up the pressure on zookeeper.  To do this, you'd change overlord properties to set druid.indexer.runner.type to httpRemote, and coordinator settings to set druid.coordinator.loadqueuepeon.type to http.  You could try that (you'd have to restart overlord and coordinator, of course).

It is important to note that the concept of aggregation (or aggregates) is not unique to Druid and is a general SQL concept (or a general data concept really - it goes beyond SQL). You can learn more about aggregation in SQL in general by searching for what is sql aggregation (here is one result from such a search <https://www.geeksforgeeks.org/aggregate-functions-in-sql/>).

Once you have a good grasp on that concept there are three things you should know when applying that knowledge to Druid specifically:

(1) As well as the common aggregates like `COUNT`, `SUM`, `MAX` etc, Druid supports really fancy sketch based aggregations that let you do things like compute (approximate) uniques and quantiles and much more very quickly (as Sergio noted).

(2) In Druid it is also common (but not required!) to aggregate or "rollup" your data at ingest time so that the data in Druid is partially aggregated already. (see <https://druid.apache.org/docs/latest/tutorials/tutorial-rollup.html>).

(3) When reading about aggregation in Druid keep in mind that as well as SQL Druid also has a JSON based language for expressing queries and ingestion jobs (you will see it in the docs). In the JSON based language "aggregates" are called "metrics" and a metric spec like `{ "type": "count", "name": "cnt" }` means the same `COUNT(*) AS "cnt"` in SQL. The good news is that as of Druid 24.0 it is possible to do both ingestion and querying via SQL so you could avoid needing to interact with the JSON based form.

Ho Balaji. Druid can aggregate data both at time of ingestion and at query time. It has a many supported aggregate functions including very fast approximations for distinct counts and quantiles. You can find more details at <https://druid.apache.org/docs/latest/querying/sql-aggregations.html|https://druid.apache.org/docs/latest/querying/sql-aggregations.html>

It’s not something you should change arbitrarily

I just tried it with 1024 and then with 2048 and it passed ok!  Thanks!


so maybe my issue is that I can only use specific numbers and not any arbitrary number.
PS I wanted to check differnet k and see the different error rates.

<https://datasketches.apache.org/docs/Theta/ThetaSize.html>

Trying some direction:

```sk1_bytes = sketch.compact().serialize()
print(f'base64 encoding of sketh {base64.b64encode(sk1_bytes)}')```
Then will use the resulting string from above and see if works.

I'm not aware of any documentation that druid has about creating data sketches outside of druid.  I hope you find some, or have luck with it!

<@U04DRDGD8UF>   One of Imply’s Sokution Consultants <@U032CA4JJV6> recently posted a blog which may be helpful in answering your question. Here is the link:  <https://blog.hellmar-becker.de/2022/12/26/ingesting-data-sketches-into-apache-druid/|https://blog.hellmar-becker.de/2022/12/26/ingesting-data-sketches-into-apache-druid/>. I would love to hear if this helps you out!

I actually managed to ingest this offline sketch, but having a look at this post and this blog in general - Wow! Exactly the kind of info I needed at this starting point! A set of patterns, experiences, best practices, I'll go over the other posts as well, thanks!

I actually managed to ingest this offline sketch(forgot to update..), but having a look at this post and this blog in general - Wow! Exactly the kind of info I needed at this starting point! A set of patterns, experiences, best practices, I'll go over the other posts as well, thanks!

I actually managed to ingest this offline sketch(forgot to update..), but having a look at this post and this blog in general - Wow! Exactly the kind of info I was looking for at this starting point! A set of patterns, experiences, best practices, I'll go over the other posts as well, thanks!

Hi Druid experts

I am having one doubt in *metrics emitted from historical and coordinator related to segments.*
As per the doc all metrics are getting emitted. However `/segment/count`
is not emitted from

`query/segments/count`-
```This metric is not enabled by default. See the QueryMetrics Interface for reference regarding enabling this metric.```

`query/segments/count` is different

i am talking about `segment/count`

Have you tried these 2 monitors:
```org.apache.druid.server.metrics.HistoricalMetricsMonitor

org.apache.druid.server.metrics.SegmentStatsMonitor```

```org.apache.druid.server.metrics.HistoricalMetricsMonitor```
this i have added in historical nodes and in druid code i can see
HistoricalMetricsMonitor.java:104 - its emitting `segment/count`  but it is not shown in grafana and others like `segment/used, segment/usedPercent`  are in same file, these are getting emitted and i can see in grafana

no no i mean all others are emitted except `segment/count`

Can you try to add this :
```org.apache.druid.server.metrics.SegmentStatsMonitor```

it seems we are on 0.22.1 and this monitor SegmentStatsMonitor is not present in this version

however i can see as per code this file

HistoricalMetricsMonitor.java:104 - its emitting `segment/count`  but not sure why its not showing up in grafana

Which emitter are you using? Some emitters include filter lists that limit which metrics and dimensions are emitted by default. This is the list for the prometheus emitter for example <https://github.com/apache/druid/blob/master/extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json>

Hi <@U02VCGJFER5>

Thanks a lot for your response
I am using prometheus emitter.

I can see

`"coordinator_segment/count"` 
  `"historical_segment/count"`
in default list

but i don't see these appearing in grafana
Can you tell me what to search for finding these 2 metrics?

Sorry for the delayed response. I'm not very familiar with the prometheus extension, but reading the code it looks like maybe it's not correctly filtering this metric on the emitter side

I'd recommend just changing the list to include `segment/count`  instead of trying to specify different dimensions based on the service that is reporting the `segment/count`

ie change the last 2 lines to
```"segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge", "help": "Number of served segments."},```
and see if that works

this was a contrib extension, so it's possible there may be a bug with how it;s trying to detect service specific metrics. I noticed the documentation for the prometheus emitter mentions that the service and the metric should be separated by a `-` but instead the defaultDimensions use `_`

See the last 2 lines in <https://github.com/apache/druid/blob/master/docs/development/extensions-contrib/prometheus.md#metric-mapping>

Thanks <@U02VCGJFER5>  for the detailed response
will do some trail and error

There are couple of ways to do that  the simple one could be -
(1) 1st start druid quick-start along with shipped Zk
(2) Once druid is running,  just start Kafka server  ( dont start the Zk) - It should use the same ZK as the druid one.
```nohup bin/kafka-server-start.sh config/server.properties &amp;```

Hi <@U030KB0NXM0>, I hope you are doing well .

Could you check and confirm if your druid cluster is stable and services are running fine (Zookeeper/Middlemanagers /Overlord etc ) ?

For the log analysis, you can follow below sequence -
1. Note the `task_id` &gt; search for it `overlord` log &gt; look for sequence of events [ Form here you should be able to see if the task was assigned to `MiddleManager`  or else if something went wrong before ]
2. If the tasks are assigned to a MM then note the Middlemanager host and search for the task_id in the middle manager log [ See what middleManager log says]
&gt;  view full logs’ shows 404.
This seems to be a configuration issue or there are no logs generated ( may be task did not run at all ).
<https://druid.apache.org/docs/latest/configuration/index.html#task-logging>

1. Try to reduce number of `index_parallel` jobs for now for sake of troubleshooting ; Aim for 1 for now
2. If it fails, check the task log

3. If possible, upgrade to 24.0.2 to leverage MSQE for batch ingestion

Thanks <@U04CGQ1P68Z> <@U031DSDAQDN> . i will check and update

Oh I found solution. Thanks!
(click “Mark as used all segments”)
<https://druid.apache.org/docs/latest/operations/rule-configuration.html#reloading-dropped-data>

nice, looks like you solved it. Yeah retention rules evaluate against the currently active segments so if you increase the load window you need to explicitly mark segments as used