Apache Druid

This is a virtual column whose  output type is STRING_ARRAY.

And we have filter on this saying vc != null

if i remove the filter this works, but with filter it is failing with this exception.

I saw the code and saw that there is no handling for ARRAY in *makeProcessorInternal ?*  we do not expect Vcs to be Array ?

i guess you are using `useSchemaDiscovery` or nested/json columns? afaik that's the only way you can get `ARRAY&lt;STRING&gt;` columns

either way, if you had some multi-valued string columns before, there would be some query rewriting to do, described in these doc sections:

• <https://druid.apache.org/docs/latest/ingestion/schema-design.html#type-aware-schema-discovery>
• <https://druid.apache.org/docs/latest/ingestion/schema-design.html#migrating-to-type-aware-schema-discovery>

the biggest implication is this first one (the change from multi-value dimensions to arrays):

&gt; Update any queries that use multi-value dimensions (MVDs) to use UNNEST in conjunction with other functions so that no MVD behavior is being relied upon. Type-aware schema discovery generates ARRAY typed columns instead of MVDs, so queries that use any MVD features will fail.

if you aren't prepared to embark on that then please avoid `useSchemaDiscovery` for now. we're building a smoother migration path for an upcoming release, so you can wait for that

thanks. Will try to go through this and see what we can do about it.

my feeling is that multi-value strings -&gt; arrays is a pretty major migration, so if you didn't mean to do it on purpose, it would be best to switch back

Trying to understand :  how is  multi value strings is  different than the Arrays ?

I think problem is we have a custom function which was returning a  MVD.

we are on 0.21.1 [Trying to move to 26], which recent version we can fallback without breaking mvd behaviour ?

you should be OK on 26 if you aren't using the new array columns or type-aware schema discovery features; they are new &amp; optional

if you aren't using those, &amp; are still seeing this error, please share some additional info about your custom function and we'll try to help

We do define the schema for each column during ingestion, i am hoping which means no schema discovery taking place.

I will check the details of  custom function and will update. But as far as i know it take a single value and generate List&lt;String&gt; from it.

so the  return type we changed from ``ExprType.STRING_ARRAY`   to `ExpressionType._STRING_ARRAY_`  assuming both are same

Hi Tom,

Without Window Functions (which are coming out soon, might be in Druid 25/26 already in experimental form), the only way I know how to do this is a self-join ... but it requires that you have unique key values to properly generate a clean contiguous list of consecutive numbers.

Here's an example using the wikipedia demo data set:

```select a.channel, 
       a.__time,
       count(b.__time) as ranknum
  from wikipedia a 
  join wikipedia b on a.channel = b.channel
 where b.__time &lt;= a.__time
 group by 1, 2
 order by 1, 2```
-- John

count(b.__time), a good one. Thanks a lot <@U04C4593VDY>

image.png

are there any additional errors? wild guess: you might need a different version of `node` or `npm`. for reference i am using node 16.20.0 and npm 8.19.4

Thanks for answering. I did not see any other errors. I updated the version, but it still has this error

you can reverse engineer the travis.yml and find the correct version number.

if you install <https://volta.sh/> then it should pick up the exact versions that the console asks for: <https://github.com/apache/druid/blob/master/web-console/package.json#L54>

what command are you using to build druid?

Do you see any errors in coordinator logs, coordinator will instruct historical to drop/load segments.

I see the following log from coordinator
```2023-06-08T06:17:33,088 WARN org.apache.druid.server.coordinator.rules.LoadRule - No available [_default_tier] servers or node capacity to assign segment [0_base_2023-06-07T20:00:00.000Z_2023-06-07T21:00:00.000Z_2023-06-07T20:00:01.729Z]! Current replication: [[_default_tier:1/2]]
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.MarkAsUnusedOvershadowedSegments - Skipping MarkAsUnused as there are no overshadowed segments.
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Assigned 0 segments among 1 servers
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Dropped 0 segments among 1 servers
2023-06-08T06:17:33,089 WARN org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
2023-06-08T06:17:33,089 INFO org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[10.233.92.61:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 343 served, 0 bytes queued, 10,442,387,194 bytes served.```

and from historical , logs the below
```2023-06-08T06:20:17,803 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1462/23769][0_base_2023-06-07T22:00:00.000Z_2023-06-07T23:00:00.000Z_2023-06-07T22:00:01.153Z_11]
2023-06-08T06:20:18,280 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1463/23769][0_base_2023-06-02T19:00:00.000Z_2023-06-02T20:00:00.000Z_2023-06-02T19:00:00.714Z_144]
2023-06-08T06:20:18,364 INFO org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment[1464/23769][0_base_2023-06-01T09:00:00.000Z_2023-06-01T10:00:00.000Z_2023-06-01T09:00:00.504Z]```

Did your retention rule changes saved in druid console , try bouncing leader coordinator if chnages didnt saved

Yes I checked retention rule changed and segments reduced. What do you mean "try bounding leader coordinator"?

i was saying if changes are not taking effect then we can restart coordinator

okay, I see. I've already tried restarting coordinator but still same.

I've only set dropBeforeByPeriod, but if no load retention are set will affect this?

I tried below but still same
1. set load retention to 1D and save
2. restart coordinator
3. restart historical
4. still trying to load 20k

You might want to try cleaning the segmentCache volume. This will be re-populated by the coordinator. At the moment it might be loading the segments it has in the volume already

<@U030MBK46BD> I cleaned the segmentcache volume as you said. It's now working properly. Thank you

<@U030MBK46BD> I thought when historical starts, it would talk to coordinator or zookeeper to get metadata for current cluster state. But seems like druid uses local cache, or local metadata first. Is this right?

Yes, the historical first looks at it’s local segment cache and attempts to make this available. When this is complete, it communicates with the coordinator to add/remove segments served

hmm, I haven’t seen those `time_*` metrics before. Can’t find references to it in the codebase as well.  Do you’ve any custom extension or metric module enabled by any chance? If not, can you please share a few sample `query/segment/time_*` metrics so we can see what’s going on?

If I were to take a guess, some internal system is rolling up `query/segment/time` into these metrics. so
`query/segment/time_count`  = count(query/segment/time)
query/segment/time_sum = sum(query/segment/time)

and so on

I haven’t had time to trace this issue. I guess you are probably right. Just to confirm that this is not some hidden feature from the Druid side. Thx

Hi guys, seems like `druid.segmentCache.numLoadingThreads` on historical node is not making any difference on how many threads will be used to load segments from S3 in parallel. I am using latest Druid version, by changing this property to 16, it still uses only 1 thread to download the segment from s3, any pointers ?

Screenshot 2023-06-09 at 12.14.00 PM.png

These are the things I have added in the common.runtime.properties file in the druid/auto dir
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.QueryCountStatsMonitor","org.apache.druid.segment.realtime.RealtimeMetricsMonitor"]
druid.emitter=http
druid.module.enable="druid-metrics"
druid.emitter.logging.logLevel=info
druid.monitoring.emissionPeriod = PT1M
druid.emitter.http.minHttpTimeoutMillis = 1000
druid.emitter.http.recipientBaseUrl = <http://localhost:8090/druid/metrics>

hi there, the stack trace tells me that Druid isn’t able to find the extension `/Users/ananya/Downloads/apache-druid-26.0.0/extensions/druid-extensions-contrib-firehose-segment-push` that you’ve specified in your `loadList`. Can you confirm that the extension exists in that path?

Also, fwiw, I can’t find the specific extension in this list <https://druid.apache.org/docs/latest/development/extensions.html> , of course unless you’re using a proprietary one

Yes, my bad. I have removed that extension. Now, I am getting this error
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No implementation for java.util.List&lt;org.apache.druid.segment.realtime.FireDepartment&gt; was bound.
 while locating java.util.List&lt;org.apache.druid.segment.realtime.FireDepartment&gt;
  for the 1st parameter of org.apache.druid.segment.realtime.RealtimeMetricsMonitor.&lt;init&gt;(RealtimeMetricsMonitor.java:52)
 while locating org.apache.druid.segment.realtime.RealtimeMetricsMonitor
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler
 at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
    at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)
    at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No implementation for java.util.List&lt;org.apache.druid.segment.realtime.FireDepartment&gt; was bound.
 while locating java.util.List&lt;org.apache.druid.segment.realtime.FireDepartment&gt;
  for the 1st parameter of org.apache.druid.segment.realtime.RealtimeMetricsMonitor.&lt;init&gt;(RealtimeMetricsMonitor.java:52)
 while locating org.apache.druid.segment.realtime.RealtimeMetricsMonitor
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler
 at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:99)
    at com.google.inject.Guice.createInjector(Guice.java:73)
    at com.google.inject.Guice.createInjector(Guice.java:62)
    at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
    at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
    at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
    at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
    ... 2 more

The `RealTimeMetricsMonitor` is obsolete - please remove it from your `monitors` list and give it a try. Also, there’s a related thread <https://www.druidforum.org/t/unable-to-configure-specific-emiiter/6454/2>

When I removed `RealTimeMetricsMonitor` , many of the errors are gone except I am getting the following error in middleManager.stdout.log
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No implementation for org.apache.druid.server.metrics.QueryCountStatsProvider was bound.
 while locating org.apache.druid.server.metrics.QueryCountStatsProvider
  for the 1st parameter of org.apache.druid.server.metrics.QueryCountStatsMonitor.&lt;init&gt;(QueryCountStatsMonitor.java:40)
 while locating org.apache.druid.server.metrics.QueryCountStatsMonitor
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler
 at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
    at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)
    at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)
    at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) No implementation for org.apache.druid.server.metrics.QueryCountStatsProvider was bound.
 while locating org.apache.druid.server.metrics.QueryCountStatsProvider
  for the 1st parameter of org.apache.druid.server.metrics.QueryCountStatsMonitor.&lt;init&gt;(QueryCountStatsMonitor.java:40)
 while locating org.apache.druid.server.metrics.QueryCountStatsMonitor
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler
 at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

1 error
    at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:99)
    at com.google.inject.Guice.createInjector(Guice.java:73)
    at com.google.inject.Guice.createInjector(Guice.java:62)
    at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
    at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
    at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
    at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
    ... 2 more

It appears to be the same issue as someone that ran into this recently - <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1684492769746299>. Please try the suggestion in the thread - add the `QueryCountStatsProvider` monitor to `druid.indexer.runner.javaOpts`


If that doesn’t work, alternatively, you can try adding `QueryCountStatsMonitor`  selectively to broker, router and the historical `runtime.properties` and remove it from `common.runtime.properties` so the middle manager doesn’t pick it up.

Yes, this thread worked well for me and now I am able to see the console. But the problem still remains that I want druid to emit query/time metrics on the http endpoint but it is not. What I am getting in the error message in my spring boot application is this :

[2m2023-06-11T23:35:57.928+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-8][0;39m [36mo.s.w.s.m.m.a.HttpEntityMethodProcessor [0;39m [2m:[0;39m Using 'application/json', given [*/*] and supported [application/json, application/*+json]
[2m2023-06-11T23:35:57.928+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-8][0;39m [36mo.s.w.s.m.m.a.HttpEntityMethodProcessor [0;39m [2m:[0;39m Writing [{timestamp=Sun Jun 11 23:35:57 IST 2023, status=404, error=Not Found, path=/druid/metrics}]
[2m2023-06-11T23:35:57.929+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-8][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m Exiting from "ERROR" dispatch, status 404
[2m2023-06-11T23:35:58.125+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m POST "/druid/metrics", parameters={}
[2m2023-06-11T23:35:58.126+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.w.s.handler.SimpleUrlHandlerMapping [0;39m [2m:[0;39m Mapped to ResourceHttpRequestHandler [classpath [META-INF/resources/], classpath [resources/], classpath [static/], classpath [public/], ServletContext [/]]
[2m2023-06-11T23:35:58.127+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.w.s.r.ResourceHttpRequestHandler [0;39m [2m:[0;39m Resource not found
[2m2023-06-11T23:35:58.127+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m Completed 404 NOT_FOUND
[2m2023-06-11T23:35:58.127+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m "ERROR" dispatch for POST "/error", parameters={}

So, can you please guide why is druid not emitting metrics on that http endpoint that I have configured?

Good to hear about the progress. Without knowing details of the actual setup, I’ll leave a few pointers to help troubleshoot:
1. Are there any errors and warnings in the Druid logs related to metrics not being emitted or so?
2. Is your configured http endpoint reachable from Druid? Can you successfully try a curl request with a `GET` or`POST` request with a sample payload to your http endpoint?
3. Do you’ve any query workload in Druid? If not, try running a few `SELECT` queries. I doubt the `QueryCountStatsMonitor` will emit any query metrics if there are no queries.
4. The `druid.emitter.http.flush*` properties determine the different thresholds when metrics are flushed, so I’d suggest looking into it if you’ve misconfigured something there or the defaults are high.

Thanks for your help Abhishek. As you said, I have checked the above points. I could not find any warnings or errors in druid's logs. And sending post requests through Postman is working well too. I have run a few queries in the console but druid still is not generating metrics for those queries. I have done the following configuration changes in the common.runtime.properties file of ../auto/_common. Am I doing it right?

druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=http
druid.module.enable="druid-metrics"
druid.emitter.logging.logLevel=info
druid.monitoring.emissionPeriod = PT1M
druid.emitter.http.minHttpTimeoutMillis = 1000
druid.emitter.http.recipientBaseUrl = <http://localhost:8090/druid/metrics>

ok. I see your common runtime.properties. A few follow up questions:
1. What’s `druid.module.enable="druid-metrics"`? I don’t see that config in the Druid docs
2. I see `JvmMonitor` in set up in common runtime properties, do you see <https://druid.apache.org/docs/latest/operations/metrics.html#jvm|any jvm> metrics in your http endpoint?
3. Specifically for `QueryCountStatsMonitor` , do you’ve them added in `druid.monitoring.monitors` in `auto/broker/runtime.properties` and `auto/historical/runtime.properties` ?

Yes, I think druid.module.enable="druid-metrics is wrong. No, I do not see any jvm metrics at the endpoint. When I added, I got the following error:

Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Problem parsing object at prefix[druid.monitoring]: Cannot construct instance of `org.apache.druid.server.metrics.MonitorsConfig`, problem: java.lang.ClassNotFoundException: QueryCountStatsMonitor
 at [Source: UNKNOWN; line: -1, column: -1].
 at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating com.google.common.base.Supplier&lt;org.apache.druid.server.metrics.MonitorsConfig&gt;
 at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.server.metrics.MonitorsConfig
  for the 2nd parameter of org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110)
 at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler
 at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
 while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

<@U053HH3C3U3>, ok just getting back. Looking at the stack trace, it seems like the fully qualified name for the monitor isn’t specified - it should be something like `"org.apache.druid.server.metrics.QueryCountStatsMonitor"`

Specifically, try adding this in your broker and historical  runtime properties: `druid.monitoring.monitors=["org.apache.druid.server.metrics.QueryCountStatsMonitor"]`

The console is working now. But the problem of druid not generating metrics still persists!

I’d suggest isolating the problem to a single process, say historical. Do you see any errors or warnings in `historical.log`:
```ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[<http://localhost:8090/druid/metrics>]```
If you don’t see the request erroring out, at the very least I’d expect to see warnings if the http server is rejecting the request for whatever reason (I spun a python server locally `python3 -m http.server 8090` and can see warnings and errors if misconfigured):
``` WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 2) in 1,976ms.
org.apache.druid.java.util.common.ISE: Emissions of events not successful[501: Unsupported method ('POST')], with message[<!DOCTYPE HTML>```

I do not see any error or warning in any of the log files. At this
```<http://localhost:8090/druid/metrics>```
all I am seeing is [ ]. And in the spring boot application's log, I am seeing these messages:
[2m2023-06-14T10:28:30.689+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-2][0;39m [36mo.s.w.s.m.m.a.HttpEntityMethodProcessor [0;39m [2m:[0;39m Using 'application/json', given [*/*] and supported [application/json, application/*+json]
[2m2023-06-14T10:28:30.689+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-2][0;39m [36mo.s.w.s.m.m.a.HttpEntityMethodProcessor [0;39m [2m:[0;39m Writing [{timestamp=Wed Jun 14 10:28:30 IST 2023, status=404, error=Not Found, path=/druid/metrics}]
[2m2023-06-14T10:28:30.690+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-2][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m Exiting from "ERROR" dispatch, status 404
[2m2023-06-14T10:28:30.823+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-3][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m POST "/druid/metrics", parameters={}
[2m2023-06-14T10:28:30.823+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-3][0;39m [36mo.s.w.s.handler.SimpleUrlHandlerMapping [0;39m [2m:[0;39m Mapped to ResourceHttpRequestHandler [classpath [META-INF/resources/], classpath [resources/], classpath [static/], classpath [public/], ServletContext [/]]
[2m2023-06-14T10:28:30.823+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-3][0;39m [36mo.s.w.s.r.ResourceHttpRequestHandler [0;39m [2m:[0;39m Resource not found
[2m2023-06-14T10:28:30.823+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-3][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m Completed 404 NOT_FOUND
[2m2023-06-14T10:28:30.824+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-3][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m "ERROR" dispatch for POST "/error", parameters={}
[2m2023-06-14T10:28:30.824+05:30[0;39m [32mDEBUG[0;39m [35m1920[0;39m [2m---[0;39m [2m[nio-8090-exec-3][0;39m [36ms.w.s.m.m.a.RequestMappingHandlerMapping[0;39m [2m:[0;39m Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)

Oh i was looking at some other log file. Yes, the historical.log file does have error messages like these:
WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 1,062ms.
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8090
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
    at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:783) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:667) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:663) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:661) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:569) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:545) ~[druid-processing-26.0.0.jar:26.0.0]
    at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) ~[druid-processing-26.0.0.jar:26.0.0]
Caused by: java.net.ConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8090
    at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) ~[async-http-client-2.5.3.jar:?]
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) ~[async-http-client-2.5.3.jar:?]
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) ~[async-http-client-2.5.3.jar:?]
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) ~[async-http-client-2.5.3.jar:?]
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
    at java.lang.Thread.run(Thread.java:1589) ~[?:?]

The warning is indicative that the http server is refusing the connection from the historical:
```WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 1,062ms.
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8090```
I’d suggest debugging your server code to see why that’s the case. Also, try setting the host explicitly to `127.0.0.1` or another ip instead of `localhost` . Because if your loopback is incorrectly setup, the address resolution from `localhost` to an ip address can go wrong. Hope that helps

Hi, I did the change as you suggested but nothing changed. I have changed my server's address to 127.0.0.1 and accordingly changed the endpoint's url as well but i am still getting the same error
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: /127.0.0.1:8090

sorry, I’m not sure how to debug the connection refused issue coming from the http server program. If you can set up a simple python web server (e.g., `python3 -m http.server 8090` ) and confirm that the server receives Druid metrics successfully, I’d look more closely into the http java program

Do you see anything in the historical logs ?
It could also be kubernetes is killing them. Can you post your historical config?

<@U030K4UM3H7> The pod is restarting with 137 error code and the logs has this
```1546.900: [GC (Allocation Failure) 1546.900: [ParNew: 791531K-&gt;46504K(843456K), 0.0636735 secs] 14092523K-&gt;13347496K(52335104K), 0.0639037 secs] [Times: user=0.70 sys=0.00, real=0.07 secs]```

I'm sorry, I should've translated lakhs before I answered. I think you have way too many segments for that volume of data. Meaning that you have segments that are like 300 kb. Which is tiny. Normal segment sizes are around 1000 times bigger than that. The <https://druid.apache.org/docs/latest/operations/segment-optimization.html|ideal segment size >is about 500MB which means that you should have about  200 segments for 100GB of data. The large number of segments is probably a part of the problem.
Also, the recommendation is that you use 24GB of Heap maximum per historical. If you share you historical config, we might be able to figure out why it is consuming so much memory.
You can use <https://druid.apache.org/docs/latest/data-management/compaction.html|compaction> to merge a lot of small segments into bigger more efficient segments.

Hello - there was an issue a while ago where MSQ queries were taking a long time to plan <https://github.com/apache/druid/pull/14123> - if this sounds like your issue it might have been fixed in v26

&gt; this impacts performance negatively, because it causes many additional outer-query scenarios to be considered, which is expensive.

I was testing the sample paylod given in the document. I don't think the above is my issue. This is a straight fotward ingestion request.

```payload = json.dumps({
  "query": "INSERT INTO wikipedia\nSELECT\n  TIME_PARSE(\"timestamp\") AS __time,\n  *\nFROM TABLE(\n  EXTERN(\n    '{\"type\": \"http\", \"uris\": [\"<https://druid.apache.org/data/wikipedia.json.gz>\"]}',\n    '{\"type\": \"json\"}',\n    '[{\"name\": \"added\", \"type\": \"long\"}, {\"name\": \"channel\", \"type\": \"string\"}, {\"name\": \"cityName\", \"type\": \"string\"}, {\"name\": \"comment\", \"type\": \"string\"}, {\"name\": \"commentLength\", \"type\": \"long\"}, {\"name\": \"countryIsoCode\", \"type\": \"string\"}, {\"name\": \"countryName\", \"type\": \"string\"}, {\"name\": \"deleted\", \"type\": \"long\"}, {\"name\": \"delta\", \"type\": \"long\"}, {\"name\": \"deltaBucket\", \"type\": \"string\"}, {\"name\": \"diffUrl\", \"type\": \"string\"}, {\"name\": \"flags\", \"type\": \"string\"}, {\"name\": \"isAnonymous\", \"type\": \"string\"}, {\"name\": \"isMinor\", \"type\": \"string\"}, {\"name\": \"isNew\", \"type\": \"string\"}, {\"name\": \"isRobot\", \"type\": \"string\"}, {\"name\": \"isUnpatrolled\", \"type\": \"string\"}, {\"name\": \"metroCode\", \"type\": \"string\"}, {\"name\": \"namespace\", \"type\": \"string\"}, {\"name\": \"page\", \"type\": \"string\"}, {\"name\": \"regionIsoCode\", \"type\": \"string\"}, {\"name\": \"regionName\", \"type\": \"string\"}, {\"name\": \"timestamp\", \"type\": \"string\"}, {\"name\": \"user\", \"type\": \"string\"}]'\n  )\n)\nPARTITIONED BY DAY",
  "context": {
    "maxNumTasks": 3
  }
})
headers = {
  'Content-Type': 'application/json'
}```

How many worker slots do you have available? With `"maxNumTasks":3` it will need 4 slots in order to start, one for the controller and 3 for the workers.

I'm having 16 , let me check by pausing some of the ingestions in Dev.

Thank you <@U030K4UM3H7>, that was the issue.

yes, for `INFORMATION_SCHEMA.TABLES` and `INFORMATION_SCHEMA.COLUMNS` tables, a user needs to be authorized to access resources (like user-created datasources).

For `INFORMATION_SCHEMA.SCHEMATA` table, there’s no auth as it only provides high-level information about the known schemas (and info about user-defined datasources aren’t leaked here)

Relevant auth code: <https://github.com/apache/druid/blob/master/sql/src/main/java/org/apache/druid/sql/calcite/schema/InformationSchema.java#L509>

yes! With <https://druid.apache.org/docs/latest/configuration/index.html#composing-emitter-module|composing> emitter, you could have multiple emitters:
```druid.emitter=composing
druid.emitter.composing.emitters=["http","graphite"]
druid.emitter.http.recipientBaseUrl=<http://foo:1234/druid>
druid.emitter.graphite.hostname=bar
...```

also, if you want to demux event feeds and send them to separate emitter modules, you could try <https://druid.apache.org/docs/latest/configuration/index.html#switching-emitter|switching> emitter

The newer segments are always loaded first.

An interesting thing I saw when I switched to http segment loading was that my streaming ingestion task was waiting to handoff segment for a really long time. This happen at the same time the cluster is loading many many segments for all the existing datasources. Since newer segments are always loaded first, I think what happened was that I was using the default value for `druid.coordinator.loadqueuepeon.http.batchSize`  which is 1. This causes the coordinator load/drop cycle to take a really long time, preventing the newer segment from my realtime ingestion task being loaded. I increased `druid.coordinator.loadqueuepeon.http.batchSize` and no longer see this issue.

Also, reducing `maxSegmentsInNodeLoadingQueue` also help coordinator run to cycle faster as this config determines the max load/drop segment operations assigned to each Historical per cycle. Coordinator cycle faster help the handoff realtime segment gets assign faster

Could you please check the historical logs corresponding to the stuck load queue?
You could try restarting the coordinator to see if it helps clear the bad segment from the load queue.
(Also, please switch from `curator` to `http` loadqueues before restarting the coordinator in case you are using the former)

```Unannouncing segment[&lt;segment&gt;] at path[..]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Completely removing &lt;segment&gt; in [30,000] millis
org.apache.druid.server.SegmentManager - Attempting to close segment &lt;segment&gt;```

Could you please share the replication factor of your segments and also run the following sys table query from the druid console?
`select num_replicas, count(*) from sys.segments group by 1`

The above query should give the count of segments with various replication levels. It's possible that the cluster is overreplicated and segments are being dropped

Oh, overreplication is not the problem here then

&gt; only these 3 log lines
Are these the only 3 log lines in total?
I interpreted it as these 3 log lines are repeated continuously on the historical earlier

Do you have any insight on `org.apache.druid.server.coordinator.ReplicationThrottler:` errors ?
Why do they occur ?

`Replicant create queue stuck after 3 runs`
I increased it from 10 to 100 to see if it can reduce the # of errors.

Screenshot 2023-06-12 at 13.05.54.png

Screenshot 2023-06-12 at 13.07.26.png

Hi Daniel, Two things come to mind (not 100% sure on either of these):

• Is there an active Supervisor task on this datasource?  If so then I would expect the DAY time chunk lock from that Supervisor to prevent an auto-compaction from being scheduled for the current year. 
• The auto-compact offset itself might be preventing auto-compaction from checking the current year since it is not completely older than P2D.  (i.e. on Jan 3 this would kick in to compact the prior year).  If this is the case then try P0D or P0Y offsets to see if that makes a difference.


Hey <@U04C4ADUGD9> , no, this is not a data source with live ingestion. Only batching, always writing or rewriting D and D-1, that’s why I set the P2D offset delay, so no time locks or interference

The second possibility is exactly what I was afraid of. I started the spec thinking like ‘the offset is used for druid to look what are the segments to ignore when checking which ones are compacted’.

But if it’s like it’s gonna run after the end of the segment + offset… then I need to trigger this via periodic reindexes (via Airflow) or change autocompaction to WEEK (+2d offset) granularity and every now and then manually force a YEAR merge.

Hi Daniel, if you want to try to trick the auto-compaction process, ingest a dummy record with __time value "2024-01-01" ... since the auto-compaction offset is from the LATEST time period, not current date/time, a P1Y offset should then try to operate on 2023 ...

I have wrestled with auto-compaction contention plenty of times when "future" data was  inadvertantly added to the customer, causing auto-compaction to try to compact current and even future time periods when we didn't want that ... in this case it might be a benefit ... :crossed_fingers:

You just need to insert a record that doesn't disrupt your application.

But adding a ‘2024-01-01’ record would make my segments from today and yesterday considered ‘compactable’, wouldn’t it?

That would hurt us as these segments arent final.

Are you replacing todays/yesterdays segments with overwrite?  Or append?

If overwrite, segment overshadowing allows overwrite of data with mixed segment granularity, e.g. you can ingest/overwrite a DAY segment on top of a YEAR segment, for query purposes that new DAY's data will overshadow/replace the old DAYs data that is contained in the YEAR segment, and then the next time compaction runs it will merge the DAY segment into the YEAR segment.

If append, then the granularity of the segments must match ... so in that case you would just change your ingestion to use YEAR granularity instead of DAY.

Would one of those two scenarios work for you?

I see a job running multiple times daily, overwriting data for D and D-1 (with corrections and new data).

If it was done only once - or at least if it was only new data -, I’d be ok with the merge. But this way looks like autocompaction will continously merging partial data into this block where it won’t be replaced if needed.

It's definitely your call, and based on what I know today I don't see a way around that particular scheduling issue.  By default compaction is kicked off every 30 minutes, so if you have several batch updates a day, then this datasource would be compacted that many times.  And if compacting a YEAR is very resource intensive, then that is a reason not to do it.  However if that is the case and MONTH granularity would work better with multiple daily compactions, maybe that's something to consider.

That’s exactly what I did yesterday, I changed the autocompaction granularity to MONTH, with good results.

It’s already a huge improvement from the small daily ones, I can set the partitioning and the offset. There will be more daily indexes close to EOM than I would like, but still huge gains compared to the previous situation.

For now is more than good enough, I’ll explore more in the future. Thanks <@U04C4ADUGD9>

In a parallel problem… can autocompaction be used for deeper rollups?

So I have this database with HOUR query granularity and DAILY segment granularity.
Can I set autocompaction to after a month, to compact/rollup data into DAILY query granularity and MONTH segment granularity

I imagine it’s doable, but I couldn’t find examples of it.

Here is the doc page that mentions rollups:  <https://druid.apache.org/docs/latest/data-management/compaction.html#rollup> ... I haven't tried it myself but the docs state that it will continue to roll up if the original segments had rollup enabled ... which makes sense.

If you don't know if the original segments were ingested with rollup:true, then I don't know if it is possible to force rollup during compaction ...

I don't know whether it will automatically go back and check for that.  You might try restarting the coordinator, then it might build a new segment list.

Sounds a lot like max_map_count <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#max_map_count>, and your # of segments just crossed the line.

See this link <https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html> a recent conversation <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1686020575973389>

Congratulations on outgrowing the defaults!

This alert means that the Coordinator has been wanting to add a replica of a segment for at least `replicantLifetime` runs (by default: 15 runs) and has not been able to. By default, Coordinator runs occur once a minute, so that's 15 minutes. However, this can be overridden by setting `druid.coordinator.period` to something else. If an environment does happen to have a more aggressive setting for `druid.coordinator.period` then I would suggest either restoring the default setting, or increasing `replicantLifetime` to make the alerting less sensitive.
Search on this page for `replicantLifetime` and `druid.coordinator.period` for more details: <https://druid.apache.org/docs/latest/configuration/index.html>

Also check if any issues with datanodes , are they often going down?

Screenshot 2023-06-13 at 3.51.59 PM.png

The datanodes are not going down. We are using the default for coordinator.period which is 1 min. I decreased the replicant lifetime to 3 to see if this would decrease the amount the errors. We also experiencing this issue <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1686513388513769>
This is the distribution of errors.

I just noted and my coordinator heap is much smaller. Im gonna increase its heap to see if it helps
```You can set the Coordinator heap to the same size as your Broker heap, or slightly smaller: both services have to process cluster-wide state and answer API requests about this state.```

Well i dont think that worked :slightly_smiling_face:

Screenshot 2023-06-13 at 11.32.45 AM.png

Do you have any monitoring set up on your master nodes? I've seen this happen when the leader master node dies.

In our case, it was due to memory pressure from having too many tasks for the size of the master

we are seeing network error mostly on console after few mins in every page just not above page.. is it still related to memory pressure

I'd start by trying to isolate which service / if a service is being restarted. Then we can investigate why that particular service is having problems

```org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[&lt;datasource&gt;_2023-05-20T00:00:00.000Z_2023-05-21T00:00:00.000Z_2023-05-23T12:09:11.679Z_1]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:289) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:266) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:343) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler$1.lambda$addSegment$1(SegmentLoadDropHandler.java:572) ~[druid-server-25.0.0.jar:25.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_292]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
Caused by: org.apache.druid.java.util.common.IAE: use read(ByteBuffer buffer, ObjectStrategy&lt;T&gt; strategy, SmooshedFileMapper fileMapper) to read version 2 indexed.
        at org.apache.druid.segment.data.GenericIndexed.read(GenericIndexed.java:185) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.query.aggregation.datasketches.tuple.ArrayOfDoublesSketchMergeComplexMetricSerde.deserializeColumn(ArrayOfDoublesSketchMergeComplexMetricSerde.java:71) ~[?:?]
        at org.apache.druid.segment.serde.ComplexMetricSerde.deserializeColumn(ComplexMetricSerde.java:60) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.serde.ComplexColumnPartSerde.lambda$getDeserializer$0(ComplexColumnPartSerde.java:89) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.column.ColumnDescriptor.read(ColumnDescriptor.java:111) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.deserializeColumn(IndexIO.java:796) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.registerColumnHolder(IndexIO.java:771) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.registerColumnHolders(IndexIO.java:730) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:654) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:202) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.loading.MMappedQueryableSegmentizerFactory.factorize(MMappedQueryableSegmentizerFactory.java:49) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.loading.SegmentLocalCacheLoader.getSegment(SegmentLocalCacheLoader.java:71) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.SegmentManager.getSegmentReference(SegmentManager.java:325) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:268) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:281) ~[druid-server-25.0.0.jar:25.0.0]
        ... 10 more```

<@U030C8H59T8> Do you have any idea about this ? <https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/data/GenericIndexed.java#L184|We are getting the VERSION_TWO> and throwing an exception from buffer while deserializing the theta sketch column
```    final GenericIndexed&lt;ArrayOfDoublesSketch&gt; ge = GenericIndexed.read(buffer, ArrayOfDoublesSketchObjectStrategy.STRATEGY);```

hmm, this is in ArrayOfDoubles serde i guess?

version 2 is for when the GenericIndexed is larger than java bytebuffer size (2gb)

it splits across multiple internal smoosh files

however, it takes different methods to read and stuff so if it isn’t actually wired up then you run into this explosion

yeah, we do not have smoosh file passed to the calling method read here.

i need to look closer at code to see if its setup to be able to handle v2 (or even should write it in the first place)

v2 can be handled in using this read <https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/data/GenericIndexed.java#L193>

i guess `LargeColumnSupportedComplexColumnSerializer` writes v2s, it does look wired up to potentialy use the v2 serializer

its a bug, it should be passing in the filemapper from the columnbuilder

should look like this
```  @Override
  public void deserializeColumn(final ByteBuffer buffer, final ColumnBuilder builder)
  {
    final GenericIndexed&lt;ArrayOfDoublesSketch&gt; ge = GenericIndexed.read(buffer, ArrayOfDoublesSketchObjectStrategy.STRATEGY, builder.getFileMapper());
    builder.setComplexColumnSupplier(new ComplexColumnPartSupplier(getTypeName(), ge));
  }```


as a workaround you can probably reduce segment max row count

well you’d have to reingest data though

a bit surprised no one has run into this yet tbh

<@U031QS4LQV6> can you please file a issue on with the details, we can take this up and fix it.
Thank you

or if you’re motivated, feel free to submit a PR since the fix is that snippet pasted above

it looks like only ArrayOfDoubles sketch has the problem, the others are using the right method

Thanks <@U04EWB06GR4> and <@U030C8H59T8> for looking into this issue  . never file a issue where should i do it

<https://github.com/apache/druid/issues/new/choose>

yeah, this code change will be part of druid-datasketches extension and it should be easy to test it as well.

ish, the tricky part of testing is that this path is only hit when column is larger than 2gb, though iirc there are methods to make that size smaller

<https://github.com/apache/druid/issues/14418>

<@U030C8H59T8> for now we will reingest data with less no.of rows per segment

<https://github.com/apache/druid/pull/14429>

Screenshot 2023-06-13 at 3.59.55 PM.png

Hi <@U03BT5HU5BL>. I think you just need to change the quotes on the clustered by column as in:
```REPLACE INTO "SpanView"
OVERWRITE WHERE __time &gt;= TIMESTAMP '2023-06-09 07:00:00' AND __time &lt; TIMESTAMP '2023-06-09 08:00:00'
select * from "SpanView"
WHERE __time &gt;= TIMESTAMP '2023-06-09 07:00:00' AND __time &lt; TIMESTAMP '2023-06-09 08:00:00'
PARTITIONED BY HOUR
CLUSTERED BY "tenantId"```
with single quotes it is just a literal value.

in middleManager log getting:
```2023-06-14T01:25:12,327 WARN [DataStreamer for file /druid/task-logs/index_kafka_.LOGGING _3d968870f9eb302_phgoknak] org.apache.hadoop.hdfs.DataStreamer - DataStreamer Exception
java.lang.NoSuchMethodError: 'sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner()'```


Hi <@U053ZB6A7S9> is this issue fixed i am also facing this while running `index_hadoop` with s3 storage

<@U031QS4LQV6> well with HDFS we are stuck with java 8.. cuz hdfs libraries only work on java 8... so i have to roll back to that version.
But with S3....
you need to add to jvm config
```--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED \
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \
--add-opens=java.base/java.nio=ALL-UNNAMED \
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED \
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED \
--add-opens=java.base/java.io=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED```
Or upgrade to druid 27+ since thats when they fully support it
<https://github.com/apache/druid/releases#27.0.0-highlights-java-17-support>

then yeah.. add these options to jvm config in middlemanager and restart and that should resolve that error

you can read about these options here:
<https://druid.apache.org/docs/25.0.0/operations/java.html#strong-encapsulation>

I think the fully qualified monitor name isn’t specified in config? Try:
`druid.monitoring.monitors=["org.apache.druid.server.metrics.QueryCountStatsMonitor"]`

(I just saw this message, so copying the the same response from the other thread :slightly_smiling_face: )

Is there an error in that log? Can you post the text for it?

The issue got fixed, one of the data source deletion got stuck I guess and it was keep on trying deletion. That dataset got ingested with 40K + small segments. After cleaning up everything the issue got fixed.

`2023-06-14T09:37:53+00:00 startup service coordinator`
Setting druid.host=10.101.45.107 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.secretKey=45Olx53YqDshGDqvqxjWiiyA7L5KOJhzY6whUUnh in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.type=s3 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage","druid-s3-extensions","druid-kafka-indexing-service"] in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.baseKey=dev/segments in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.endpoint.url=<http://s3.amazonaws.com|s3.amazonaws.com> in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.s3Bucket=nebulastaging-druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.http.recipientBaseUrl=<http://druid_exporter_url>:druid_exporter_port/druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.type=postgresql in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.bucket=nebulastaging-druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.endpoint.signingRegion=eu-north-1 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.connectURI=jdbc:<postgresql://dev-druid-postgresql:5432/druid> in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.accessKey=AKIAWLVNMFA7M2EVDFMS in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.type=s3 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.disableAcl=true in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.user=druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.directory=/opt/data/indexing-logs in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.zk.service.host=dev-druid-zookeeper-headless:2181 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.endpoint.serviceName=s3 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter=noop in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.s3Prefix=dev/druid-logs in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.storageDirectory=/mnt/druid/druid/var/druid/segments in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.password=druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.cache.sizeInBytes=1000000000 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.logging.logLevel=error in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.broker.cache.populateCache=true in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.broker.cache.useCache=true in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Terminating due to java.lang.OutOfMemoryError: Java heap space

`Terminating due to java.lang.OutOfMemoryError: Java heap space`



*`Anant Sharma`*

*config*:
  *DRUID_XMX*: 512m
  *DRUID_XMS*: 512m

should we increase this in this case

```2023-06-14T09:50:32+00:00 startup service coordinator
Setting druid.host=10.101.57.60 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.secretKey=45Olx53YqDshGDqvqxjWiiyA7L5KOJhzY6whUUnh in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.type=s3 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage","druid-s3-extensions","druid-kafka-indexing-service"] in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.baseKey=dev/segments in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.endpoint.url=<http://s3.amazonaws.com|s3.amazonaws.com> in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.s3Bucket=nebulastaging-druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.http.recipientBaseUrl=<http://druid_exporter_url>:druid_exporter_port/druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.type=postgresql in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.bucket=nebulastaging-druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.connectURI=jdbc:<postgresql://dev-druid-postgresql:5432/druid> in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.endpoint.signingRegion=eu-north-1 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.accessKey=AKIAWLVNMFA7M2EVDFMS in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.type=s3 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.disableAcl=true in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.user=druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.directory=/opt/data/indexing-logs in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.s3.endpoint.serviceName=s3 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.zk.service.host=dev-druid-zookeeper-headless:2181 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter=noop in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.s3Prefix=dev/druid-logs in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.storageDirectory=/mnt/druid/druid/var/druid/segments in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.cache.sizeInBytes=1000000000 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.password=druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.logging.logLevel=error in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.broker.cache.populateCache=true in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.broker.cache.useCache=true in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release```

not getting that error anymore but the situation is same

Please waiting for another service starting and double check

thanks for your help it was the same issue <@U059U81SRB3>

Maybe u can using doctor Druid for troubleshooting problem

yeah i think i need to figure them out , Those looks to suspicious

Btw do you have any idea what could be the reason if compaction stuck on Running from a log time

Hi guys, need some help with removing segments from a data source. I am using the wrench icon on a data source to mark segments unused and then running the kill task for unused segments. The time stamp I'm using is for the interval - `20150912-01-01T01:00:00.000Z/20150912-01-01T02:00:00.000Z`but  the interval being picked up  in the logs is totally random ie `1000-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z` and of-course my data is not getting taken out.

Any ideas on what might be going on?

The interval seems to be saying year 20150912.  What date are you trying to use?

Hi <@U0411DE2SS0>
That's the one I am setting the retention to.

This interval - 20150912-01-01T01:00:00.000Z/20150912-01-01T02:00:00.000Z

Year 20150912, ie about 20M years in the future?

Oh, then maybe you want something like 2015-09-12T01:00:00.000Z-2015-09-12T02:00:00.000Z?

Oh wait, do you have segments in that time interval, and you're using the UI to try to drop them, right?

That's what I'm specifying, as posted above

If you mark as unused in GUI, then issue kill in GUI, this is what happens?

I created a datasource from the packaged tutorial

Check out the timestamp on the log output

:disappointed:  I wonder if it's some kind of overflow.  Do you need the other segments (ie, the datasource)?  If not, maybe a "mark all segments unused", etc, would work?

Yepp that works, if I mark all as unused and then kill, it wipes out the datasource and all of the data is deleted from the deep storage

Which tutorial is this?  If this is what it does, maybe the tutorial needs to get fixed.

I'm wondering whether there's something in the config I can look at

So basically I used the tutorial stuff to create a datasource with 24 hourly segments.

I'm not sure about the config.  I thought of using API instead of GUI, but I have a feeling the same thing might happen.

Prolly, I'm gonna have to look into the config and setup.

Ah actually I'm referring to the tutorial folder under the druid install path

There's a wikipedia data tar.gz file one can use as input data and then copy one of the json files to run an ingestion workload to create a datasource

But regardless, when I set up retention rules and issue a kill task it should remove the segment and wipe out the data

Have you played around with that feature? Maybe I'm missing something out

It should work, I don't know why the kill task is changing the interval like that.  I suspect the year is too big, I'll play around and check it out, maybe file a bug.  (Or, if you have the json and ingestion spec, feel free to create a bug report.)  Maybe the ingestion spec isn't handling the input timestamp format correctly, causing weird years.

Sounds good mate. I'll try it out with a few tweaks to see whether I can go beyond this. Let me share the ingestion spec here.

Yes,For example , " druid.indexer.storage.recentlyFinishedThreshold=PT30M "  adding  with overlord runtime properties and it will remove the tasks from sys.tasks table  , tasks which are older than 30 minutes as per my configured duration and you will not able to see those tasks in druid console. You can configure as suggested earlier druid.indexer.storage.recentlyFinishedThreshold=PT6H or PT4H . For cleaning tasks from(druid_tasks) metaDB its better to configure auto log kill setting for auto metadata cleanup than doing it manually.
Please see "druid.indexer.logs.kill.enabled" and related parameters in <https://druid.apache.org/docs/latest/configuration/index.html#log-retention-policy>

<https://docs.imply.io/2021.09/druid/operations/clean-metadata-store/#configure-automated-metadata-cleanup>

<@U04GEHS4TRU> we are looking into it, can you take thread dump if possible?
Also what is the jdk version that you are using ?

Did you find any root cause of for this,  I have also noticed the same problem with broker, running JDK 11. upon restart of services things started working.

i believe this is related to parallel merge for metadata queries.

*<@U04GEHS4TRU> Could you share your solution? That will help me much!*