This message was deleted.
# troubleshootings
Slackbot
06/09/2023, 6:44 AMThis message was deleted.
a
Ananya Bhushan
06/09/2023, 6:46 AMThese are the things I have added in the common.runtime.properties file in the druid/auto dir
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor", "org.apache.druid.server.metrics.QueryCountStatsMonitor","org.apache.druid.segment.realtime.RealtimeMetricsMonitor"]
druid.emitter=http
druid.module.enable="druid-metrics"
druid.emitter.logging.logLevel=info
druid.monitoring.emissionPeriod = PT1M
druid.emitter.http.minHttpTimeoutMillis = 1000
druid.emitter.http.recipientBaseUrl = http://localhost:8090/druid/metrics
a
Abhishek Balaji Radhakrishnan
06/09/2023, 7:38 AMhi there, the stack trace tells me that Druid isn’t able to find the extension
/Users/ananya/Downloads/apache-druid-26.0.0/extensions/druid-extensions-contrib-firehose-segment-pushloadLista
Ananya Bhushan
06/09/2023, 9:29 AMYes, my bad. I have removed that extension. Now, I am getting this error
Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) No implementation for java.util.List<org.apache.druid.segment.realtime.FireDepartment> was bound.
while locating java.util.List<org.apache.druid.segment.realtime.FireDepartment>
for the 1st parameter of org.apache.druid.segment.realtime.RealtimeMetricsMonitor.<init>(RealtimeMetricsMonitor.java:52)
while locating org.apache.druid.segment.realtime.RealtimeMetricsMonitor
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler
at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)
1 error
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)
at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) No implementation for java.util.List<org.apache.druid.segment.realtime.FireDepartment> was bound.
while locating java.util.List<org.apache.druid.segment.realtime.FireDepartment>
for the 1st parameter of org.apache.druid.segment.realtime.RealtimeMetricsMonitor.<init>(RealtimeMetricsMonitor.java:52)
while locating org.apache.druid.segment.realtime.RealtimeMetricsMonitor
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler
at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.server.metrics.MetricsModule)
while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:470)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:184)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49)
at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118)
at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73)
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85)
... 2 more
a
Abhishek Balaji Radhakrishnan
06/09/2023, 3:45 PMThe
RealTimeMetricsMonitormonitorsa
Ananya Bhushan
06/10/2023, 6:56 AMWhen I removed
RealTimeMetricsMonitora
Abhishek Balaji Radhakrishnan
06/10/2023, 4:43 PMIt appears to be the same issue as someone that ran into this recently - https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1684492769746299. Please try the suggestion in the thread - add the
QueryCountStatsProviderdruid.indexer.runner.javaOptsQueryCountStatsMonitorruntime.propertiescommon.runtime.propertiesa
Ananya Bhushan
06/11/2023, 6:11 PMYes, this thread worked well for me and now I am able to see the console. But the problem still remains that I want druid to emit query/time metrics on the http endpoint but it is not. What I am getting in the error message in my spring boot application is this :
Ananya Bhushan
06/11/2023, 6:12 PM[2m2023-06-11T233557.928+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-8][0;39m [36mo.s.w.s.m.m.a.HttpEntityMethodProcessor [0;39m [2m:[0;39m Using 'application/json', given [/] and supported [application/json, application/*+json]
[2m2023-06-11T233557.928+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-8][0;39m [36mo.s.w.s.m.m.a.HttpEntityMethodProcessor [0;39m [2m:[0;39m Writing [{timestamp=Sun Jun 11 233557 IST 2023, status=404, error=Not Found, path=/druid/metrics}]
[2m2023-06-11T233557.929+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-8][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m Exiting from "ERROR" dispatch, status 404
[2m2023-06-11T233558.125+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m POST "/druid/metrics", parameters={}
[2m2023-06-11T233558.126+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.w.s.handler.SimpleUrlHandlerMapping [0;39m [2m:[0;39m Mapped to ResourceHttpRequestHandler [classpath [META-INF/resources/], classpath [resources/], classpath [static/], classpath [public/], ServletContext [/]]
[2m2023-06-11T233558.127+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.w.s.r.ResourceHttpRequestHandler [0;39m [2m:[0;39m Resource not found
[2m2023-06-11T233558.127+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m Completed 404 NOT_FOUND
[2m2023-06-11T233558.127+05:30[0;39m [32mDEBUG[0;39m [35m7695[0;39m [2m---[0;39m [2m[nio-8090-exec-9][0;39m [36mo.s.web.servlet.DispatcherServlet [0;39m [2m:[0;39m "ERROR" dispatch for POST "/error", parameters={}
Ananya Bhushan
06/11/2023, 6:13 PMSo, can you please guide why is druid not emitting metrics on that http endpoint that I have configured?
a
Abhishek Balaji Radhakrishnan
06/11/2023, 7:03 PMGood to hear about the progress. Without knowing details of the actual setup, I’ll leave a few pointers to help troubleshoot:
1. Are there any errors and warnings in the Druid logs related to metrics not being emitted or so?
2. Is your configured http endpoint reachable from Druid? Can you successfully try a curl request with a
GETSELECTQueryCountStatsMonitordruid.emitter.http.flush*a
Ananya Bhushan
06/12/2023, 7:04 AMThanks for your help Abhishek. As you said, I have checked the above points. I could not find any warnings or errors in druid's logs. And sending post requests through Postman is working well too. I have run a few queries in the console but druid still is not generating metrics for those queries. I have done the following configuration changes in the common.runtime.properties file of ../auto/_common. Am I doing it right?
Ananya Bhushan
06/12/2023, 7:05 AMdruid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=http
druid.module.enable="druid-metrics"
druid.emitter.logging.logLevel=info
druid.monitoring.emissionPeriod = PT1M
druid.emitter.http.minHttpTimeoutMillis = 1000
druid.emitter.http.recipientBaseUrl = http://localhost:8090/druid/metrics
a
Abhishek Balaji Radhakrishnan
06/13/2023, 1:45 AMok. I see your common runtime.properties. A few follow up questions:
1. What’s
druid.module.enable="druid-metrics"JvmMonitorQueryCountStatsMonitordruid.monitoring.monitorsauto/broker/runtime.propertiesauto/historical/runtime.propertiesa
Ananya Bhushan
06/13/2023, 4:07 AMYes, I think druid.module.enable="druid-metrics is wrong. No, I do not see any jvm metrics at the endpoint. When I added, I got the following error:
Ananya Bhushan
06/13/2023, 4:07 AMException in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Problem parsing object at prefix[druid.monitoring]: Cannot construct instance of
org.apache.druid.server.metrics.MonitorsConfiga
Abhishek Balaji Radhakrishnan
06/14/2023, 3:33 AM@Ananya Bhushan, ok just getting back. Looking at the stack trace, it seems like the fully qualified name for the monitor isn’t specified - it should be something like
"org.apache.druid.server.metrics.QueryCountStatsMonitor"druid.monitoring.monitors=["org.apache.druid.server.metrics.QueryCountStatsMonitor"]a
Ananya Bhushan
06/14/2023, 3:43 AMThe console is working now. But the problem of druid not generating metrics still persists!
a
Abhishek Balaji Radhakrishnan
06/14/2023, 3:57 AMI’d suggest isolating the problem to a single process, say historical. Do you see any errors or warnings in `historical.log`:
Copy code
ERROR [HttpPostEmitter-1] org.apache.druid.java.util.emitter.core.HttpPostEmitter - Failed to send events to url[<http://localhost:8090/druid/metrics>]python3 -m http.server 8090 Copy code
WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 2) in 1,976ms.
org.apache.druid.java.util.common.ISE: Emissions of events not successful[501: Unsupported method ('POST')], with message[<!DOCTYPE HTML>a
Ananya Bhushan
06/14/2023, 5:01 AMI do not see any error or warning in any of the log files. At this
Copy code
<http://localhost:8090/druid/metrics>Ananya Bhushan
06/14/2023, 7:28 AMOh i was looking at some other log file. Yes, the historical.log file does have error messages like these:
WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 1,062ms.
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: localhost/[00000001]8090
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:202) ~[async-http-client-2.5.3.jar:?]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.send(HttpPostEmitter.java:783) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.access$1900(HttpPostEmitter.java:464) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:667) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread$1.perform(HttpPostEmitter.java:663) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.sendWithRetries(HttpPostEmitter.java:661) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emit(HttpPostEmitter.java:569) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.emitBatches(HttpPostEmitter.java:545) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.emitter.core.HttpPostEmitter$EmittingThread.run(HttpPostEmitter.java:503) ~[druid-processing-26.0.0.jar:26.0.0]
Caused by: java.net.ConnectException: Connection refused: localhost/[00000001]8090
at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) ~[async-http-client-2.5.3.jar:?]
at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) ~[async-http-client-2.5.3.jar:?]
at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) ~[async-http-client-2.5.3.jar:?]
at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) ~[async-http-client-2.5.3.jar:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
a
Abhishek Balaji Radhakrishnan
06/14/2023, 6:07 PMThe warning is indicative that the http server is refusing the connection from the historical:
Copy code
WARN [HttpPostEmitter-1] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 1,062ms.
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8090127.0.0.1localhostlocalhosta
Ananya Bhushan
06/21/2023, 5:56 PMHi, I did the change as you suggested but nothing changed. I have changed my server's address to 127.0.0.1 and accordingly changed the endpoint's url as well but i am still getting the same error
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused: /127.0.0.1:8090
a
Abhishek Balaji Radhakrishnan
06/25/2023, 9:42 PMsorry, I’m not sure how to debug the connection refused issue coming from the http server program. If you can set up a simple python web server (e.g.,
python3 -m http.server 8090