https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
v
Overlord Logs:
Copy code
2022-09-16T09:32:25,820 DEBUG [HttpClient-Netty-Worker-5] org.apache.druid.java.util.http.client.NettyHttpClient - [POST <http://192.168.28.87:8091/druid/worker/v1/task/index_kafka_networks_stage_72d587d4679eb62_jmloppea/shutdown>] Got response: 500 Server Error
         vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
27212.057: RevokeBias                       [     159          0              0    ]      [     0     0     0     0     0    ]  0   
2022-09-16T09:32:25,820 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: 192.168.28.87:8091, status 500 Server Error, response: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 500 org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [MyBasicMetadataAuthenticator]</title>
</head>
<body><h2>HTTP ERROR 500 org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [MyBasicMetadataAuthenticator]</h2>
<table>
<tr><th>URI:</th><td>/druid/worker/v1/task/index_kafka_networks_stage_72d587d4679eb62_jmloppea/shutdown</td></tr>
<tr><th>STATUS:</th><td>500</td></tr>
<tr><th>MESSAGE:</th><td>org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [MyBasicMetadataAuthenticator]</td></tr>
<tr><th>SERVLET:</th><td>org.eclipse.jetty.servlet.DefaultServlet-46ea78f0</td></tr>
<tr><th>CAUSED BY:</th><td>org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [MyBasicMetadataAuthenticator]</td></tr>
</table>
<h3>Caused by:</h3><pre>org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [MyBasicMetadataAuthenticator]
	at org.apache.druid.security.basic.authentication.validator.MetadataStoreCredentialsValidator.validateCredentials(MetadataStoreCredentialsValidator.java:64)
	at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:200)
	at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
	at java.lang.Thread.run(Thread.java:748)
</pre>

</body>
</html>

         vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
27212.057: RevokeBias                       [     159          0              0    ]      [     0     0     0     0     0    ]  0   
2022-09-16T09:32:25,821 ERROR [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown failed for index_kafka_networks_stage_72d587d4679eb62_jmloppea! Are you sure the task was running?
         vmop                    [threads: total initially_running wait_to_block]    [time: spin block sync cleanup vmop] page_trap_count
Middle Manager Logs:
Copy code
2022-09-16T09:21:36,325 DEBUG [main-SendThread(druid-zookeeper-1.druid-zookeeper-headless.druid.svc.cluster.local:2181)] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x2009bc5e62d0008 after 0ms
2022-09-16T09:21:38,166 DEBUG [prometheus-http-1-2] com.sun.net.httpserver - GET /metrics HTTP/1.1 [200  OK] ()
2022-09-16T09:21:40,408 DEBUG [prometheus-http-1-3] com.sun.net.httpserver - GET /metrics HTTP/1.1 [200  OK] ()
2022-09-16T09:21:40,815 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (9 of 9) in 58,764ms.
org.apache.druid.java.util.common.IOE: No known server
	at org.apache.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:267) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:122) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:252) ~[?:?]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:192) ~[?:?]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:190) ~[?:?]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:122) ~[?:?]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:97) [druid-core-0.23.0.jar:0.23.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2022-09-16T09:21:43,456 DEBUG [MonitorScheduler-0] org.apache.druid.emitter.prometheus.PrometheusEmitter - Unmapped metric [jvm/cpu/user]
Historicals Logs:
Copy code
2022-09-16T08:42:57,813 DEBUG [main-SendThread(druid-zookeeper-2.druid-zookeeper-headless.druid.svc.cluster.local:2181)] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x303b27dcd460007 after 0ms
2022-09-16T08:43:01,090 WARN [BasicAuthenticatorCacheManager-Exec--0] org.apache.druid.java.util.common.RetryUtils - Retrying (9 of 9) in 66,809ms.
org.apache.druid.java.util.common.IOE: No known server
	at org.apache.druid.discovery.DruidLeaderClient.getCurrentKnownLeader(DruidLeaderClient.java:267) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.discovery.DruidLeaderClient.makeRequest(DruidLeaderClient.java:122) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.tryFetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:252) ~[?:?]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$fetchUserMapFromCoordinator$1(CoordinatorPollingBasicAuthenticatorCacheManager.java:192) ~[?:?]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.fetchUserMapFromCoordinator(CoordinatorPollingBasicAuthenticatorCacheManager.java:190) ~[?:?]
	at org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager.lambda$start$0(CoordinatorPollingBasicAuthenticatorCacheManager.java:122) ~[?:?]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:55) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$1.call(ScheduledExecutors.java:51) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:97) [druid-core-0.23.0.jar:0.23.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2022-09-16T08:43:01,613 DEBUG [prometheus-http-1-5] com.sun.net.httpserver - GET /metrics HTTP/1.1 [200  OK] ()
l
For how long was the Zookeeper down. Also were there any changes to the metadata store during the time?
v
It would have been down max for 2 minutes. No, there was no change in the metadata.
g
When ZK goes down, you may need to restart your Druid Coordinator/Overlord if they got stuck not being able to elect a leader Setting these properties is also helpful, as it moves 3 important pieces of functionality from ZK -> HTTP, which increases reliability when recovering from ZK unavailability
Copy code
druid.indexer.runner.type = httpRemote
druid.serverview.type = http
druid.coordinator.loadqueuepeon.type = http
Open in Slack
PreviousNext
Powered by