https://linen.dev logo
Join Slack
Powered by
# troubleshooting
  • s
    This message was deleted.
    j
    j
    k
    • 4
    • 34
  • s
    This message was deleted.
    a
    u
    +2
    • 5
    • 8
  • r
    What are the reasons MM can get blacklisted? Is it failing jobs? if so, I think that failures dues to losing lock should not count
    Last completed task: 2023-03-23T095547.461Z Blacklisted until: 2023-03-23T101047.461Z
  • l
    Hi all, My Druid cluster (in k8s) is suffering from a huge performance degradation that I'm unable to find the root cause. We use Druid to power some analytics dashboards and we notice that upon loading a dashboard any other query starts timing out (no Druid time limit has been set it's our load balancer which timeouts connections after 60 seconds). The cluster enter in a degraded state, we tried running a single query that tipically take <1s and it keeps running for more than 60s. The cluster takes around and hour to perform back to normal but forcing a restart on the broker services also mitigates the issue. I've attached an screenshot of our cluster topology and I been able to collect log lines (broker) associated to the dashboard queries that trigger the issue. These queries seems to fail because of the subquery results default limitation of number of rows
    Copy code
    2023-03-22T17:06:40+00:00	2023-03-22T17:06:40,593 WARN [sql[a97928d9-bd83-4478-840f-9d6332a33916]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [553272ba-21ec-4655-bf43-704133b9bcbb] (org.apache.druid.query.ResourceLimitExceededException: Subquery generated results beyond maximum[100000])
2023-03-22T17:06:41+00:00	2023-03-22T17:06:41,409 WARN [sql[d8a66e60-4cee-4388-bfac-fa18edb370f5]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [6652a9f4-3980-4533-b861-7b0e37eccdc2] (org.apache.druid.query.ResourceLimitExceededException: Subquery generated results beyond maximum[100000])
2023-03-22T17:06:42+00:00	2023-03-22T17:06:42,919 WARN [sql[b344d453-54ed-4814-97f1-2711b9478c87]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [1167828c-f2fb-46e5-bdca-539e979214e3] (org.apache.druid.query.ResourceLimitExceededException: Subquery generated results beyond maximum[100000])
    but then another cryptic error start appearing rigth afterwards
    Copy code
    2023-03-22T17:11:35+00:00	2023-03-22T17:11:35,564 WARN [ForkJoinPool-1-worker-14] org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence - Failed to close result cursors
org.apache.druid.query.QueryTimeoutException: Query[c5df16ae-12f3-45aa-8fe9-c18bed6c9ef6] url[<http://172.42.0.173:8100/druid/v2/>] timed out.
	at org.apache.druid.client.DirectDruidClient$1.checkQueryTimeout(DirectDruidClient.java:427) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.client.DirectDruidClient$1.access$500(DirectDruidClient.java:173) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.client.DirectDruidClient$1$2.hasMoreElements(DirectDruidClient.java:282) ~[druid-server-0.23.0.jar:0.23.0]
	at java.io.SequenceInputStream.nextStream(SequenceInputStream.java:109) ~[?:1.8.0_275]
	at java.io.SequenceInputStream.close(SequenceInputStream.java:232) ~[?:1.8.0_275]
	at com.fasterxml.jackson.dataformat.smile.SmileParser._closeInput(SmileParser.java:309) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
	at com.fasterxml.jackson.dataformat.smile.SmileParserBase.close(SmileParserBase.java:384) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
	at org.apache.druid.client.JsonParserIterator.close(JsonParserIterator.java:138) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.utils.CloseableUtils.closeAndWrapExceptions(CloseableUtils.java:139) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.client.DirectDruidClient$3.cleanup(DirectDruidClient.java:515) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.client.DirectDruidClient$3.cleanup(DirectDruidClient.java:498) ~[druid-server-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.guava.BaseSequence$1.close(BaseSequence.java:136) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.close(ParallelMergeCombiningSequence.java:1072) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.io.Closer.close(Closer.java:182) ~[druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.utils.CloseableUtils.closeAndSuppressExceptions(CloseableUtils.java:162) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence.closeAllCursors(ParallelMergeCombiningSequence.java:1385) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence.access$400(ParallelMergeCombiningSequence.java:63) [druid-core-0.23.0.jar:0.23.0]
	at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$MergeCombineAction.compute(ParallelMergeCombiningSequence.java:666) [druid-core-0.23.0.jar:0.23.0]
	at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_275]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_275]
	Suppressed: org.apache.druid.query.QueryTimeoutException: Query[c5df16ae-12f3-45aa-8fe9-c18bed6c9ef6] url[<http://172.42.0.156:8100/druid/v2/>] timed out.
		at org.apache.druid.client.DirectDruidClient$1.checkQueryTimeout(DirectDruidClient.java:427) ~[druid-server-0.23.0.jar:0.23.0]
		at org.apache.druid.client.DirectDruidClient$1.access$500(DirectDruidClient.java:173) ~[druid-server-0.23.0.jar:0.23.0]
		at org.apache.druid.client.DirectDruidClient$1$2.hasMoreElements(DirectDruidClient.java:282) ~[druid-server-0.23.0.jar:0.23.0]
		at java.io.SequenceInputStream.nextStream(SequenceInputStream.java:109) ~[?:1.8.0_275]
		at java.io.SequenceInputStream.close(SequenceInputStream.java:232) ~[?:1.8.0_275]
		at com.fasterxml.jackson.dataformat.smile.SmileParser._closeInput(SmileParser.java:309) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
		at com.fasterxml.jackson.dataformat.smile.SmileParserBase.close(SmileParserBase.java:384) ~[jackson-dataformat-smile-2.10.5.jar:2.10.5]
		at org.apache.druid.client.JsonParserIterator.close(JsonParserIterator.java:138) ~[druid-server-0.23.0.jar:0.23.0]
		at org.apache.druid.utils.CloseableUtils.closeAndWrapExceptions(CloseableUtils.java:139) ~[druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.client.DirectDruidClient$3.cleanup(DirectDruidClient.java:515) ~[druid-server-0.23.0.jar:0.23.0]
		at org.apache.druid.client.DirectDruidClient$3.cleanup(DirectDruidClient.java:498) ~[druid-server-0.23.0.jar:0.23.0]
		at org.apache.druid.java.util.common.guava.BaseSequence$1.close(BaseSequence.java:136) ~[druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.close(ParallelMergeCombiningSequence.java:1072) ~[druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.java.util.common.io.Closer.close(Closer.java:182) ~[druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.utils.CloseableUtils.closeAndSuppressExceptions(CloseableUtils.java:162) [druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence.closeAllCursors(ParallelMergeCombiningSequence.java:1385) [druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence.access$400(ParallelMergeCombiningSequence.java:63) [druid-core-0.23.0.jar:0.23.0]
		at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$MergeCombineAction.compute(ParallelMergeCombiningSequence.java:666) [druid-core-0.23.0.jar:0.23.0]
		at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) [?:1.8.0_275]
		at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_275]
		at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_275]
		at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_275]
		at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_275]
    I'll also attached the configuration of the historicals and brokers. I'm running Druid 
    v0.23.0
    I think this is a backpressure issue but my knowdlege is really limited. What do you think is the root cause of this severe performance degradation? Let me know if you need any other logs. Thanks
    broker-settings.txthistorical-settings.txtbroker_queries_issues.txt
    v
    r
    • 3
    • 25
  • s
    This message was deleted.
    j
    b
    d
    • 4
    • 23
  • s
    This message was deleted.
    u
    a
    +2
    • 5
    • 43
  • s
    This message was deleted.
    b
    s
    +2
    • 5
    • 25
  • s
    This message was deleted.
    b
    • 2
    • 1
  • s
    This message was deleted.
    ❓ 1
    h
    b
    • 3
    • 16
  • s
    This message was deleted.
    b
    • 2
    • 1
  • s
    This message was deleted.
    s
    j
    +2
    • 5
    • 9
  • s
    This message was deleted.
    a
    • 2
    • 2
  • s
    This message was deleted.
    v
    e
    +2
    • 5
    • 10
  • s
    This message was deleted.
    s
    h
    +3
    • 6
    • 7
  • s
    This message was deleted.
    b
    s
    +2
    • 5
    • 22
  • s
    This message was deleted.
    ✅ 1
    w
    a
    +2
    • 5
    • 11
  • s
    This message was deleted.
    g
    • 2
    • 1
  • s
    This message was deleted.
    g
    s
    • 3
    • 8
  • s
    This message was deleted.
    v
    g
    +2
    • 5
    • 7
  • s
    This message was deleted.
    v
    a
    • 3
    • 3
  • s
    This message was deleted.
    d
    • 2
    • 1
  • p
    Stuck from many days upon this, really need help.
  • s
    This message was deleted.
    b
    • 2
    • 1
  • s
    This message was deleted.
    d
    g
    +3
    • 6
    • 48
  • s
    This message was deleted.
    b
    h
    +3
    • 6
    • 40
  • s
    This message was deleted.
    a
    s
    +4
    • 7
    • 37
  • s
    This message was deleted.
    v
    b
    • 3
    • 12
  • b
    I can see 92 gb utilised but only 3percent cpu used
  • b
    What wrong I am doing in conf
  • s
    This message was deleted.
    n
    s
    • 3
    • 3
1...262728...53Latest