https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
d
Copy code
druid-broker-27 druid-broker 2022-10-19T05:17:55,959 ERROR [DruidSchema-Cache-0] org.apache.druid.query.ResultLevelCachingQueryRunner - Error while preparing for result level caching for query e5b21375-030e-48ec-9d2a-0ca41f75215d with error Faulty channel in resource pool 
druid-broker-27 druid-broker org.apache.druid.query.QueryInterruptedException: Faulty channel in resource pool
druid-broker-27 druid-broker      at org.apache.druid.client.JsonParserIterator.convertException(JsonParserIterator.java:273) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.JsonParserIterator.init(JsonParserIterator.java:191) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.JsonParserIterator.hasNext(JsonParserIterator.java:93) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:89) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:69) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:879) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:929) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.compensatedBlock(ForkJoinPool.java:3450) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3433) ~[?:?]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:918) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:1025) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:732) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
druid-broker-27 druid-broker      at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]
druid-broker-27 druid-broker Caused by: org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
druid-broker-27 druid-broker      at org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:132) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.java.util.http.client.CredentialedHttpClient.go(CredentialedHttpClient.java:48) ~[druid-core-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:453) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.getSimpleServerResults(CachingClusteredClient.java:706) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.lambda$addSequencesFromServer$8(CachingClusteredClient.java:668) ~[druid-server-0.23.0.jar:0.23.0]
druid-broker-27 druid-broker      at java.util.TreeMap.forEach(TreeMap.java:1282) ~[?:?]
Copy code
druid-broker-27 druid-broker Caused by: java.nio.channels.UnresolvedAddressException
druid-broker-27 druid-broker      at sun.nio.ch.Net.checkAddress(Net.java:149) ~[?:?]
druid-broker-27 druid-broker      at sun.nio.ch.Net.checkAddress(Net.java:157) ~[?:?]
druid-broker-27 druid-broker      at sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816) ~[?:?]
druid-broker-27 druid-broker      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839) ~[?:?]
a
looks like a network/infra problem?
d
Yeah, I think Kube’s CoreDNS was caching old ip addresses. After waiting a bit and performing rolling restart on broker and router, the problem went away.
gave me a tiny heart attack, haha. “We performed so many upgrades and restarts already, how come there’s still an unaccounted fail-case?”
a
🙂
t
We hit exact same issue on multiple clusters after upgrading
Open in Slack
PreviousNext
Powered by