https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
l
@Vinay Gupta Are you able to comment on the issue that you are also facing the issue? What type of Pulsar cluster did this happen in your case? (Which HW/Cloud/Virtualization, exact JVM version.) If you could share the first page of the JVM crash report, that would be useful too. (Don't share the full crash report since it might contain some of your confidential data in the environment variable dump) The issue comment https://github.com/apache/pulsar/issues/19307#issuecomment-1788094363 explains when the JDK contains the fix. If you really need the fix immediately, you should be able to run on 21.0.1, however that configuration isn't officially supported.
v
I had reported this before
Copy code
Sep 11, 2023 @ 15:24:38.386# Problematic frame:
Sep 11, 2023 @ 15:24:38.386# A fatal error has been detected by the Java Runtime Environment: - 
Sep 11, 2023 @ 15:24:38.386#  SIGSEGV (0xb) at pc=0x00007fabbb647632, pid=1, tid=601 - 
Sep 11, 2023 @ 15:24:38.386# JRE version: OpenJDK Runtime Environment (17.0.5+8) (build 17.0.5+8-Ubuntu-2ubuntu120.04) -  Sep 11, 2023 @ 15:24:38.386# Java VM: OpenJDK 64-Bit Server VM (17.0.5+8-Ubuntu-2ubuntu120.04, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) -  
Sep 11, 2023 @ 15:24:38.386# /pulsar/hs_err_pid1.log 
Sep 11, 2023 @ 15:24:38.386# -  
Sep 11, 2023 @ 15:24:38.386# Compiler replay data is saved as:
I did not commented on the issue JVM : 17.0.7 Pulsar: 2.11.0 Hosted on Kubernetes Cluster of 3 Zookeeper/bookie/Broker with proxy. AS reported under the fix we upgraded to 2.11.2 which contains 17.0.7 JDK.
l
Please comment on the issue since the details will get lost in Slack
v
sure will do
thankyou 1
l
Is it a frequent problem? Please also share how often it happens
Does the stacktrace match the one in the issue?
v
Yes whenever we put a slight high load these happens. (with 125+tps) Weirdly the issue happens on one broker and subsequently all the brokers goes down. Hence, the downtime.
l
What happen to the other brokers? Do they get overloaded?
v
No they goes down looking for the broker that went down. The issue that we found (for JVM) relates to this issue https://github.com/apache/pulsar/issues/20571. But, seems more or less identical.
Not sure is there a config issue or not but when this issue occurs on one broker all three broker goes down together and restarted.
l
They are seem to be different issues, the common thing is that the JVM crashes . That's why sharing your crash dump stack information would be useful
Do the other brokers have any error messages?
v
java exit with code 1/0 . host not found error
l
Do you have the exact log message to share?
v
Oct 6, 2023 @ 103609.253 2023-10-06T045619,691+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 0 - Oct 6, 2023 @ 103609.253 2023-10-06T045642,375+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 0 Oct 6, 2023 @ 115655.921 2023-10-06T045807,707+0000 [pulsar-proxy-io-2-1] ERROR org.apache.pulsar.proxy.server.ProxyConnection - [/xx1:33042] Error validating target broker 'xx-pulsar-211-broker-2.xx-pulsar-211-broker.pulsar.svc.cluster.local:6650'. authenticated with none role null. - Oct 6, 2023 @ 115655.920 2023-10-06T045807,106+0000 [pulsar-client-io-8-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/xx] [my-sub] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-2.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries
this is one.
and this org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' and search domain query for configured domains failed as well: [pulsar.svc.cluster.local, svc.cluster.local, cluster.local]{"previous":[ {"attempt":0,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' and search domain query for configured domains failed as well: [pulsar.svc.cluster.local, svc.cluster.local, cluster.local]"} ,{"attempt":1,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries "},{"attempt":2,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries "},{"attempt":3,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries "},{"attempt":4,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries "},{"attempt":5,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries "},{"attempt":6,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' after 4 queries "},{"attempt":7,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' and search domain query for configured domains failed as well: [pulsar.svc.cluster.local, svc.cluster.local, cluster.local]"},{"attempt":8,"error":"org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Failed to resolve 'xx-pulsar-211-broker-0.xx-pulsar-211-broker.pulsar.svc.cluster.local' and search domain query for configured domains failed as well: [pulsar.svc.cluster.local, svc.cluster.local, cluster.local]"}]}
l
Cluster of 3 Zookeeper/bookie/Broker with proxy.
how are these clustered? Which type of deployment is this exactly? Do you use an OSS Helm chart?
which pods did the logs come from that you shared?
v
broker 0 is the one that have jvm crash and broker 1 and 2 have other issue.
l
how can you have proxy logs on the broker ? or zookeeper originated logs?
v
Yes its through helm chart.
l
which helm chart out of the multiple options? version?
where did these log message appear:
Oct 6, 2023 @ 103609.253 2023-10-06T045619,691+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 0 -
Oct 6, 2023 @ 103609.253 2023-10-06T045642,375+0000 [main] ERROR org.apache.zookeeper.util.ServiceUtils - Exiting JVM with code 0
v
this is from broker
l
are you using an OSS Pulsar Helm chart? which one? version?
The Zookeeper server log messages on the broker don't yet make sense to me.
what would something like 
kubectl get pods -n pulsar
show for your cluster?
v
deployment repo access I do not have so asking the team
l
do you have a way to list the pods, something like 
kubectl get pods -n pulsar
?
v
open source pulsar helm chart with some our own customization. 211 Version.
👍 1
l
is it 2.11.0 ?
v
yes
l
as the first step, I'd recommend upgrading to 2.11.2 which was released in July
v
yes we did that fortunately that do not have JVM crash with even higher load.
l
so the problem is resolved?
does that issue reproduce on 2.11.2 ?
v
For JVM crash issue is resolved.
for other one : We are not able to get to the root cause why all three brokers goes down at once. For that we introduced proxy but that did not resolve the issue alltogether.
l
how about the 3 broker crash? does that reproduce on 2.11.2?
v
not yet.
l
for investigating the issue there would need to be some clarity in the logging so that each log line would be prefixed by the pod name
It's also necessary to reproduce with 2.11.2 (or newer) before investigating further
v
ok
2 Views
Open in Slack
PreviousNext
Powered by