https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
j
is this a bug?
y
j
Sorry, but I am confused. Not so sure what I need to do to fix the problem.
y
in each component, there is a securityCountext section. add securityContext: runAsUser: 0 runAsGroup: 0
✅ 1
j
thank you again
y
This happens when you upgrade from 2.9 to 2.10. in 2.9, the overlay fs is owned by root. in 2.10, the image is nonroot. This is a conflict. If you have ephemeral container, you can use kubectl debug to fix this, instead of runasroot.
j
hmmmm that didn't work. so what i did is i put proxy.securityContext as you mentioned above.
Copy code
Warning  Unhealthy  110s (x8 over 3m)  kubelet            Liveness probe failed: Get "<http://10.44.1.92:80/status.html>": dial tcp 10.44.1.92:80: connect: connection refused
  Warning  Unhealthy  60s (x13 over 3m)  kubelet            Readiness probe failed: Get "<http://10.44.1.92:80/status.html>": dial tcp 10.44.1.92:80: connect: connection refused
10.44.1.92 is the IP of the proxy pod itself. By the way, I am not upgrading Pulsar. This is a brand new deployment of Pulsar using the HELM chart.
Copy code
[conf/proxy.conf] Applying config authenticationEnabled = true
[conf/proxy.conf] Applying config authenticationProviders = org.apache.pulsar.broker.authentication.AuthenticationProviderToken
[conf/proxy.conf] Applying config authorizationEnabled = false
[conf/proxy.conf] Applying config brokerClientAuthenticationParameters = file:///pulsar/tokens/proxy/token
[conf/proxy.conf] Applying config brokerClientAuthenticationPlugin = org.apache.pulsar.client.impl.auth.AuthenticationToken
[conf/proxy.conf] Applying config brokerServiceURL = <pulsar://pulsar-broker:6650>
[conf/proxy.conf] Applying config brokerWebServiceURL = <http://pulsar-broker:8080>
[conf/proxy.conf] Applying config clusterName = pulsar
[conf/proxy.conf] Applying config forwardAuthorizationCredentials = true
[conf/proxy.conf] Applying config httpNumThreads = 8
[conf/proxy.conf] Applying config servicePort = 6650
[conf/proxy.conf] Applying config statusFilePath = /pulsar/status
[conf/proxy.conf] Applying config superUserRoles = admin,broker-admin,proxy-admin
[conf/proxy.conf] Applying config tokenPublicKey = file:///pulsar/keys/token/public.key
[conf/proxy.conf] Applying config webServicePort = 80
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/pulsar/lib/io.netty-netty-common-4.1.77.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2022-11-13T16:33:45,176+0000 [main] INFO  org.apache.pulsar.broker.authentication.AuthenticationService - [org.apache.pulsar.broker.authentication.AuthenticationProviderToken] has been loaded.
2022-11-13T16:33:45,470+0000 [main] INFO  org.apache.pulsar.proxy.extensions.ProxyExtensionsUtils - Searching for extensions in /pulsar/./proxyextensions
2022-11-13T16:33:45,472+0000 [main] WARN  org.apache.pulsar.proxy.extensions.ProxyExtensionsUtils - extension directory not found
2022-11-13T16:33:45,521+0000 [main] INFO  org.eclipse.jetty.util.log - Logging initialized @3734ms to org.eclipse.jetty.util.log.Slf4jLog
2022-11-13T16:33:45,660+0000 [main] INFO  org.apache.pulsar.proxy.server.ProxyService - Started Pulsar Proxy at /0.0.0.0:6650
2022-11-13T16:33:45,943+0000 [main] INFO  org.eclipse.jetty.server.Server - jetty-9.4.48.v20220622; built: 2022-06-21T20:42:25.880Z; git: 6b67c5719d1f4371b33655ff2d047d24e171e49a; jvm 11.0.16+8-post-Ubuntu-0ubuntu120.04
2022-11-13T16:33:45,981+0000 [main] INFO  org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
2022-11-13T16:33:45,981+0000 [main] INFO  org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
2022-11-13T16:33:45,984+0000 [main] INFO  org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
2022-11-13T16:33:46,001+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@4349754{/metrics,null,AVAILABLE}
2022-11-13T16:33:46,432+0000 [main] WARN  org.glassfish.jersey.server.wadl.WadlFeature - JAXBContext implementation could not be found. WADL feature is disabled.
2022-11-13T16:33:46,731+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@b967222{/,null,AVAILABLE}
2022-11-13T16:33:46,801+0000 [main] WARN  org.glassfish.jersey.server.wadl.WadlFeature - JAXBContext implementation could not be found. WADL feature is disabled.
2022-11-13T16:33:46,886+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@6a2eea2a{/proxy-stats,null,AVAILABLE}
2022-11-13T16:33:46,920+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@2bf94401{/admin,null,AVAILABLE}
2022-11-13T16:33:46,921+0000 [main] INFO  org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@2532b351{/lookup,null,AVAILABLE}
2022-11-13T16:33:46,930Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5ffd2b27] error Uncaught exception in thread main: Failed to start HTTP server on ports [80]
java.io.IOException: Failed to start HTTP server on ports [80]
        at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:243)
        at org.apache.pulsar.proxy.server.ProxyServiceStarter.start(ProxyServiceStarter.java:223)
        at org.apache.pulsar.proxy.server.ProxyServiceStarter.main(ProxyServiceStarter.java:185)
Caused by: java.io.IOException: Failed to bind to /0.0.0.0:80
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
        at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
        at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
        at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.eclipse.jetty.server.Server.doStart(Server.java:401)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at org.apache.pulsar.proxy.server.WebServer.start(WebServer.java:221)
        ... 2 more
Caused by: java.net.SocketException: Permission denied
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:459)
        at java.base/sun.nio.ch.Net.bind(Net.java:448)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
        at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
        ... 9 more
m
@Je Sum Yip pointed out that the proxy does not expose the configuration for making a service listen on port 80. I'd recommend just making the proxy listen on 8080 and then having the proxy's service map port 80 to port 8080
Note that I do not recommend running the pod as the root user. It is unnecessary.
j
i sorta ended up doing that. the proxy is now configured to listen on 8080, i also have istio doing tls termination on 443 which then routes the traffic to proxy:8080. works fine. 🙂
thanks @Michael Marshall
m
Great, glad you got it working
m
It happens for me as well But I don’t understand why does it happen, since I have like 4-5 Pulsar clusters spread on several EKS clusters and from some reason I suddenly got this error, and the configuration is totally identical on all clusters (pulsar and EKS) Also, what do you mean by “having the proxy’s service map port 80 to port 8080” when I change proxy.ports to 8080 the service changes as well to 8080
Open in Slack
PreviousNext
Powered by