This message was deleted.

Please report this in https://github.com/apache/pulsar/issues (search first if there's already a match). It might be related to the advertised listeners/address as you suggest.

it is intermittent and very much random.... we noticed that sometimes frequency is 20 times in a min and sometimes 10 times in an hour..

@Manmeet this is most likely a bug. Please report an issue to https://github.com/apache/pulsar/issues.

sure

I'll work on confirming a possible issue and a fix.

Thank you for the update, issue report and the PR. Your speed is unmatched 🙂. While we are also testing this behavior with the workaround -> but we use the isolation group feature where our ns isolation group policy regex is based on broker hostname. just want to validate that there is no impact on that?

@Lari Hotari Thank you for your quick response. Reported the issue -https://github.com/apache/pulsar/issues/21897 so that fix PR can be attached to it. Furthermore, could you please assist with the query posed by Girish?

but we use the isolation group feature where our ns isolation group policy regex is based on broker hostname. just want to validate that there is no impact on that?

@Girish Sharma Based on looking at the code, it doesn't look like isolation group feature is impacted.

So, we tried a bunch of things and it is messy.. I haven't looked into what all the PR is changing in the master, but from the way the code works, i doubt this situation is improved. consider this case:

advertisedAddress: <pod-name>.<service-name>
advertisedListeners: XYZ:pulsar://<ip>:6650, XYZ:http://<ip>:80, internal:http://<pod-name>.<service-name>:80
internalListenerName=internal

> I haven't looked into what all the PR is changing in the master, but from the way the code works, @Girish Sharma The bug itself is fixed. The PR contained a test case the reproduced the issue. The test case is https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apach[…]adbalance/AdvertisedListenersMultiBrokerLeaderElectionTest.java > as soon as we use an

http

protocol internal listener in the advertised listeners, that one gets being used as the "web url" everywhere in pulsar, and not just for internal load balancing.. i.e., in a lookup call when it has to return back with redirect, it would use that

http://<pod-name>.<service-name>:80

as the prefix in the redirect url. > > This basically makes the

XYZ

lisntner's http protocol IP based address useless.. > This leads to failing lookup calls as clients are not able to resolve the redirect url as those k8s based addresses only resolve inside a cluster. It's a valid observation. There is definitely room for improvements as you are pointing out. > the fix is that the clients (where possible) add a client configuration of

listenerName: XYZ

during client creation.. but that might not be a feasible thing in every tech stack/connector.. yes, this is a good point. > by any chance, your PR fixes this situation? The PR doesn't improve the advertised listener feature. It fixes the load manager's leader broker issue #21897. > Why is an "internal" address being given out to clients.. I'm not sure why it has exactly been implemented this way. It is PIP-61 https://github.com/apache/pulsar/wiki/PIP-61%3A-Advertised-multiple-addresses . Checking the document , PRs and mailing list discussions and reference documentation might reveal more of the context. PIP-61 does mention "only return the corresponding service URL". Perhaps the implementation contains some TODOs. 🙂

Checking the document , PRs and mailing list discussions and reference

documentation might reveal more of the context. PIP-61 does mention

"only return the corresponding service URL". Perhaps the implementation

contains some TODOs. 🙂

I was thinking about it. I can't see a "fix" in the current approach. But I definitely think it shouldn't be the default behavior to send in the internal address back in the lookup call in case the client is missing the listener name.. but the only way to do that now is to introduce a broker property called

lookupListener

which defines the name of the listener to be used for lookup if listenerName is missing from lookup call.

> but the only way to do that now is to introduce a broker property called

lookupListener

which defines the name of the listener to be used for lookup if listenerName is missing from lookup call. +1. It also comes to mind that there could be a need to support multiple "listener" instances that bind to different server socket port ports so that a listener on a specific port could have a specific default listener. Obviously this would be a lot of additional complexity. I personally fear the way how things are configured. The configuration framework in Pulsar isn't great for hierarchical information which would be better off in yaml files or such. (sidenote: Migrating to https://github.com/gestalt-config/gestalt is something that I have been dreaming about. 🙂 Gestalt Config is very similar to Spring Boot's configuration solution without the coupling to Spring Framework and Spring Boot.)

multiple "listener" instances that bind to different server socket port ports so that a listener on a specific port could have a specific default listener.

A per port thing might not cater to all use cases as there is an internal vs external address notation on the same port ..

A per port thing might not cater to all use cases as there is an internal vs external address notation on the same port ..

The reason why I mentioned it is that I think that the per port solution would be necessary for implementing what PIP-61 mentions about "only return the corresponding service URL". It's not exclusive for the the default

lookupListener

solution what you proposed. Without a separate listener instance on a different port, it would be hard to distinguish between internal and external traffic. This is just my gut feeling.

I was checking the touch points for LookupOptions builder and it does seem like it is used at external places only..

perhaps. To me it seems that cluster internal access isn't split from others.

Could you elaborate a bit more?

@Girish Sharma Within the Pulsar cluster, the Pulsar admin REST API is used for many operations broker-to-broker. The redirects are also used in that case. The Pulsar cluster does also use Pulsar binary protocol for broker-to-broker connections for system topics which are for example used for topic policies. Adding the internal listener selection code could be added to that, but somehow the connections would have to indicate the selection of the internal listener. It would be possible to modify all broker-to-broker connections to pass the internal listener name, but I guess that implies that the broker returns it to the client. I haven't checked the details, but this is what I had in mind.

the admin APIs also use the 307 approach, i.e. the redirect url goes back to the client. so returning an external name is a good thing there as well. I will have to check the BinaryProtoLookupService.java class as well to see if changes are needed there as well.

Good observations @Girish Sharma.

now if we make this consistent, there may be backward incompatible result for users who are used to this inconsistency.. any thoughts here?

It might require creating a PIP to cover the changes although it seems that it would be a bug fix. The benefit of a PIP is that it also serves as a design document for the changes. Would you like to start a drafting a PIP for this where the problem is described and then proposing possible solutions?

we are in process of the pip creation. almost finalized.

PIP regarding the same - https://github.com/apache/pulsar/pull/22039

@Manmeet Yes, I'll take a look.

IMO, even this change is also not right, at one point it talks about internal vs external IP, but for redirects, sends back the internal non resolvable ip back to clients

yes I agree. However it seems to include some useful changes.