Hi everyone We are facing a weird issue upgrading to 2 10 6 pulsar #general

Hi everyone, We are facing a weird issue. upgradi...

Girish Sharma

03/21/2024, 1:51 PM

Hi everyone, We are facing a weird issue. upgrading to 2.10.6, we see that there was a bugfix for setting the

enforceMinNumRacksPerWriteQuorum

flag correctly on the bookie client. But now its putting all our available bookies into exclude bookies even though there are enough bookies per rack.. The issue which is coming is that

RackawareEnsemblePlacementPolicyImpl::addDefaultRackBookiesIfMinNumRacksIsEnforced

seems to be adding all the bookies.. it might be due to the call

topology.getLeaves(getDefaultRack())

inside that method (Referencing bookie client 4.14.8) because that's the only way it will run the following for loop and then reach the

Copy code

<http://LOG.info|LOG.info>("enforceMinNumRacksPerWriteQuorum is enabled, so Excluding bookies of defaultRack: {}",
        bookiesInDefaultRack);

line which we see in logs. We have not set any bookie's rack as the default rack -

/default-rack

... So not sure why

topology.getLeaves(getDefaultRack())

is returning non empty list.. Any pointers here?

Girish Sharma

03/21/2024, 1:52 PM

For reference - https://github.com/apache/bookkeeper/blob/branch-4.14/bookkeeper-server/src/main/j[…]che/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Girish Sharma

03/21/2024, 2:09 PM

For debugging purposes, is there a way to print the topology that bookie client has parsed and stored?

Girish Sharma

03/21/2024, 2:45 PM

basically, it looks like we are getting this `

Copy code

LOG.error("Only one rack available and minNumRacksPerWriteQuorum is enforced, so giving up");

.. so the topology thinks that we have only one rack.. .. while when we do

pulsar-admin bookies racks-placement

gives correct result with 3 groups and multiple bookies under each group with more than 3 unique racks overall per group

Girish Sharma

03/21/2024, 2:46 PM

sharing snippet of that output..

Copy code

{
    "prod:_shared_": {
        "prod-pulsar-bookie-13.prod-pulsar-bookie.prod.svc.cluster.local:3181": {
            "rack": "3"
        },
        "prod-pulsar-bookie-12.prod-pulsar-bookie.prod.svc.cluster.local:3181": {
            "rack": "2"
        },
        "prod-pulsar-bookie-10.prod-pulsar-bookie.prod.svc.cluster.local:3181": {
            "rack": "3"
        },
        "prod-pulsar-bookie-9.prod-pulsar-bookie.prod.svc.cluster.local:3181": {
            "rack": "3"
        },
        "prod-pulsar-bookie-8.prod-pulsar-bookie.prod.svc.cluster.local:3181": {
            "rack": "1"
        },
...

Lari Hotari

03/21/2024, 3:44 PM

Rings some bells of a bug that has been fixed in the past. However, I'm not sure about that.

Lari Hotari

03/21/2024, 3:46 PM

first search hit is https://github.com/apache/pulsar/pull/16825 , but I haven't checked

Lari Hotari

03/21/2024, 3:46 PM

well that is included in 2.10.2 so not that

Lari Hotari

03/21/2024, 3:48 PM

found https://github.com/apache/pulsar/pull/20944 which isn't yet in a 2.10.x release

Lari Hotari

03/21/2024, 3:48 PM

@Girish Sharma please check if PR 20944 would fix your issue

Lari Hotari

03/21/2024, 3:49 PM

well the description doesn't look exactly something for your case

Lari Hotari

03/21/2024, 3:54 PM

> Try to set

bookkeeperClientRegionawarePolicyEnabled=true

in the broker.conf in https://github.com/apache/pulsar/issues/20918 . something to try well other comment is "no need for this"

Girish Sharma

03/21/2024, 4:36 PM

Thanks for the reply Lari, I will check 20944 but the 20918 doesn't seem like related. We don't have regions

Girish Sharma

03/21/2024, 4:55 PM

@Lari Hotari looking at 20944 , i am unclear of the trigger for the bug that the issue creator is mentioning.. ..

Girish Sharma

03/21/2024, 5:14 PM

when i restart a broker - it spews the right logs - adding new node "/2/<bookie name>" .. and like wise..

Girish Sharma

03/21/2024, 5:15 PM

unsure what moving the

Copy code

updateRacksWithHost(racksWithHost);
            watchAvailableBookies();

after the for loop is doing there..

Open in Slack

Previous Next