We have a Pulsar cluster (version 2.7.2) with the Function Worker enabled in the broker (

functionsWorkerEnabled=true

). Functions are deployed using the Process Runtime in the broker/Function Worker (FW) under a single container. (Note: This is a legacy deployment. Our plan is to upgrade to pulsar 3.0 and deploy the Function Worker as a set of standalone containers with Process Runtime, with the final step of moving to a Kubernetes StatefulSet and deploying functions on Kubernetes.) The cluster had been running for an extended period without any downtime. However, after restarting the broker container, the Function Worker (along with all function endpoints) became unresponsive, with only about 10% of the expected functions restarting. Despite observing the Function Worker leadership reassigning after the broker container restart, it would not respond to any

pulsar-admin

commands. Has anyone encountered a similar issue or has any suggestions on how to debug this situation?

When you restarted the broker, did its IP address change?

The broker is a docker container running on a GCP VM with host networking , service ports mapped to host; so the IPs would not be changing (if i'm following you correctly) Thanks

curl <broker-ip>:8080/admin/v2/worker/cluster

gives what output?

[root@umf-prod1-pulsar-4 ~]# curl -s -X GET -H "Authorization: Bearer $(cat /etc/pulsar/auth/admin.token)" http://10.61.1.34:8080/admin/v2/worker/cluster | sed 's/****/MY_DOMAIN/g' | jq . [ { "workerId": "c-pulsar-dataeng-core-2-fw-pulsar-gcp-prod1-broker05.MY_DOMAIN.com-8080", "workerHostname": "pulsar-gcp-prod1-broker05.MY_DOMAIN.com", "port": 8080 }, { "workerId": "c-pulsar-dataeng-core-2-fw-pulsar-gcp-prod1-broker08.MY_DOMAIN.com-8080", "workerHostname": "pulsar-gcp-prod1-broker08.MY_DOMAIN.com", "port": 8080 }, { "workerId": "c-pulsar-dataeng-core-2-fw-pulsar-gcp-prod1-broker12.MY_DOMAIN.com-8080", "workerHostname": "pulsar-gcp-prod1-broker12.MY_DOMAIN.com", "port": 8080 }, { "workerId": "c-pulsar-dataeng-core-2-fw-pulsar-gcp-prod1-broker13.MY_DOMAIN.com-8080", "workerHostname": "pulsar-gcp-prod1-broker13.MY_DOMAIN.com", "port": 8080 }, { "workerId": "c-pulsar-dataeng-core-2-fw-pulsar-gcp-prod1-broker06.MY_DOMAIN.com-8080", "workerHostname": "pulsar-gcp-prod1-broker06.MY_DOMAIN.com", "port": 8080 }, { "workerId": "c-pulsar-dataeng-core-2-fw-pulsar-gcp-prod1-broker04.MY_DOMAIN.com-8080", "workerHostname": "pulsar-gcp-prod1-broker04.MY_DOMAIN.com", "port": 8080 } ]

workerHostname

is the

advertisedAddress

in

broker.conf

that might be the issue then.

i will investigate around that, thanks Aside, I do wonder could it be a zk issue, i see this znode with 3 childern ls /namespace/public/functions [0x4c000000_0x50000000, 0x58000000_0x5c000000, 0x68000000_0x6c000000] getting each of those get /namespace/public/functions/0x4c000000_0x50000000 {"nativeUrl":"pulsar://pulsar-gcp-prod1-broker12.MY_DOMAIN.com:6650","nativeUrlTls":"<pulsar+ssl://pulsar-gcp-prod1-broker12.MY>_DOMAIN.com:6651","httpUrl":"<http://pulsar-gcp-prod1-broker12.MY>_DOMAIN.com:8080","httpUrlTls":"<https://pulsar-gcp-prod1-broker12.MY>_DOMAIN.com:8443","disabled":false,"advertisedListeners":{}} get /namespace/public/functions/0x58000000_0x5c000000 {"nativeUrl":"pulsar://pulsar-gcp-prod1-broker13.MY_DOMAIN.com:6650","nativeUrlTls":"<pulsar+ssl://pulsar-gcp-prod1-broker13.MY>_DOMAIN.com:6651","httpUrl":"<http://pulsar-gcp-prod1-broker13.MY>_DOMAIN.com:8080","httpUrlTls":"<https://pulsar-gcp-prod1-broker13.MY>_DOMAIN.com:8443","disabled":false,"advertisedListeners":{}} get /namespace/public/functions/0x68000000_0x6c000000 {"nativeUrl":"pulsar://pulsar-gcp-prod1-broker05.MY_DOMAIN.com:6650","nativeUrlTls":"<pulsar+ssl://pulsar-gcp-prod1-broker05.MY>_DOMAIN.com:6651","httpUrl":"<http://pulsar-gcp-prod1-broker05.MY>_DOMAIN.com:8080","httpUrlTls":"<https://pulsar-gcp-prod1-broker05.MY>_DOMAIN.com:8443","disabled":false,"advertisedListeners":{}} these are 3 of the 6 broker node, i dont know if this is right or wrong - should the other 3 brokers be listed? also there is a znode for one particular function

ls /pulsar/functions/test/test/masked_func_name

we have 170 functions , why would only 1 have an entry in zk and not the others. Is this normal?

ZK should be aware of all 6 brokers.

I see connections from all brokers IPs to zookeeper, and no zk issues in the logs, but will keep checking - thanks for you input