Thomas MacKenzie
05/20/2025, 5:58 PMRunLLM
05/20/2025, 5:59 PMoxia:// scheme, replacing any ZooKeeper connection strings. For example, set metadataStoreUrl=oxia://<oxia-service-address>. The Pulsar codebase now supports Oxia as a metadata backend, and the configuration parameters have been updated to use generic metadata store names rather than ZooKeeper-specific ones. You should also ensure you are running a recent Pulsar version that includes Oxia support and that your Oxia service is deployed and reachable by the cluster components. For BookKeeper metadata, the same interface is used, so it can also point to Oxia if desired. See OxiaMetadataStoreProvider.java and PIP-335: Add support Oxia as a metadata store.
You may also need to update any deprecated configuration keys (e.g., replace zookeeperServers with metadataStoreUrl) and ensure your deployment scripts or Helm charts reference the correct parameters. For more details on the configuration and migration, refer to the StreamNative documentation and Pulsar metadata store tests. Would you like a step-by-step migration guide or example configuration files?
Sources:
- StreamNative documentation: Data Streaming Engine
- OxiaMetadataStoreProvider.java
- PIP-335: Add support Oxia as a metadata store
- Pulsar metadata store tests 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.Lari Hotari
05/20/2025, 10:16 PMThomas MacKenzie
05/20/2025, 10:18 PMThomas MacKenzie
05/20/2025, 10:20 PMpulsar-admin clusters create pulsar
otherwise the cluster does not accept tenant/namespace/topic creation with a Cluster can not be empty errorLari Hotari
05/21/2025, 8:47 AMLari Hotari
05/21/2025, 9:31 AMWe currently have a lot of issues when brokers are rolling out (topic ownership changing) and trying to see if the metadata store can helps us in that regard.What type of issues are you seeing? They could be related to the load spike that happens when ownership changes. The broker cache is currently poorly optimized for catch up reads that occur after ownership changes (some details in comments of https://github.com/apache/pulsar/issues/23466). There's also https://github.com/apache/pulsar/blob/master/pip/pip-307.md to improve the "handover" from one broker to another, however this isn't completed in the project and documented how this could be achieved. One detail that you should pay attention to is whether the broker has enough time to shutdown gracefully after Kubernetes In shutdown, most of the time would be spent in bundle unloading. You can check how much time you need for this by searching for this log line
"Unloading {} namespace-bundles completed in {} seconds" in your logs.
https://github.com/apache/pulsar/blob/46c2b74f4c9727d6399e9a1b48b031696d8c7cfd/pul[…]c/main/java/org/apache/pulsar/broker/service/BrokerService.java
If the broker doesn't completely shutdown and close open ledgers, additional load will occur on other brokers to bookkeepers when "dirty" ledgers need to be recovered and closed before topics can resume serving on the other broker. When ledgers are cleanly closed, this additional load wouldn't occur.
In Apache Pulsar Helm chart, it's recommended to increase the .Values.broker.gracePeriod value from default 30 seconds to a higher value so that brokers have time to shutdown cleanly.
An additional improvement to the Helm chart would be to issue a graceful shutdown command in a `preStop` lifecycle hook with pulsar-admin --admin-url=<http://localhost:8080> brokers shutdown
This feature was added in https://github.com/apache/pulsar/pull/14114, however there could be a challenge that some clients might not respond and the shutdown wouldn't get completed at all, resulting in "dirty" ledgers.Thomas MacKenzie
05/23/2025, 4:24 AMFailed to create producer at reconnect: connection closed
Failed to create producer at send PRODUCER request: connection closed
We are using the extensible load manager which we felt it gave us better results (remove the dependency on ZK for topic ownership), but still quite some disruption when brokers roll out. Here's some graph to show what we're currently facing.
We're at 600K TPM for 3 topics (1.8M TPM in the entire cluster), with each 12 partitions. There is 47 bundles in the namespace, spread across between 8 to 10 brokers on k8s, using pulsar 4.0.4. Deleting the applications (publishings apps) so that they can restart can help, but we still get some of these errors after a restart or at least a few retries (producer reconnect) before being able to publish so publish latencies are higher but not necessarily timing out. It's worth noting we're using the Go client v0.15.0 (not JAVA)
I'll look at the shutdown message, I guess this is a info or warn? (we only have errors currently). Regarding the grace period, there is 30 seconds (defaults from the helm chart).
I know topic ownership change / bundle unloading is quite sensitive and known to be an "issue" with Pulsar, but are timeouts (let's say 10s timeouts) a common issue with the technology? I'm trying to understand if what we're experiencing an expected behavior.Lari Hotari
05/23/2025, 7:33 AMI know topic ownership change / bundle unloading is quite sensitive and known to be an "issue" with Pulsar, but are timeouts (let's say 10s timeouts) a common issue with the technology? I'm trying to understand if what we're experiencing an expected behavior.What timeouts are you exactly referring to?
Thomas MacKenzie
05/23/2025, 4:04 PMLari Hotari
05/23/2025, 4:44 PMThomas MacKenzie
06/02/2025, 6:58 PM1m, see if that helps. I might DM you for a trial if it can be approved
I have a follow up question, do you think that 12 partitions for 600,000 msgs / min is enough? (I've been playing with the partition number for a while but never noticed any performance improvement above 10 when the cluster is steady - no broker restart)
I ask because I was reading this blog, and I see the cluster has 3 topics of 100+partitions: https://streamnative.io/blog/client-optimization-how-tencent-maintains-apache-pulsar-clusters-100-billion-messages-dailyLari Hotari
06/02/2025, 8:01 PMI have a follow up question, do you think thatIt could be sufficient, but feels low. The total "entry" (batch message) size (msgs & bytes) matters a lot and also how the messages are produced, since that's where the batching happens. Key-ordered processing requirement also impacts the decision. Also the way how consuming is handled. I haven't really thought about how to generalize the recommendations. Experimenting with your own configuration could be the best way to find out. Testing also with a rolling restart scenario is necessary to see how differently things behave with different number of partitions. There's some gotchas with producers and multiple partitions too. Producing could get blocked if any of the partitions is really slow or offline for too long. This is due to hitting the client side limits so that the client doesn't run out of memory. When there are more partitions, the partitions would spread out across the brokers and namespace bundles and a single load balancing event won't cause so much harm for producers and consumers. That's one reason to have more partitions that you'd think you'd need. You might need to slow down your rolling restart procedure to even be able to have the system online while performing maintenance operations like upgrades. (sidenote: I didn't notice until now that there has been a PIP for improving producer availability with "PIP-210: Retry producing on next partition if possible when a partition is not available", https://github.com/apache/pulsar/issues/18510 . It looks like that got stalled. There was recently a related discussion about something similar, but I couldn't find the discussion now.)partitions for12msgs / min is enough?600,000
Lari Hotari
06/02/2025, 8:12 PMkill -9 <pid> . The reason why this is relevant is that it will require a ledger recovery for all ledgers that were active in the broker. Ledger recovery will be a relatively expensive operation before topics can continue to serve produce requests. Keeping the maximum ledger size small will help with the tradeoff of creating more metadata when ledger rollovers happen more often.Thomas MacKenzie
06/02/2025, 11:25 PMLari Hotari
06/03/2025, 9:39 AMmanagedLedgerMinLedgerRolloverTimeMinutes) will take preference over max size parameters which is slightly misleading. That is documented in broker.conf, but it's easy to miss.
https://github.com/apache/pulsar/blob/a1a2b363cfaa1bbc38933a742484a70a0a56e761/conf/broker.conf#L1233-L1256Thomas MacKenzie
06/03/2025, 4:54 PM