<@U078Y59KC3B> what are the changes required in or...
# ask-ai
t
@RunLLM what are the changes required in order to use oxia instead of zookeeper?
r
It seems likely that to use Oxia instead of ZooKeeper as the metadata store in Pulsar or StreamNative, you need to update your configuration to set the metadata store URL to use the
oxia://
scheme, replacing any ZooKeeper connection strings. For example, set
metadataStoreUrl=oxia://<oxia-service-address>
. The Pulsar codebase now supports Oxia as a metadata backend, and the configuration parameters have been updated to use generic metadata store names rather than ZooKeeper-specific ones. You should also ensure you are running a recent Pulsar version that includes Oxia support and that your Oxia service is deployed and reachable by the cluster components. For BookKeeper metadata, the same interface is used, so it can also point to Oxia if desired. See OxiaMetadataStoreProvider.java and PIP-335: Add support Oxia as a metadata store. You may also need to update any deprecated configuration keys (e.g., replace
zookeeperServers
with
metadataStoreUrl
) and ensure your deployment scripts or Helm charts reference the correct parameters. For more details on the configuration and migration, refer to the StreamNative documentation and Pulsar metadata store tests. Would you like a step-by-step migration guide or example configuration files? Sources: - StreamNative documentation: Data Streaming Engine - OxiaMetadataStoreProvider.java - PIP-335: Add support Oxia as a metadata store - Pulsar metadata store tests 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
l
https://github.com/apache/pulsar-helm-chart contains support for Oxia, so that could be a way to see how things are configured and wired.
t
Thank you, that's what I'm currently using. We currently have a lot of issues when brokers are rolling out (topic ownership changing) and trying to see if the metadata store can helps us in that regard.
When initializing the cluster, I noticed I have to run
Copy code
pulsar-admin clusters create pulsar
otherwise the cluster does not accept tenant/namespace/topic creation with a
Cluster can not be empty
error
l
@Thomas MacKenzie In https://github.com/apache/pulsar-helm-chart, there's CI that validates Oxia with Pulsar and BookKeeper, so perhaps that doesn't catch that issue.
We currently have a lot of issues when brokers are rolling out (topic ownership changing) and trying to see if the metadata store can helps us in that regard.
What type of issues are you seeing? They could be related to the load spike that happens when ownership changes. The broker cache is currently poorly optimized for catch up reads that occur after ownership changes (some details in comments of https://github.com/apache/pulsar/issues/23466). There's also https://github.com/apache/pulsar/blob/master/pip/pip-307.md to improve the "handover" from one broker to another, however this isn't completed in the project and documented how this could be achieved. One detail that you should pay attention to is whether the broker has enough time to shutdown gracefully after Kubernetes In shutdown, most of the time would be spent in bundle unloading. You can check how much time you need for this by searching for this log line
"Unloading {} namespace-bundles completed in {} seconds"
in your logs. https://github.com/apache/pulsar/blob/46c2b74f4c9727d6399e9a1b48b031696d8c7cfd/pul[…]c/main/java/org/apache/pulsar/broker/service/BrokerService.java If the broker doesn't completely shutdown and close open ledgers, additional load will occur on other brokers to bookkeepers when "dirty" ledgers need to be recovered and closed before topics can resume serving on the other broker. When ledgers are cleanly closed, this additional load wouldn't occur. In Apache Pulsar Helm chart, it's recommended to increase the
.Values.broker.gracePeriod
value from default 30 seconds to a higher value so that brokers have time to shutdown cleanly. An additional improvement to the Helm chart would be to issue a graceful shutdown command in a `preStop` lifecycle hook with
pulsar-admin --admin-url=<http://localhost:8080> brokers shutdown
This feature was added in https://github.com/apache/pulsar/pull/14114, however there could be a challenge that some clients might not respond and the shutdown wouldn't get completed at all, resulting in "dirty" ledgers.
t
Sorry for the late response, I was investigating, trying to see if errors could be related to network issues with the service mesh but it does not look like it. We're getting the same errors now it's disabled. Errors are
Copy code
Failed to create producer at reconnect: connection closed
Failed to create producer at send PRODUCER request: connection closed
We are using the extensible load manager which we felt it gave us better results (remove the dependency on ZK for topic ownership), but still quite some disruption when brokers roll out. Here's some graph to show what we're currently facing. We're at 600K TPM for 3 topics (1.8M TPM in the entire cluster), with each 12 partitions. There is 47 bundles in the namespace, spread across between 8 to 10 brokers on k8s, using pulsar
4.0.4
. Deleting the applications (publishings apps) so that they can restart can help, but we still get some of these errors after a restart or at least a few retries (producer reconnect) before being able to publish so publish latencies are higher but not necessarily timing out. It's worth noting we're using the Go client v0.15.0 (not JAVA) I'll look at the shutdown message, I guess this is a info or warn? (we only have errors currently). Regarding the grace period, there is
30
seconds (defaults from the helm chart). I know topic ownership change / bundle unloading is quite sensitive and known to be an "issue" with Pulsar, but are timeouts (let's say 10s timeouts) a common issue with the technology? I'm trying to understand if what we're experiencing an expected behavior.
l
I know topic ownership change / bundle unloading is quite sensitive and known to be an "issue" with Pulsar, but are timeouts (let's say 10s timeouts) a common issue with the technology? I'm trying to understand if what we're experiencing an expected behavior.
What timeouts are you exactly referring to?
t
They are publish timeouts, so when the clients send the messages with the producer.
l
publish timeouts would definitely occur if the ledgers cannot be closed cleanly due to the ledger recovery process that I mentioned in one of the comments. 30 seconds might not be sufficient for a clean shutdown. I'd first check the logs to see whether brokers shutdown cleanly in a rolling restart. One of the problems of rolling restart is also that bundles could get assigned to a broker that will be soon shutdown. That causes extra shuffle of topics (namespace bundles). There's room for many improvements in this area. In the StreamNative Cloud offering, there's the "Graceful Cluster Rollout" feature to deal with one of the main root causes. This feature is explained in https://docs.streamnative.io/private/private-cloud-graceful-cluster-rollout . Essentially a new pod in a separate statefulset will be started to replace a pod one-by-one and all bundles from the old pod will be assigned to this new pod. This way the extra "shuffling" is reduced to minimum while doing rolling restarts since a bundle will move once instead of getting assigned multiple times when all broker pods get restarted. This feature builds upon PIP-307 and ExtensibleLoadManager. The SN operator is no longer open source, so this feature is only available to paying customers. Please DM me if you'd be interested in a trial.
t
Thank you, I'm going to look at logs and also bump the gracePeriod to
1m
, see if that helps. I might DM you for a trial if it can be approved I have a follow up question, do you think that
12
partitions for
600,000
msgs / min is enough? (I've been playing with the partition number for a while but never noticed any performance improvement above
10
when the cluster is steady - no broker restart) I ask because I was reading this blog, and I see the cluster has 3 topics of 100+partitions: https://streamnative.io/blog/client-optimization-how-tencent-maintains-apache-pulsar-clusters-100-billion-messages-daily
l
I have a follow up question, do you think that
12
partitions for
600,000
msgs / min is enough?
It could be sufficient, but feels low. The total "entry" (batch message) size (msgs & bytes) matters a lot and also how the messages are produced, since that's where the batching happens. Key-ordered processing requirement also impacts the decision. Also the way how consuming is handled. I haven't really thought about how to generalize the recommendations. Experimenting with your own configuration could be the best way to find out. Testing also with a rolling restart scenario is necessary to see how differently things behave with different number of partitions. There's some gotchas with producers and multiple partitions too. Producing could get blocked if any of the partitions is really slow or offline for too long. This is due to hitting the client side limits so that the client doesn't run out of memory. When there are more partitions, the partitions would spread out across the brokers and namespace bundles and a single load balancing event won't cause so much harm for producers and consumers. That's one reason to have more partitions that you'd think you'd need. You might need to slow down your rolling restart procedure to even be able to have the system online while performing maintenance operations like upgrades. (sidenote: I didn't notice until now that there has been a PIP for improving producer availability with "PIP-210: Retry producing on next partition if possible when a partition is not available", https://github.com/apache/pulsar/issues/18510 . It looks like that got stalled. There was recently a related discussion about something similar, but I couldn't find the discussion now.)
not directly related, but since I mentioned "Experimenting with your own configuration could be the best way to find out" and "Testing also with a rolling restart scenario", adding one more test case: testing a broker crash. it can be simulated with
kill -9 <pid>
. The reason why this is relevant is that it will require a ledger recovery for all ledgers that were active in the broker. Ledger recovery will be a relatively expensive operation before topics can continue to serve produce requests. Keeping the maximum ledger size small will help with the tradeoff of creating more metadata when ledger rollovers happen more often.
t
Ok thanks for the details. I remember looking at the ledger size and had concerns of too small ledgers and ledgers created too often basically (I remember a max size and max time as well if I'm not mistaken). Will look at these values again and add this test scenario to my list 💯
l
there's a gotcha with max size parameters, that min time (
managedLedgerMinLedgerRolloverTimeMinutes
) will take preference over max size parameters which is slightly misleading. That is documented in broker.conf, but it's easy to miss. https://github.com/apache/pulsar/blob/a1a2b363cfaa1bbc38933a742484a70a0a56e761/conf/broker.conf#L1233-L1256
t
Ok that's good to know, I actually thought they were both being applied as thresholds. Much appreciated thanks. I'll probably run more tests next week in our performance env and circle back here.