pulsar

Hi, if I understand correctly, I think we want to make a distributed/consistent metadata store on top of pulsar.

For this metadata store, one of the solutions could be a partitioned persistent topic table-view, using the bundle state channel proposed in pip-192.

So, with this following distributed metadata store design, I think the bundle state channel from pip-192 could be a good cornerstone for the new metadata store as well.

Idea:
For put(key,value) operation, each broker publishes a put(key, value) message to the metadata topic. Also, the owner broker consumes the partition topic itself via a TableView Client, to consistently and persistently serve Get(key) requests for the metadata partition.

For get(key) operation, each broker makes an RPC call to the metadata topic partition owner broker. To identify the owner broker, it looks up the bundle state channel(introduced in pip-192).


Consistency:
• There are no replicas for metadata topic partitions(only the owner broker itself subscribes to the partition topic via the table view).
• This means the single broker, the topic partition owner serves all put/get requests for that metadata partition -- strong consistency.
• we could make some(all) other brokers subscribe to some(all) partitions as well as replicas -- eventual(sequential) consistency. If all brokers subscribe to all partitions, this will be the full(n)-replication. Then, we don't need RPC calls for the get(key) operations(local table views lookup will be enough).
Scalability:
• metadata store is partitioned to multiple brokers.
Latency:
• each get(key) request requires an RPC call(equivalent to ZK node lookup without caching)
Fault tolerance
• If any owner broker crashes, the BSC identifies and releases all bundle ownerships in BSC. The new owner broker will be selected and soon build the metadata partition table view from the topic(with compaction).
FYI, I haven't fully discussed the above with the SN team. This is purely from my head today. I have attached a rough design diagram here.

Urgent issue with Pulsar and lack of cgroup v2 support: <https://lists.apache.org/thread/p5kk81r70rsdwb29w1rhfg1gp117v5gl>
&gt; Hi all,
&gt; 
&gt; Pulsar doesn't support cgroup v2 which becomes default in Kubernetes v1.25+.
&gt; Kubernetes announcement:
&gt; <https://kubernetes.io/blog/2022/08/31/cgroupv2-ga-1-25>/ .
&gt; Pulsar issue: <https://github.com/apache/pulsar/issues/16601>
&gt; 
&gt; The impact of this is that the Pulsar load balancer won't have correct
&gt; CPU and memory information for making load balancing decisions.
&gt; 
&gt; The cloud provider managed Kubernetes services have already switched
&gt; to cgroup v2 as the default. This happened in AKS v1.25, GKE v1.26 and
&gt; in EKS v1.26.
&gt; For GKE, it's possible to keep using cgroup v1 also in GKE v1.26
&gt; (<https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options>).
&gt; For AKS and EKS, it's unknown whether such a configuration option
&gt; exists.
&gt; 
&gt; There's a previous attempt in this PR to add cgroup v2 support to
&gt; Pulsar: <https://github.com/apache/pulsar/pull/16832> . Would it be
&gt; possible to continue the work for supporting cgroup v2 in Pulsar
&gt; either with the existing PR or a new one?
&gt; 
&gt; Who would like to pick up this work?
&gt; This is urgent since cgroup v2 is enabled by default for all latest
&gt; managed Kubernetes services (AKS v1.25, GKE v1.26 and EKS v1.26).
&gt; 
&gt; Regards,
&gt; 
&gt; -Lari

It looks like it is a bug, we are not registerNamespace to `ExtensibleLoadManagerImpl`…

Oh interesting. Can you check if the tableview in the ServiceUnitStateChannelImpl has that bundle?

Thanks for the bug report and quick fix, Kai!

Thanks for reporting this issue! I’ll handle this:eyes:

logs in <https://gist.github.com/lhotari/cfa59d8dacddbebb7c860cefa0c04b4a>

This test is very flaky <https://github.com/apache/pulsar/issues/20157|https://github.com/apache/pulsar/issues/20157> . <@U03BN2F8K8R> <@U02F5AYBL0M>. Any chance to take a look?

It looks like it is the same issue as <https://github.com/apache/pulsar/issues/20618>, I will take a look.

Thank you <@U02F5AYBL0M>, that will be very helpful