@RunLLM we have problem with deleteting tenant. When deleting topics step by step, i get error, topic already offloaded. Reapeting doesn't help. How to avoid it?

The "topic already offloaded" or "topic is already fenced" error when deleting topics or tenants in Pulsar typically means the topic is in a transitional or locked state, often due to concurrent operations or incomplete cleanup. This is a known issue, especially when deleting resources rapidly or if the topic is being offloaded or closed at the same time. Retrying the deletion usually does not help, as the topic remains fenced or locked in the broker's state machine. This can also occur if the topic is not fully loaded when deletion is attempted, or if there are race conditions in the deletion process. See GitHub Issue #15451 and GitHub Issue #19083. To avoid this, ensure no active producers/consumers are connected, and consider restarting the broker to clear the fenced state. If using automation (like Pulsar Resources Operator), ensure your deletion logic tolerates "not found" or "already deleted" errors and retries after a delay. There is no guaranteed fix in current Pulsar releases, but improvements are being discussed in the community. Would you like more detail on workarounds or automation best practices? Sources: - GitHub Issue #15451 - GitHub Issue #19083 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how to avoid same problem with automation?

To avoid this problem with automation, ensure your deletion scripts or operators handle "already deleted" or "not found" errors gracefully and do not treat them as failures. Add retry logic with backoff, and check for topic existence before attempting deletion. The Pulsar Resources Operator recommends ignoring "not found" errors during deletion, as these can occur if the resource was deleted in a previous attempt or by another process. Additionally, ensure that your automation waits for all topics and subscriptions to be fully deleted before deleting the namespace or tenant, and consider adding delays between operations to allow the broker to clean up internal state. Would you like more detail on implementing these patterns? Sources: - PulsarTopic cannot be deleted: already deleted 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM some times only brokers restart will help avoid this problem. how to delete topic without restart?

To delete a topic without restarting the broker, ensure there are no active producers or consumers, and use the

force

option with the admin API (e.g.,

pulsar-admin topics delete --force

). If the topic is fenced or stuck, you may need to unload the topic first using

pulsar-admin topics unload

, which releases ownership and resets its state, then retry deletion. This approach can clear transient errors without a full broker restart, as described in Pulsar admin API docs and GitHub Issue #15451. If the topic remains undeletable, check for lingering subscriptions or replication, and remove them before retrying. Would you like more detail or example commands? Sources: - Pulsar admin API docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM error with request

13:12:53  DEBUG: Try request 1/5 to <http://k8s-pulsar-dev-broker.pulsar-dev:8080/admin/v2/namespaces/server_jenkins1/driver>
13:12:53  DEBUG: [1/5] Error: status 500, body: {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: Topic is already fenced\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException: Topic is already fenced

./bin/pulsar-admin topics list server_jenkins1/driver
Namespace is deleted

Reason: Namespace is deleted

What should i do for delete namespace?

This error occurs because the namespace is already marked as deleted, but some topics (likely fenced or in a transitional state) still exist in metadata, preventing full cleanup. This is a known Pulsar issue—deletion can leave behind "fenced" topics, and further delete attempts return "Namespace is deleted" or "Topic is already fenced" errors. See GitHub Issue #15451 and GitHub Issue #19083. To resolve, wait a few minutes and retry the namespace deletion. If the error persists, you may need to manually clean up orphaned topics or metadata using Pulsar admin tools or by directly editing metadata (not recommended for production). A broker restart can sometimes clear the fenced state, but is not always required. Would you like more detail on manual cleanup steps? Sources: - GitHub Issue #15451 - GitHub Issue #19083 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM error still exist. Waiting doesn't helped. Please give more details about manual steps?

If waiting and retries do not resolve the "Topic is already fenced" or "Namespace is deleted" errors, you may need to manually clean up Pulsar metadata. This is an advanced operation and should be done with caution, ideally in a test environment first. The main steps are: 1. Check for Orphaned Topics in Metadata: Use Pulsar admin tools or directly inspect the metadata store (ZooKeeper or etcd) for leftover topics or managed ledgers under the deleted namespace path (e.g.,

/managed-ledgers/<tenant>/<namespace>

). 2. Delete Orphaned Metadata: If you find entries for topics or ledgers under the deleted namespace, manually remove them using ZooKeeper CLI (

zkCli.sh

) or etcdctl, depending on your Pulsar metadata backend. Be careful—removing the wrong entries can cause data loss or instability. 3. Retry Namespace Deletion: After cleaning up, retry the namespace deletion with the admin tool. Warning: Direct metadata edits are risky and not officially recommended for production unless you have backups and understand the implications. Always back up your metadata before making changes. Would you like step-by-step commands for ZooKeeper or etcd? Sources: - https://github.com/apache/pulsar/pull/13337 - https://github.com/apache/pulsar/pull/19374 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.