https://pulsar.apache.org/ logo
Join Slack
Powered by
anyone can help me on this ? after decommissioned ...
# general
w
anyone can help me on this ? after decommissioned one set of bookies, healthcheck failed, all the other working fine . how can I fix this internal topic so it can success on healthcheck ?
Copy code
bin/pulsar-admin topics stats-internal <persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck>
No such ledger exists on Metadata Server

Reason: No such ledger exists on Metadata Server
y
It doesn’t look like a topic name
w
It is an internal topic used for broker health check. It looks like still checking old bookies but i did update the helm values so it will use new bookie cluster .
y
It is not a legit topic name. Topic name should be tenant/namespace/topic
w
Correct. it is a special internal topic. error looks like this
it seems it caches old ledger, i want to reset but i cannot
k
@David K can you help with this?
d
@Wallace Peng It might be caused by the issue outlined in this PIP. FWIW, the PIP is supposed to prevent this from happening and documents the scenario fairly well.
w
@David K thanks for the info. i will double read again . so there is pending deleted ledger cannot be removed then it blocks for ever, any way i can go to zk to delete so it can get out of the loop ?
d
When are you seeing this error? During topic compaction?
w
i had two bookkeeper cluster, bookkeeper1 and bookkeeper2 . each of the cluster has 3 bookie nodes now i decommissioned all 3 nodes from bookkeeper1. then i restarted broker, it ran into this loop .
d
This is the ledger deletion flow, and from your error, it looks like the ledger was deleted in step 2.
Maybe you can remove the ledger id from the managed ledger, since it isn’t in the metadata server either. All that would be left would be the ledger entry on disk.
w
right, it is not in meta data server, also not in bookkeeper (i did ran bookkeeper listledger and ledgermetadata commands to confirm) . how do I remove from managed ledger , is this in zk's folder managed-ledgers ?
d
What about the ledger command?
“how do I remove from managed ledger , is this in zk’s folder managed-ledgers”, Yes. Also try unloading the topic, to force the managed ledger to be rebuilt vs. read from cache
w
ledger not exist if use deleteledger
Copy code
AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=3596046.  Returning success.
Copy code
[healthcheck]
[zk: localhost:2181(CONNECTED) 7] ls /managed-ledgers/pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/persistent/healthcheck
[]
the folder is empty . this is a special topic, i tried to unload but it does not seem help as well.
Copy code
[<persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck>] [null] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: {"errorMsg":"org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=3596046 - operation=Failed to open ledger caused by org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=3596046 - operation=Failed to open ledger","reqId":3411430584494320639, "remote":"pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/172.20.197.45:6650", "local":"/172.20.197.45:45554"} -- Will try again in 4.715 s
ERROR org.apache.pulsar.client.impl.ProducerImpl - [<persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck>] [null] Failed to create producer: {"errorMsg":"org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=3596046 - operation=Failed to open ledger caused by org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=3596046 - operation=Failed to open ledger","reqId":3411430584494320639, "remote":"pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local/172.20.197.45:6650", "local":"/172.20.197.45:45554"}
i can use 
topics stats
but i cannot use 
topics stats-internal
i also tried to delete the topic but it does not work
d
Can you tell if the ledgerId=3596046 is consistent with the new BK cluster or the old one?
w
i guess this ledger is from old bk cluster that i decommissioned. somehow broker did not switch to use new bookie cluster for health check topic
maybe there was a timing like restarted broker too soon ...
👍 1
d
So the managed ledger has a reference to a non-existent ledger, which is causing the issue. Just delete it, and you should be fine. I know it is easier said than done, but….
w
ok ic. as i dont see it in the folder, can u point me the direction how i can delete it 🙂 i can try programming if needed.
d
What is in 
ls /managed-ledgers/pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/persistent
?
w
Copy code
ls /managed-ledgers/pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/persistent
[healthcheck]
it just has 
healthcheck
Copy code
bin/pulsar-managed-ledger-admin delete-managed-ledger-ids --zkServer pulsar-zookeeper:2181 --managedLedgerPath /managed-ledgers/pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/persistent --ledgerIds 3596046
and
Copy code
bin/pulsar-managed-ledger-admin delete-managed-ledger-ids --zkServer pulsar-zookeeper:2181 --managedLedgerPath /managed-ledgers/pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/persistent/healthcheck --ledgerIds 3596046
return no Node found as it seems empty there.
Copy code
bin/pulsar-admin schemas get <persistent://pulsar/pulsar/pulsar-broker-0.pulsar-broker.pulsar.svc.cluster.local:8080/healthcheck>

 --- An unexpected error occurred in the server ---

Message: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=3596046 - operation=Failed to open ledger

Stacktrace:

java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.schema.exceptions.SchemaException: No such ledger exists on Metadata Server -  ledger=3596046 - operation=Failed to open ledger
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1099)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.readSchemaEntry(BookkeeperSchemaStorage.java:527)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$getSchema$10(BookkeeperSchemaStorage.java:254)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$getSchema$11(BookkeeperSchemaStorage.java:244)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.getSchema(BookkeeperSchemaStorage.java:240)
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.get(BookkeeperSchemaStorage.java:162)
	at org.apache.pulsar.broker.service.schema.SchemaRegistryServiceImpl.getSchema(SchemaRegistryServiceImpl.java:99)
	at org.apache.pulsar.broker.service.schema.SchemaRegistryServiceImpl.getSchema(SchemaRegistryServiceImpl.java:85)
	at org.apache.pulsar.broker.service.schema.validator.SchemaRegistryServiceWithSchemaDataValidator.getSchema(SchemaRegistryServiceWithSchemaDataValidator.java:52)
	at org.apache.pulsar.broker.admin.impl.SchemasResourceBase.getSchema(SchemasResourceBase.java:89)
	at org.apache.pulsar.broker.admin.v1.SchemasResource.getSchema(SchemasResource.java:84)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
it loooks like the local meta data store has a cached entry for this topic . cannot get rid of
Copy code
private CompletableFuture<Optional<LocatorEntry>> getSchemaLocator(String schema) {
        return locatorEntryCache.getWithStats(schema)
                .thenApply(o ->
                        o.map(r -> new LocatorEntry(r.getValue(), r.getStat().getVersion())));
    }
it calls this method to return the schema locator that has a position referencing the non-existing ledger. i dont know how to flush this cache
🤔 1
d
If this code is inside the broker, then a broker restart should work.
w
that does not seem work, as it seems it automatically reads from the zk meta data store rocksdb ?
Copy code
this.pulsar = pulsar;
        this.store = pulsar.getLocalMetadataStore();
        this.config = pulsar.getConfiguration();
        this.locatorEntryCache = store.getMetadataCache(new MetadataSerde<SchemaStorageFormat.SchemaLocator>() {
            @Override
            public byte[] serialize(String path, SchemaStorageFormat.SchemaLocator value) {
                return value.toByteArray();
            }

            @Override
            public SchemaStorageFormat.SchemaLocator deserialize(String path, byte[] content, Stat stat)
                    throws IOException {
                return SchemaStorageFormat.SchemaLocator.parseFrom(content);
            }
        });
it looksl ike
Copy code
return new ZKMetadataStore(metadataURL, metadataStoreConfig, enableSessionWatcher);
d
Unless you can delete the entry with the invalid ledgerID from ZK, it looks like we are stuck.
w
so i set up broker1 cluster, i first decommission old broker, cleaned up all the namespaces. then scale up default broker (the error persisted), then decommission broker1 . somehow it magically fixed the issue 😄
👏 1
it looks like the issues is that it can only clean up the healthcheck namespace if completely shutdown broker, so the broker1 helped .
3 Views
Open in Slack
PreviousNext
Powered by