Hi Team, After upgrading my datahub version from

0.8.43

to

0.9.6.1

I am facing errors with reindexing...

17:30:57 [main] INFO  c.l.m.s.e.i.ESIndexBuilder - Reindexing dataset_operationaspect_v1 to dataset_operationaspect_v1_1674751305780 task has completed, will now check if reindex was successful
17:31:00 [main] INFO  c.l.m.s.e.i.ESIndexBuilder - Post-reindex document count is different, source_doc_count: 34822915 reindex_doc_count: 15463000
17:31:00 [main] WARN  o.s.w.c.s.XmlWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'metadataChangeLogProcessor' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-3785592998662924994/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/MetadataChangeLogProcessor.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'updateIndicesHook' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-3785592998662924994/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/hook/UpdateIndicesHook.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is java.lang.RuntimeException: Reindex from dataset_operationaspect_v1 to dataset_operationaspect_v1_1674751305780 failed
17:31:00 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient - Shutdown requested
17:31:00 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient - Shutting down

Any body else faced this issue? Any tips would help....

Hey Arun! Mismatched doc counts can happen when the number of entities in the MySQL DB does not align with what is in the ElasticSearch index or errors occur when updating the field mappings for documents. This is saying that there are approximately double the count of docs in your source index to what the reindex job is producing which is concerning.

@orange-night-91387 Thanks for looking into it... There were no other information on the gms logs other than

Reindexing dataset_operationaspect_v1 to dataset_operationaspect_v1_1674751305780......

What are really part of the operationaspect index?

There were some changes to the schema since 0.8.43, the only real concerning one:

lastUpdatedTimestamp: long

being a required field and not being present in the original type maybe? I would think that would cause everything to fail though if it were just an incompatible schema. All the other added fields are optional so I wouldn't think they would cause issues.

It was able to ingest data yesterday, today when doing reindexing is what it failed....Does the reindexing has a compatible schema as well?

It'll just use whatever is in the current models. Basically what it's doing is creating a temporary index, running a full restoreIndices on the MySQL data with the new models and that index, then compares the historical index to the newly created one.

Is there any other fix other deleting the index itself?

There is a complicated manual process you could go through of setting up the alias manually and moving things over, but if you can wait for this PR to go in your options open up a bit as you can clone the previous index to prevent data loss and skip over the check, then compare the difference in records. Should be within the next couple of weeks.

As the application was not starting up in production due to this issue i had to rename the index and reingest the data to start... Issue is resolved for now but the root cause is still not clear