Another question independent from the above - we observe exponential raise in time of processing aspect when running reindexing job - with 1.6 mln aspects we observe that initial batches are processed very quickly:

metrics so far RestoreIndicesResult(ignored=0, rowsMigrated=1000, timeSqlQueryMs=2, timeGetRowMs=0, timeUrnMs=6, timeEntityRegistryCheckMs=0, aspectCheckMs=0, createRecordMs=92, sendMessageMs=854)
Successfully sent MAEs for 1000/1651106 rows (0.06% of total). 0 rows ignored (0.00% of total)
0.02 mins taken. 33.25 est. mins to completion. Total mins est. = 33.27.

While time increases with each batch to reach 20 mins per batch at the end:

Reading rows 1650000 through 1651000 from the aspects table completed.
Successfully sent MAEs for 1649942/1651106 rows (99.93% of total). 58 rows ignored (0.00% of total)
2113.96 mins taken. 1.49 est. mins to completion. Total mins est. = 2115.45.
metrics so far RestoreIndicesResult(ignored=58, rowsMigrated=1650942, timeSqlQueryMs=40, timeGetRowMs=0, timeUrnMs=1821, timeEntityRegistryCheckMs=376, aspectCheckMs=296, createRecordMs=31830, sendMessageMs=80009)
Args are RestoreIndicesArgs(start=1651000, batchSize=1000, numThreads=1, batchDelayMs=100, aspectName=null, urn=null, urnLike=null)
Successfully sent MAEs for 1650942/1651106 rows (99.99% of total). 58 rows ignored (0.00% of total)
Reading rows 1651000 through 1652000 from the aspects table started.
2117.09 mins taken. 0.21 est. mins to completion. Total mins est. = 2117.30.

Did somebody experience similar problems with restore job? Can we improve performance somehow?

Hello Piotr, This behaviour is not uncommon due to the nature of how the reindexing job works. You can try and tweak the job’s configuration around batch sizes and number of threads to increase the performance of the job.

the nature of how the reindexing job works

Can you elaborate a little bit on this? what are the key and critical elements involved in the re-indexation? so we can monitor while doing our operations. Thanks

Reindexing in DataHub amounts to scanning the metadata graph in SQL for the latest aspects (with column version = 0) and for each one emitting an MCL event into the Kafka topic which when processed will upsert the graph and search databases just like a new piece of metadata would. Effectively it is re-processing the entire metadata graph found in SQL into the graph and search databases. The job itself that does the SQL scan and MCL message generation is multi-threaded and scalable through configuration. The most critical is really the mcl message processing and elasticsearch processing.

MCL processing is done by MAE consumer right? We did not see any significant increase in MAE consumer CPU usage even though there is a huge lag in the topic

Correct, mcl processing is done by the MAE consumer if you have enabled standalone consumers.

still connectors send mces though?

Which connectors?

i.e. https://github.com/datahub-project/datahub/blob/47be95689edebb0a847f4ab2b3aed9f1eabdbf4c/metadata-ingestion/src/datahub/ingestion/source/ldap.py#L336

Ah, yes there maybe some old connectors that have not yet been updated

This is code of the MCL consumer (formerly MAE consumer), right ? https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mae-consumer/[…]ava/com/linkedin/metadata/kafka/MetadataChangeLogProcessor.java Can we configure the poll/batch and commit strategy when consuming? https://kafka.apache.org/documentation/#consumerconfigs_max.poll.records https://kafka.apache.org/documentation/#consumerconfigs_auto.commit.interval.ms

Yes that is the MCL consumer code. You can configure them through

SPRING_KAFKA

environment variables.

Thanks Pedro