Hi all. I have a problem restoring Search and Grap...
# troubleshoot
n
Hi all. I have a problem restoring Search and Graph Indices from the local database. Steps are: - install datahub (0.10.2) via the own helm chart - restore Postgres data (about 6 million rows) - run datahub-upgrade/datahub-upgrade.sh -u RestoreIndices (It took several hours without errors in logs) After that, I see some data in UI, but not all of it, and they keep adding slowly. Also, I see a big lag (about 3 million items) for generic-mae-consumer-job-client for MetadataChangeLog_Versioned Kafka topic. The mae-consumer logs do not contain any errors (elasticsearch too), but look like slow-processing data What could be the problem? Thanks!
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: 1️⃣ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? Yes button 2️⃣ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Yes button
a
Hi, it can take a long time to restore indices on that much data, MAE consumer has to process the new data as well
this may be something that’s fixed with a little more waiting for stuff to come in
n
generic-mae-consumer-job-client hasn't finished processing the MetadataChangeLog_Versioned_v1 topic yet, and this process is already taking several days. Can we configure this or is it planned to fix it in future releases?
a
Hi, is this issue still affecting you?
i
@astonishing-answer-96712 this problem is still present in 0.10.4 release. After 0.10.0 with restoring every time at 1000 events in bulk - current speed is really slow (like in first picture).
plus1 2
r
i am facing this in v0.11.0
plus1 2
s
same here v0.11.0