Hi everyone! We frequently need to run a re-index ...
# all-things-deploymentb
brief-oyster-50637
10/28/2023, 5:27 PMHi everyone! We frequently need to run a re-index when we notice the data is not correct on the UI or when we need to restore from the backup due to a problem with docker (using the quickstart/docker-compose deployment on a VM).
But the re-index is veeeerrry slow, it can take 2 days. We have about 8k data assets in the DB (Bigquery + Tableau). How can I expedite the re-index? I noticed that the Elasticsearch container only uses 1GB of memory, even though I configured it to use up to 2GB, so this doesn’t seem to be bottleneck. What else could be then? What is usually the bottleneck of the indexing process?
I appreciate any suggestions! 🙏🏽 🙂
r
rich-barista-93413
11/01/2023, 9:11 AM@billions-tailor-78534 might be able to speak to this!
thank you 1
rich-barista-93413
11/01/2023, 5:18 PMThere's a section in the Elasticsearch manual about tuning for indexing speed: https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html
b
billions-tailor-78534
11/01/2023, 5:23 PMBut it's likely that the limiting factor here might be the deployment. You might want to look at the options for Docker storage drivers or something like that.
thank you 1
r
rich-barista-93413
11/01/2023, 5:25 PMYou could consider running Elasticsearch externally as well (e.g. on AWS)
rich-barista-93413
11/01/2023, 5:26 PMAnd of course Acryl Data provides a fully managed setup where we manage these services for you
b
brief-oyster-50637
11/05/2023, 1:45 PMNice, thank you @delightful-ram-75848 and @billions-tailor-78534. I’ll look at the options for Docker storage drivers so I can try to figure it out. Thank you!
3 Views