Hi team, for some reason, I use v0.9.6.1 in dev en...
# troubleshoot
m
Hi team, for some reason, I use v0.9.6.1 in dev env and use v0.8.43 in prod env and there are 3000 tables in dev and 5000 tables in prod from hive source. To ingest the metadata into Datahub, it takes about 1h in dev, while more than 5 days in prod. I'm wondering what caused the huge difference. Does it have anything to do with the version or how should I troubleshoot?
Copy code
[2023-03-31 14:47:57,830] INFO     {datahub.cli.ingest_cli:170} - DataHub CLI version: 0.8.43
[2023-04-04 10:50:44,875] INFO     {datahub.cli.ingest_cli:137} - Finished metadata ingestion
Command exiting with ret '0'
πŸ“– 1
πŸ” 1
l
Hey there πŸ‘‹ I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: βœ… There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? βœ… It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react βœ… to your original message to let us know!
a
There have been a lot of performance improvements since v8- you could try spinning up more ES instances or increasing resources on the pods, but it’s hard to know in a version that old
m
@astonishing-answer-96712 which version at least do you suggest to use?
a
0.10.1 is the most recent, 0.9.6.1 is stable for most