hi,team。Asked about the issue of lineage loading. ...
# all-things-deploymentc
cuddly-arm-8412
04/03/2023, 9:33 AMhi,team。Asked about the issue of lineage loading. version:v0.10.1
1. I found that although I have more than 500+downstream lineages, only 100 downstream lineages are displayed by default.
2. When I modify the parameter to obtain 2000 lineages, I prompt "Async processing failed"; nested exception is java.lang.StackOverflowError
Is there a better plan or distributed loading plan for loading a large amount of data.
🔍 1
📖 1
l
lively-cat-88289
04/03/2023, 9:33 AMHey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request.
Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
astonishing-answer-96712
04/03/2023, 4:35 PMHi @cuddly-arm-8412, how many ES containers are you running? Increasing resources there or adding some backups might help ehre
c
cuddly-arm-8412
04/04/2023, 1:26 AM@astonishing-answer-96712 hi,thank you ,I started the datahub quickstart command directly from the official command. Currently, only one container has been started. How can I verify multiple containers locally, while copying a single container?
cuddly-arm-8412
04/04/2023, 1:36 AM@astonishing-answer-96712 hi,may I ask a question,I am looking at the logs. If we obtain 1000 bloodlines at once, whether it is batch or item by item, I currently feel that the more possible quantity, memory consumption, and time consumption caused by item by item acquisition, is there a logic for batch acquisition.
a
astonishing-answer-96712
04/07/2023, 9:26 PMHey, so the previous issue was resolved?
astonishing-answer-96712
04/07/2023, 9:27 PMsee our batch ingestion logic here: https://datahubproject.io/docs/managed-datahub/metadata-ingestion-with-acryl/ingestion/#batch-ingestion
c
cuddly-arm-8412
04/10/2023, 5:38 AM@astonishing-answer-96712 not resolved,We are currently facing data with over 1000 blood relatives downstream, and loading is still not supported. Only partial data can be rendered
a
astonishing-answer-96712
04/12/2023, 6:25 PMHi, this would be a good github bug to file- it could also by a resources issue with the frontend or GMS pods, that’s a lot of lineage!
2 Views