Hi all 🙂 I am having a few issues when running the Powerbi ingestion on a large project containing ±60 workspaces (the data I managed to extract so far is 85MB): 1 - Some workspaces ingestion crash with

404

. This happens when datahub tries to access some dashboard tiles using the admin

tiles

endpoint. I think this is a bug on PowerBI's side since these dashboards do exist, and other endpoints seem to work fine for them, but I think datahub shouldnt crash in such a case. 2 - overall the ingestion is painfully slow (more than a day) and some of the really large workspaces which have many input sources, never complete ingestion. I tried to debug this a little and it seems that the code is stuck on the

m_query

parsing in the

lark

library. Did anyone else try to use the Powerbi ingestion for large projects?

Thanks @bulky-evening-37162 for reporting these issues. Could you please attach the debug log for #1. tag me while replying

@gentle-hamburger-31302 thanks for the reply 🙏 attaching the log

just to pitch in, ingesting all of our workspaces takes just over 2 hours (we have around 250), this is the first run, haven't tested updating as we are just getting started. Also haven't look into only ingesting updated content.

does it succeed for you? there is another bug that causes ingestion to fail after some time due to expired token I have an open PR fixing it https://github.com/datahub-project/datahub/pull/8680

@bulky-evening-37162 I added one comment on the PR for CI failure

Will these fixes be merged? I see that the pull requests have been closed.

I had issues with running the tests locally and making them pass in the

404

PR and I didnt have the time to fix it so I closed the PR 😞 you can reopen it and make the tests pass if you want