Hello, we're trying to use the "download as csv" feature on search results. We filter results using the left sidebar, then click the "Download" link from the "..." menu at the top right. We've found that it works fine if there are only a few hundred results, but if we get up to a thousand or more results, it never returns a result. Even if we leave it for several hours, it still shows "Downloading" under the "..." menu. Should this feature work for 1000+ results? And if not, is there any other way to get this out of DataHub?

This approach should be able to handle up to 10,000 results in the exported list, if I remember correctly. Do you have any insight into how you elasticsearch cluster health looks? That will be the primary bottleneck for this download process cc @green-football-43791

Thanks John, great news that we should be able to do these exports. I do have insight into the elastic cluster, we have stats and alerts on it, and it always comes back healthy. Here's a screenshot of the dashboard from the elastic cluster. Some of the recent spikes might be from when I last tried to do a download from DataHub. I can see a bunch of calls to elasticsearch in the DataHub GMS service, but no errors from GMS or Elasticsearch that I could see, and still nothing returned in the frontend.

I have the same issue https://datahubspace.slack.com/archives/C029A3M079U/p1667500242731699 Works with < 100 lineage items, bonks when it's ~3k Are there any other ways to get this lineage data? An API or some other workaround?

Thanks, glad to hear it isn't just us 🤞