Hi, we have recently found the 10K elements limita...
# ui
w
Hi, we have recently found the 10K elements limitation when fetching datasets with a given tag. This PR is somehow addressing this issue, in this case with the domain. https://github.com/datahub-project/datahub/pull/5838 While the counter text could be fixed following that approach (
10000+
), my concern is when actually the user navigates the listing. He cannot reach datasets beyond item 10000. What’s your view on that? Could this be fixed by increasing or removing the limit?
CCing the ones involved in the PR above since you already have the context: @green-football-43791 @big-carpet-38439
CCing my colleague @prehistoric-lamp-99382 who was looking into this too
g
Hey @witty-butcher-82399! The limitation is coming from elasticsearch, not a limitation within datahub. Here is some explanation form elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.5/search-request-body.html#request-body-search-track-total-hits
we are looking into using elasticsearch’s deep pagination api to get a more accurate total
but its unclear if it would be responsive enough
w
Thanks for the reply! And good to know you are looking into ways to overcome this.