A short question: is it correct that there is a ha...
# all-things-deploymentb
brave-businessperson-3969
01/28/2022, 4:18 PMA short question: is it correct that there is a hard limit of 10 000 on the number of results you can get from a search query? I've run into strange errors when sending search requests via GraphQL API programmatically: "code 500 SERVER_ERROR, classification "DataFetchingException" " as soon as I tried to set the "start" parameter in GraphQL search queries to values >= 10 000
b
big-carpet-38439
01/28/2022, 10:33 PMcc @early-lamp-41924 Is this part of those elastic limits?
e
early-lamp-41924
01/28/2022, 10:34 PMyes. elasticsearch puts a limit on number of search results returned
early-lamp-41924
01/28/2022, 10:34 PMit just doesn’t handle it
b
brave-businessperson-3969
01/31/2022, 10:08 AMAh, OK. Thanks for the info. So this is not a DataHub setting/issue but something I have to configure in elasticsearch. (or find another approache because elastic documentation discourage increasing the limit as I have just read ...)
b
big-carpet-38439
02/18/2022, 2:54 AMYes 😞 What are you hoping to achieve uwe?
b
brave-businessperson-3969
03/08/2022, 10:12 PMWell, the usecase was quite simple:
• Iterate over all datasets of a DataHub installation and perform a set of actions (add links, remove tags, etc.).
The logic to perform the actions was quite complex. So I wrote a small python script which essentially performed a "search * limit to DataSets" search via the graphql interface and looped over the results. The catalog contains some quite large DBs and therefore the search query ran into the 10 000 results limit.
brave-businessperson-3969
03/08/2022, 10:16 PMAt the moment I've solved the problem by creating smaller artificial "groups" (adding additional constrains to the search query to limit the expected number of results). But as soon as one of my groups gets over the 10 000 the problem will appear again. So, not an ideal and especially not a long term solution.
e
early-lamp-41924
03/08/2022, 10:34 PMRight. Seems like we should have the ability to scroll through the whole list. We could use the elasticsearch scroll api for this but that also has some limitations like there can only be 500 concurrent scrolls at any time
early-lamp-41924
03/08/2022, 10:34 PMThe otherway would be to do a full scan of the mysql table
b
big-carpet-38439
03/08/2022, 11:39 PMYou can currently do that with the "entities" "listUrns" entity, but we don't publicize this as a public API.
e
early-lamp-41924
03/08/2022, 11:54 PMBut that uses elasticsearch right? so it has the same limitation