#troubleshoot it occured error in the front when i viewing dateset belong to a tag. and it work well when pageNum is small, but it always fail when pageNum is more than 309. My colleage and i found it may be timeout when accessing graphql api

searchAcrossEntities

We looked through related code and found that

com.linkedin.metadata.search.cache.CacheableSearcher#getSearchResults

is the root case. just like its comment, it walks through all over the index from the beginning even though from a big pageNum. we are confused about this logic, I don't know what's side effect if using elasticSearch paging interface directly.

/**
   * Get search results corresponding to the input "from" and "size"
   * It goes through batches, starting from the beginning, until we get enough results to return
   * This let's us have batches that return a variable number of results (we have no idea which batch the "from" "size" page corresponds to)
   */
public SearchResult getSearchResults(int from, int size) {
  try (Timer.Context ignored = MetricUtils.timer(this.getClass(), "getSearchResults").time()) {
    int resultsSoFar = 0;
    int batchId = 0;
    boolean foundStart = false;
    List<SearchEntity> resultEntities = new ArrayList<>();
    SearchResult batchedResult;
    // Use do-while to make sure we run at least one batch to fetch metadata
    do {
      batchedResult = getBatch(batchId);
      int currentBatchSize = batchedResult.getEntities().size();
      // If the number of results in this batch is 0, no need to continue
      if (currentBatchSize == 0) {
        break;
      }
      if (resultsSoFar + currentBatchSize > from) {
        int startInBatch = foundStart ? 0 : from - resultsSoFar;
        int endInBatch = Math.min(currentBatchSize, startInBatch + size - resultEntities.size());
        resultEntities.addAll(batchedResult.getEntities().subList(startInBatch, endInBatch));
        foundStart = true;
      }
      // If current batch is smaller than the requested batch size, the next batch will return empty.
      if (currentBatchSize < batchSize) {
        break;
      }
      resultsSoFar += currentBatchSize;
      batchId++;
    } while (resultsSoFar < from + size);
    return new SearchResult().setEntities(new SearchEntityArray(resultEntities))
        .setMetadata(batchedResult.getMetadata())
        .setFrom(from)
        .setPageSize(size)
        .setNumEntities(batchedResult.getNumEntities());
  }
}

Thanks @astonishing-cartoon-6079! Our team is actively considering this PR: @orange-night-91387 @brainy-tent-14503