https://apify.com/ logo
Join Discord
Powered by
How to cleanly cancel/abort a queued request befor...
# crawlee-js
f
Hey everyone! I'm using 
CheerioCrawler
and need a way to conditionally drop/abort requests that are already in the 
RequestQueue
when a runtime condition is met. The goal is to completely discard the request so it: - Does not make the network request. - Does not execute the 
requestHandler
. - Does not count as a failed request or log an error. ## What I tried ### 1. Throwing a custom error in 
preNavigationHooks
Copy code
ts
preNavigationHooks: [
  async ({ request }) => {
    if (shouldAbortRequest(request)) {
      request.noRetry = true;
      throw new CustomAbortError();
    }
  },
]
I also swallow the error in 
errorHandler
and 
failedRequestHandler
. Problem: Crawlee still marks the request as FAILED, logs 
Request failed and reached maximum retries
, and increments failed request metrics. ### 2. Using 
request.skipNavigation = true
Copy code
ts
preNavigationHooks: [
  async ({ request }) => {
    if (shouldAbortRequest(request)) {
      request.skipNavigation = true;
    }
  },
]
Problem: This skips the HTTP request, but 
requestHandler
still executes unless I add manual guard logic. ## Question Is there an official/idiomatic way to completely cancel an enqueued request so it is silently discarded without: - Making the network request. - Running the route/request handler. - Being treated as a failed request. Thanks!
a
message has been deleted
a
@fierDeToiMonGrand just advanced to level 6! Thanks for your contributions! 🎉
f
There is no dedicated public “silently cancel this currently-processing request” API in CheerioCrawler that behaves exactly like return from the crawler lifecycle. Your observations are correct: Throwing from preNavigationHooks is treated as a processing failure/retry path, so it affects failed-request handling and metrics. request.skipNavigation = true only skips navigation; Crawlee explicitly defines it as “skip the navigation and process the request directly,” so the requestHandler can still run. The cleanest idiomatic solution is therefore to filter/remove the request before Crawlee starts processing it, rather than aborting from preNavigationHooks. RequestQueue supports dynamically adding and removing requests. If the condition can only be evaluated once the request has been fetched from the queue, the practical pattern is to use skipNavigation plus an early return in the handler:
const crawler = new CheerioCrawler({ preNavigationHooks: [ async ({ request }) => { if (shouldAbortRequest(request)) { request.skipNavigation = true; request.userData.skipRequest = true; } }, ], async requestHandler({ request, $ }) { if (request.userData.skipRequest) { return; } // Normal processing console.log(
Processing ${request.url}
); }, });
his gives you the behavior you want from an operational perspective: Network request ❌ Real handler logic ❌ Failure/retry ❌ Failed request metric ❌ Error log ❌ The small caveat is that the requestHandler function itself is technically invoked; it simply exits immediately. That follows directly from Crawlee's semantics for skipNavigation.
If you truly require the handler to never be invoked at all, the decision needs to happen at the queue/provider level. For example, if you control the pending requests: const queue = await RequestQueue.open(); const request = await queue.getRequest(requestId); if (request && shouldAbortRequest(request)) { await queue.deleteRequest(request.id); } Then the crawler never receives that request. RequestQueue is explicitly designed as the dynamic source of requests consumed by Crawlee crawlers and supports removing requests.
-------------------------------------------------------- result
there currently isn't an equivalent of context.abort() / RequestCancelled that you can throw from preNavigationHooks to silently terminate processing. Throwing is part of Crawlee's failure lifecycle, while skipNavigation only prevents navigation. If possible, remove/filter the request at the RequestQueue level before the crawler takes it. Otherwise, the usual workaround is request.skipNavigation = true plus a flag in userData and an immediate return at the beginning of requestHandler. Don't use exceptions for this case, because Crawlee intentionally interprets exceptions from request processing as failures/retries.
f
Ill test some solutions based on your feedback and come back to you, thanks!
2 Views
PreviousNext
Powered by