This is the forum of the developer community of Apify. 

Find answers to pre-existing questions about Crawlee, Apify, and our other open-source projects here.

Apify

As title says, not sure why we're receiving this message now, I don't want ANY urls skipped and there doesn't seem to be a way to make sure they do get re-queued?  Is this benign internal tracking messaging? 

I upgraded from 3.13.0 -> 3.16.0 and noticed this behavior.  In researching it seems like its tied to `maxRequestsPerCrawl` but this setting is to prevent infinite looping of certain sites. What am I missing?

Current setting is `maxRequestsPerCrawl: 20000`

Thanks.

Well for now, I'm going to no longer pass `enqueueLinks` to the `requestHandler` and rely on
`.addRequests()` method, this seems to be the only way to keep using `maxRequestsPerCrawl` usable

Not sure if I understood the issue correctly. Crawlee generates a `uniqueKey` for every request in the queue, based on its URL, unless you define a `uniqueKey` yourself. If you enqueue two requests with the same URL, without giving them different `uniqueKey`s, the second request will be skipped. This behavior prevents Crawlee from handling the same request twice after a restart.