PlaywrightCrawler not competing the jobs I added in the queue using addRequests(). I am self hosting crawlee and running playwrightCrawler and I add 100-150 requests in queue and and maxComcurrency is 12 and the crawlee don’t finish all the requests that I added. It is never started the crawler for 3 url. I try different urls different websites but it is skipping or whatever but it is no started or no pick 2-4 request from 100-150

message has been deleted

Hey @Hamid Jutt, could you please provide more details? I don't really understand what you are saying. > I am self hosting crawlee and running playwrightCrawler So you are running the Actor locally, not on the Apify platform? > I add 100-150 requests in queue and and maxComcurrency is 12 and the crawlee don’t finish all the requests that I added What is the reason it doesn't finish the requests? Does it fail/crash? Or does it just ignore some of the requests (could be the case if you run it without purging the RequestQueue first, or you have a maxRequests limitation set up). Also if your requests share url (

#

part is ignored) the requests are processed as single unless you set a specific

uniqueKey

for each request.

Yes, I am running the actor locally. There is not any crash, it is just ignored. I also set uniqueKey,

Ok, we can investigate it further if you want. - Make sure all your requests get into a request queue - after the run, check your local

storage/request_queues/default

(or name/id of your RequestQueue, if you don't use default), and count the number of requests there. - If the number is correct, uniqueKey works well, and the issue is with the processing of the requests. - Check

maxRequestPerCrawl

crawler option is either not set or higher than/equal to the number of requests you have. Note that this kind of investigation is hard for me, you should either provide the code example or do it yourself...

Here is code, you can check: https://docs.google.com/document/d/1-8e9sURTmpcCJ0iomG3FE0i1KIJOCN1l7pWLdvTt0Fk/edit?usp=drivesdk

Have you compared the number of requests as I wrote above?

i test it:

console.log(requestId, targetUrl, "Recived in ScrapeUrl for Playwright and adding it to Queue")

try {
        const result = await crawler.addRequests([{
            url: targetUrl,
            uniqueKey: requestId, // Now guaranteed unique
            userData: { requestId, formats, targetUrl }
        }], { waitForAllRequestsToBeAdded: true }); 

        // 2. Correctly check if it was skipped (Matous's fix)
        const myRequest = result.addedRequests.find((req) => req.uniqueKey === requestId);

        // 3. Check if it was skipped/ignored
        if (myRequest.wasAlreadyPresent || myRequest.wasAlreadyHandled) {
            console.error(`⚠️ [${requestId}] Crawlee ignored this because it's a duplicate ID!`);
        }


    } catch (err) {
        return { url: targetUrl, success: false, error: err.message };
    }

I see the logs for

requestId, targetUrl, "Received in ScrapeUrl for Playwright and adding it to Queue

. But I never received the error in the console. Means it is added to the queue. I also test to log in preNavigationHooks

export const crawler = new PlaywrightCrawler({
    keepAlive: true,
    maxRequestRetries: 1,
    maxSessionRotations: 1,
    maxRequestsPerCrawl: 999999, 
    maxConcurrency: 12,
    preNavigationHooks: [
        async ({ request }) => {
            const { requestId } = request.userData;
            console.log(`🚀 [${requestId}] START PlaywrightCrawler for ${request.url}`);
        }
    ],..

But not log it in console, which means the request does not reach the preNavigationHooks. It logs it

[Example-ID] START PlaywrightCrawler for https://example.com

for other requests, but not for 1-5 requests, which are skipped.

Nice, you have have identified your requests are in the ReuqestQueue (and you have potentially checked the

storage/request_queues/default

) as I mentioned before. Now, the problem is that the crawler is not processing them. For this case, check the previous suggestions I gave: - You run your crawler before you add any requests to it -> it will stop, so unless you run it again somewhere later, your requests won't be processed - I would encourage you to check whether the crawler is running when you add your requests, and if not, just run it - I don't see any Actor.init() in your code, which might also affect the request queue

You told

You run your crawler before you add any requests to it -> it will stop, so unless you run it again somewhere later, your requests won't be processed

but I am using

keepAlive: true,

so I think it will work, because it works for many, only some skipped 1-5.

Ok, now it seems the storage configuration is broken, so just avoid using it altogether (at least for debugging; it might be the cause). Then I am not sure how you add requests via the API to the local Actor. I suggest simplifying everything so you can identify the bug. I am still worried about how you run the Actor with no requests before adding the requests, you are basically working with unknowns - it's not deterministic and thus very hard to debug.

@Hamid Jutt just advanced to level 2! Thanks for your contributions! 🎉

Hey, from the messages you deleted, it seems we are back at square 1 - not all the requests are added. Please check this code (main for template cheerio crawler) and notice the WAIT part - if it is

true

I get both request processed, if it is

false

, they are not.

Typescript
// Crawlee - web scraping and browser automation library (Read more at https://crawlee.dev)
import { CheerioCrawler } from '@crawlee/cheerio';
// Apify SDK - toolkit for building Apify Actors (Read more at https://docs.apify.com/sdk/js/)
import { Actor } from 'apify';

const WAIT = true;

// The init() call configures the Actor to correctly work with the Apify-provided environment - mainly the storage infrastructure. It is necessary that every Actor performs an init() call.
await Actor.init();

const proxyConfiguration = await Actor.createProxyConfiguration({ checkAccess: true });

const crawler = new CheerioCrawler({
    proxyConfiguration,
    requestHandler: async ({ request, log }) => {
        log.info(`Processing ${request.url}...`);
    },
    keepAlive: true,
});

const crawlerPromise = crawler.run();

await crawler.addRequests([
    {
        url: 'https://www.example.com',
    },
]);

await crawler.addRequests([
    {
        url: 'https://www.apify.com',
    },
]);

if (WAIT) {
    await new Promise((resolve) => {
        setTimeout(resolve, 5000);
    });
}

crawler.stop();
await crawlerPromise;

// Gracefully exit the Actor process. It's recommended to quit all Actors with an exit()
await Actor.exit();

My bad, but I checked that all requests are adding in the queue successfully. but i write some code to debug:

app.get('/scrape/status', async (req, res) => {
  try {
    const playwrightQueueInstance = await crawler.getRequestQueue();
    const info = await playwrightQueueInstance.getInfo();

    const all_pending_Requests_in_db = await playwrightQueueInstance.client.listHead({ limit: 1000 });
    const crawleeIds = all_pending_Requests_in_db.items.map(i => i.userData.requestId);

    const statusReport = {
      timestamp: new Date().toLocaleTimeString(),
      local_trackers: {
        total_ever_received: allRequests.size,
        currently_waiting_for_n8n: requestsNotCompleted.size,
        stuck_request_ids: Array.from(requestsNotCompletedYetWaitingOrStuck),
      },
      playwright_crawler_db: {
        total_added: info.totalRequestCount, 
        waiting_in_line: info.pendingRequestCount, 
        finished_successfully: info.handledRequestCount,
        isFinished: await playwrightQueueInstance.isFinished()
      },
      engine_realtime: {
        active_browsers: crawler.autoscaledPool?.currentConcurrency || 0,
        is_system_overloaded: crawler.autoscaledPool?.isOverloaded || false
      },
      bridge: {
        ids_added_but_not_in_db: Array.from(requestsNotCompleted).filter(id => !crawleeIds.includes(id)),

      }
    };
    res.json(statusReport);
  } catch (err) {
    res.status(500).json({ error: "Failed to fetch status", details: err.message });
  }
});

app.post('/scrape/status/id', async(req, res)=>{
  const { id } = req.body;
  try{
    const playwrightQueueInstance = await crawler.getRequestQueue();
    const data = await playwrightQueueInstance.getRequest(id);
    res.json(data)
  }catch (err) {
        res.status(500).json({ error: err.message });
    }
    

})

Hey, thanks for your question. I don't really understand your

stuck_request_ids

and

ids_added_but_not_in_db

since I don't know where they come from (what are

requestsNotCompletedYetWaitingOrStuck

and

requestsNotCompleted

?) But In the final request you get in the end has

handledAt

field, so I would think, the request was already handled.

Hey @Matous , thanks for looking into this! To clarify, here is where those trackers come from and why they prove a problem: 1. What are the Trackers? requestsNotCompleted: This is a local JavaScript Set in my Express server. When n8n hits my API, I add the ID to this list. I only remove it when the requestHandler actually finishes and sends data back to me. ids_added_but_not_in_db: I check the active head of the Crawlee queue. If an ID is in my local list but not in Crawlee’s active queue, it means the request is 'finished' in the database but my code never received the data. 2. The Mystery: The "Ghost Success" You are right—Crawlee says handledAt: 12:22:26 and state: 3. This means the database thinks it’s done. However: My requestHandler (where I scrape the data) never fired for this ID. No logs, no data extraction. Because the requestHandler didn't run, my store never got the 'Finished' signal. As a result, n8n stayed waiting , even though the database says 'Success.' 3. The Conclusion The issue is that Crawlee is marking these requests as handled (State 3) without actually opening the browser or running my code.

Hey, I am sorry, but I feel like we are getting nowhere this way, and I don't want to waste more of your time - I am unable to help you like this. Here are some of my thoughts, and at the end of the message, I will write a suggestion for your next steps. The best approach to debugging a complex issue is to simplify it to only the crucial, verified parts that cause the bug. That way, you can monitor the flow reliably and identify it with ease. What we are doing here is just guessing. You provide some kind of explanation of what you think is happening (and most likely you are right), but can you prove it so we are sure? This way, we can't set up a valid base of common ground on which we could build the debugging process. Also, you don't let me actually test it myself, so I could confirm it. So, for your next steps, I suggest following the common bug-report process: You should create a minimal repro = some code that: - other developer can run themself - doesn't contain extra logic that is not relevant to the problem - clearly proving bug is present In your case, that might be very simple code consisting of something like this: - Actor - smallest possible list of requests - logic simulating your way of adding requests - (not necessarily) logic for checking the request But make sure the logic for checking is as simple as possible, using conventional methods. Once you have something like that, we can discuss it further, or you can preferably file an issue in the crawlee repo: https://github.com/apify/crawlee/issues/new?template=bug_report.yml Good luck with your troubles!

Hi @Matous, Check it here: https://github.com/apify/crawlee/issues/3427

Hey @Hamid Jutt, I have checked it just now and saw Jindra from the crawlee team already replied to you - he should help you tackle this. Good luck!

message has been deleted

@Matous but he is not replying !