Newer Crawlee Versions Cause Double Request Queue ...
# crawlee-js
f
I'm experiencing a critical cost issue where Crawlee 3.15.3 writes 2x more request queue operations compared to version 3.12.1 for the exact same actor and input on Apify platform. This doubles my costs, so I need to use 3.12.1, but I'm hitting compatibility issues. ### The Problem (When running locally on my machine) When trying to use
crawlee@3.12.1
with
apify@^3.0.0
(3.2.6), I encounter two blocking issues 1. TypeScript compilation error - Type mismatch in proxy configuration:
Copy code
typescript
const proxyConfiguration = await Actor.createProxyConfiguration();
const crawler = new CheerioCrawler({
  proxyConfiguration: proxyConfiguration, // ❌ Type error
  requestHandler: router,
});
2. Runtime error when running locally (even after ignoring TypeScript with
as any
):
Copy code
Error: Detected incompatible Crawlee version used by the SDK.
User installed 3.12.1 but the SDK uses 3.15.3.
    at checkCrawleeVersion (node_modules/apify/utils.js:68)
    at Actor.init (node_modules/apify/main.js:123)
### Why I Need 3.12.1 Specifically Cost Impact: For the same run with identical input, Crawlee 3.15.3 generates roughly double the request queue writes compared to 3.12.1: (See attached screenshot) I don't know what changed between these versions, but 3.15.3 is significantly more expensive to run. ### My Current Solution (Need Validation) To force deduplication locally and replicate Apify's behavior, I added npm overrides to package.json:
Copy code
json
{
  "dependencies": {
    "apify": "^3.0.0",
    "crawlee": "3.12.1"
  },
  "overrides": {
    "@crawlee/core": "3.12.1",
    "@crawlee/types": "3.12.1",
    "@crawlee/utils": "3.12.1"
  }
}
### My Questions 1. Is using npm "overrides" the correct approach here? Will this cause issues when deploying to Apify platform? 2. Why does Crawlee 3.15.3 generate 2x more request queue writes than 3.12.1? What changed between these versions that causes this cost increase? https://cdn.discordapp.com/attachments/1459284049213390930/1459284049670439085/image.png?ex=697bc42d&is=697a72ad&hm=cdbd36bd99fd062bbd4206205376088a70de3b9b6f17430b956ddad86820a2a1& https://cdn.discordapp.com/attachments/1459284049213390930/1459284050295656488/image.png?ex=697bc42d&is=697a72ad&hm=89afe8139411796d65547e38c74c182e5726b16c91336a58baa1f3bc2b4e523d&
n
Hello, I verified the setup locally, and there are no type mismatches with version 3.15.3. Please check your project for any dependency conflicts. Regarding the increased request queue operations: this change was introduced in Crawlee 3.13 to fix race conditions in RequestQueue v2. If you only have a single Actor instance working with the queue at any time, you can switch to RequestQueueV1 to reduce operations/costs: https://crawlee.dev/api/core/class/RequestQueueV1. Thank you.
f
How to use it inside a Crawlee router handler context? For example:
Copy code
javascript
router.addHandler(labels.START, async ({ crawler, request }) => {
  const urls = [
    { url: 'some-url', label: 'some-label' }
  ];
  await crawler.addRequests(urls); // <--- Here
});
- It would be good to mention in the docs that using V1 is cheaper than V2. - Are there any other new features or changes in recent Crawlee versions that could increase operation costs that we should be aware of?
n
Hello, You can set the request queue instance here: https://crawlee.dev/js/api/browser-crawler/class/BrowserCrawler#requestQueue Thank you for your feedback. Regarding news and updates, you can check our changelogs: https://crawlee.dev/js/api/core/changelog https://crawlee.dev/python/docs/changelog
2 Views