How can I save the categories of a product with multiple categories? Currently, I'm passing the category in

userData

to the category handler. However, since a URL is only scraped once, only the first category gets saved and all others discarded. Here's a minimal example.

ts
import { Router } from "crawlee";

export const router = Router.create();

// ... other handlers

// imagine the category handler is called twice
router.addHandler("CATEGORY", async ({ enqueueLinks }) => {
  // ... parsing

  // result of first call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "foo" },
    label: "DETAIL",
  });

  // result of second call
  await enqueueLinks({
    urls: ["https://example.com/my-product"],
    userData: { category: "bar" },
    label: "DETAIL",
  });
});

router.addHandler("DETAIL", async ({ request }) => {
  // todo: how to get both "foo" and "bar" here?
  const category = request.userData.category;

  // ... saving
});

message has been deleted

Hi @vwkd , If it is not possible to get the category list from the product detail, then you need to make sure, that you will crawl through all the categories first to make your own list of categories per product and then crawl through the product details. This mostly done by configuring and running 2 crawler Instances - first for the categories and the second one for product details (each crawler instance needs to be configured with its own RequestQueue and own router instance). During crawling categories you need to track and add the current category to every product, you can use

useState

to store the information and load the information withing multiple crawler instances:

js
import { useState } from 'crawlee';

// initialization categoryRouter, defaultRouter, categoryRequestQueue, detailRequestQueue

categoryRouter.addHandler("CATEGORY", async ({ enqueueLinks, crawler }) => {
  const categoryName = ...;

  const productCategories = await Actor.useState('productCategories', {}); // shared state

  const detailUrls = await page.$$eval('.product-detail a', (linkElements) => linkElements.map((linkEl) => linkEl.href)

  for (const detailUrl of detailUrls) {
    productCategories[detailUrl] ??= [];
    productCategories[detailUrl].push(categoryName);
  }

  // ...add product detail urls to the detailRequestQueue
});

detailRouter.addHandler("DETAIL", async ({ request }) => {
  const productCategories = await Actor.useState('productCategories', {});
  const categories = productCategories[request.url];

  // ... saving
});

Let me know if that makes sense.