Hi my problem is crawler.run(['https://keepa.com/#!product/4-B07GS6ZB7T', 'https://keepa.com/#!product/4-B0BZSWWK48']) only scrap the first URL I think this is because crawlee think they are the same URL , if i replace the "#" with a "?" it works , is there any way to make it work with url like this ?

message has been deleted

Hi @FoudreTower The #! fragment are used for client-side navigation only. So the crawler sees these as duplicates. When you you change it for ? its no longer the hashtag fragment and the crawlee takes the whole url when deduping. One way around this would be to to add

uniqueKey

when enqueuing.

thanks @Lukas Celnar it works with uniquekey