save HTML file using crawlee
# crawlee-jsn
Nyanmaru
08/28/2024, 6:00 PMHas anybody tried downloading the HTML file of the URL using Crawlee? Was wondering if Crawlee has a capacity of downloading the HTML file of the URL since I've just been using Crawlee and really loving the experience.
h
Hall
08/28/2024, 6:00 PMmessage has been deleted
e
Exp
08/28/2024, 11:38 PMYou can download HTML content of a webpages using Crawlee
m
Marco
08/29/2024, 7:32 AMIt depends on which crawler you are using:
- Cheerio: https://cheerio.js.org/docs/api/classes/Cheerio#html
- Playwright: https://playwright.dev/docs/api/class-page#page-content
- Puppeteer: https://pptr.dev/api/puppeteer.page.content
n
Nyanmaru
08/30/2024, 3:12 PMThanks for this awesome answer! Was wondering if Crawlee has examples on how to save it to a file?
m
Marco
09/02/2024, 10:04 AMYou can use the KeyValueStore: https://crawlee.dev/api/core/class/KeyValueStore. E.g., with Cheerio:
Copy code
await store.setValue('my-html', $.html('html'), { contentType: 'text/html' });c
Crafty
09/04/2024, 3:32 PMeverything you save to a crawlee store is saved to disk, and can be accessed through crawlee or otherwise. Something my company is using is saving page.content() to a cloud storage bucket during the request handler which has worked quite well for us since it offloads the data nice and quickly.
s
shanky
09/22/2024, 7:59 PM1. I want to retrieve html for some specific table, say I want to retrieve html for "item-list" div in the code below (not the data inside each element). How to do this?
Copy code
<body>
...
...
<div class="item-list">
<div class="item">
<div class="product-label"></div>
<div class="product-image"></div>
<div class="product-cost"></div>
</div>
<div class="item">
<div class="product-label"></div>
<div class="product-image"></div>
<div class="product-cost"></div>
</div>
...
...
</div> <!-- end item-list -->
...
...
<div class="testimonials">
...
...
...
<div>
</body>