https://discord.gg/botpress logo
Crawling Company Website
# 💻developers
f
Hello all! ✨ I am wondering, what is your favorite method to include company website as a knowledge base? As far as I see there are several options to include them: 1) Add A Website Source To {Knowledge Base} 1a) A Website 1b) Specific Web Pages 2) Search the web 2a) Search on specific websites 2b) The entire web What are your experiences with these methods runtime-wise and knowledge-extraction-wise towards successfully extracting relevant company informations? Thank you very much for sharing 🙏
Testing it as a Thread
w
@freezing-country-76570 i think it really depends on scale. if you're making a bot for one specific company, you're better off condensing the website into a single, well organized, well formatted file with clear information. if not, importing whole websites and webpages work nicely as well
how well organized your information is, is the main thing defining how good the results of the RAG retrieval are
runtime-wise, both solutions are indexed on the vector database so they're all extremely fast
f
@wide-umbrella-5226 thanks—that matches my gut feel. I’m building a bot for one specific company site, so I’m leaning toward a curated “company handbook” instead of a full crawl. My plan (tell me if this tracks with your experience): - Extract & normalize the site into 3–5 docs (About, Products/Services as tables, Pricing/Licensing, Support/Contact, FAQ). Keep canonical product names + synonyms. - Pre-index in the KB, answer only from KB by default. - Use domain-scoped web search only for things that change often (press/news/downloads). - Keep chunks small-ish and pass ~4–6 top hits; favor headings/tables to improve retrieval.
w
i think that's a great and very smart plan! just one suggestion: i don't think you even need to split the website into multiple docs. i think a single one would be fine - it'll already be chunked nicely
then after that, just play around a bit with the RAG model to find one that balances speed and intelligence well for your use case
f
Ah I understand - that makes it even easier! Thanks a lot for your kind help 🙂
s
Hi there, trying this thread as I need to do something similar. What is my best approach if I have a large website with about 7000 pages I need to build a knowledge base on? I try the normal url KB but it seems to timeout on about 100 pages?