one architectural decision that consistently improved long-term reliability was separating crawling from extraction. Website navigation, retries, session management, and scheduling remained stable infrastructure services, while parsing and data normalization evolved independently as websites changed. That separation dramatically reduced maintenance effort because adapting to content changes rarely required touching the crawling layer itself.