hey everyone! question for people running production RAG systems, how do you handle document updates? suppose you have thousands or millions of chunks already embedded and indexed. a source document changes (new section added, policy updated, docs edited etc.) do you re-embed the entire document, re-embed only affected chunks, use some kind of diffing/hash-based approach?or not worry about the extra embedding cost? i'm asking because i built a small experiment that tracks chunk-level hashes and only re-embeds chunks whose content changed. before i spend more time on this, i'm trying to understand whether this is an actual pain point people experience in production or whether most people simply re-embed everything or use some other techniques.

yeah this is a really good question. there are a few ways to do this. let me find someone that can answer this for you

sure thanks

Hey @clean-butcher-75125 , it depends, but I think most implementations just re-chunk/re-embed the whole document. This is probably good enough in terms of costs and is the easiest to implement, since the new content could have ripple affects on all the chunks (boundaries, overlap, etc.) depending on the chunking strategy you're using. Some of the key inputs to the decision are understanding how often updates happen in the first place, as well as how large the documents are. If updates are occasional, then re-chunking and embedding the whole document is fine/negligible. If docs are updated frequently, and if the documents are on the larger side, it might be worth getting more granular. Another thing to consider is how documents change. For example, if the edits are localized to one small section vs edits throughout the document. If you end up implementing a granular strategy to only re-process what has changed, then implementing a hashing mechanism to identify when content has/hasn't changed is probably the way to go. There are some edge cases too, like needing to know when a given chunk shouldn't be re-embedded but should rather be deleted from the index. Hope this helps!

Thanks, this is helpful. One thing I'm curious about: in practice, when teams have very large corpora (millions of chunks), do they usually accept the cost of full re-embedding because updates are infrequent, or have you seen teams build more incremental pipelines internally?

I think it comes down to whether you can identify which chunks changed, and how much of the corpora has changed overall. Like is it 1% of the chunks or 50% that changed. If out of 1M chunks you know specifically just these 100 need to be updated, then doing it incremental makes sense. If you're not sure what changed, or if say several hundred thousand chunks changed, then you might lean more towards just re-doing the whole thing