I'm building a receipts extraction pipeline. Some of the receipts can have 100+ line items. Different vendors may list net amount per line item (after discount) or other vendors may list gross amount then the discount amount below on its own line item. It's causing a nightmare when I try to make a one-size-fits-all BAML schema. When I optimize for one receipt type, it breaks the other receipt types. Should I just make one BAML file with different prompt sections dedicated to each receipt type - (single-pass with embedded vendor detection, but gets bloated/polluted context window) - the union return type IS the routing. OR is it a better practice to do multi-pass routing (multiple API calls so double the token cost and latency). I do think that the image file size for the prompt tokens is exponentially larger than more prompt instruction text though. I'm using gpt-4o-mini it's like 30-100k tokens ($0.005-0.015 per API call). And are there any known tricks for having the LLM iterate over long lists of say 100+ items - i.e. keeping the attention mechanism from truncating?

hmm have you tried perhaps running the same image with various schemas? and then creating some sort of heuristic programatically to choose the best one?

Thanks for the response @User. I spent hours breaking down my BAML file. granularly because there are only a finite number of receipt formats I need to account for. So now I have A function that can classify which receipt format it is. And then based on a given format, it will route to a particular BAML schema For that given format. So this has got my accuracy much higher. The remaining problem is for the super long receipts The dumber models like GPT-4o mini We're not extracting every line item. Opus called it 'lazy' so I finally got the best bang for my buck using Gemini 3 Flash Preview it's $0.50 in / $3.00 out /mm tokens. My alternative to beef it up and more deterministically extract every line item for the very long receipts will be to preprocess each receipt image using OCR like tesseract, then in the LLM prompt using BAML submit both the receipt image + raw OCR output and say 'here's a receipt and corresponding OCR output - you act as corrector and fix the garbled OCR as applicable'. Because OCR is just processing algorithmically so it won't be 'lazy' and skip line items. And the OCR acts as a way to prime the attention mechanism of the LLM. But it's a mixed bag because if you have it act as 'error corrector' now you're sending the LLM bad signal (it's like telling someone not to think of an elephant) so I'll have to play around with it. But so far gemini 3 flash is doing well.

Yep the elephant problem can definitely show up. Every part of context matters. Thats great to hear though! Definitely worth spending time dealing with finite set of receipts