My current problem: I have a group of messages I need filtered based on specific criteria outlined in my prompt. My prompt is hypothetically 1000 tokens long. Each message may only be 10-100 tokens. Instead of sending 1000 tokens of prompt for maybe only 10 tokens of actual content, I would like to send multiple messages in at once, especially since I may have a few hundred messages at a time, and the latency of waiting for each request would add up. My current solution: I changed the function signature from a single Message object to a list of Message objects. Each Message object has an "id" field that is passed along with the message text. The response object is a list of Responses with an "id" field to match the input. This way I can pass 20-50 messages at a time per prompt. The responses are filtered out based on the boolean criteria, then joined back to the original data on the id field. A few questions: Is there some caching that occurs on the model provider's end that makes this unnecessary? Is there a more consistent way of doing this? The model seems to sometimes use the wrong id in the response. Is there a good way for parallelization of the requests through baml? I would like to send each chunk of messages to the model one after another without needing to wait for the response of the first to send the second. I can do so in the coding language i'm using (python), but I don't know if there is a way to do so through baml.

1) what model do you use? Caching usually helps get rid of this kind of cost/latency issues, however you will get better data if you reduce the amount of context / messages for the LLM to process. 2) There isn't a way to parallelize within baml itself, you will have to do it like this https://docs.boundaryml.com/guide/baml-basics/concurrent-calls

I was using gpt-5-nano, but still have the incorrect id issue using gpt-5-mini.

if you want the LLM to literally fill out each ID you want, without misses, you may want to use dynamic types to build this kind of map as the output type at runtime: { "id1": { ...} "id2": {...} } https://docs.boundaryml.com/guide/baml-advanced/dynamic-types

Does the prompt end up getting cached if it is the same across multiple message requests?

yes, OpenAI does automatic caching. Claude doesn't IIRC. But if the other msgs in the context are not relevant to answering the question / outputting what you want, don't add them in

My response object contains the id, the boolean filtering field, and a reasoning field, the latter to help inform what needs changing in the prompt. I will have reasoning which don't correspond to the correct message id, which makes me think it is giving the correct reasoning, but the incorrect id.

so like it fills out:

{
   id: 123
   filter: false
   reason: the id456 is blahblahblah
}

class Message {
id int @description("Unique identifier for the message")
text string @description("The message to evaluate")
}
class MessageResponse {
id int @description("Unique identifier for the message")
filter bool @description("Whether the message meets the specified criteria")
reason string @description("The reason for the classification")
}

function Filter(messages: Message[]) -> MessageResponse[] {
...
}

I could be more explicit in my descriptions. I'm taking a python dataframe, selecting the relevant columns (id, message), converting it to a dict, batching them into N groups, and passing those into the function. Grabbing all the messages with (filter == True), combining them into a dataframe, and finally joining the original dataframe and the filtered results together.

it's not used, but you can add it to the options:

client MyClient {
   options {
      promptCacheKey "hi"
   }
}