Hey @Vaibhav (BAML 🐑) , I am just catching up on your podcasts and had a conceptual question (btw really love the podcast, it helps a lot with understanding, that's why I am trying to catchup from the start. ). The question is mainly around should we do our own LLM input serialisation or let the LLM provider does it. - **we should do it**: You mentioned in this [podcast - Building a 12 Factor Agent - EP #4]() that we need to own our serialisation, and the example shows instead of building a chat history of [sys, user, assistant, tool, assistant, user...], we might be able to just do [sys, user] by shoving [user, assistant, tool, assistant, user...] into [user] using XML format for example. I completely understand the token saving and keeping context short and clean argument. - **LLM provider should do it**: But on the flip side, you also mentioned about the meaning of special token in a thinking model episode, where the thinking token like [reasoning] and [/reasoning] will have weight emphasis that allow the LLM to pay more attention, etc. For the same argument, wouldn't preserving the structure of [sys, user, assistant, tool] also important given the LLM provider like OpenAI will be able to map them to the special tokens that LLM can recognise as well? What's your take on this? Does the benefit of cutting token count generally outweigh the benefit of special tokens treatment in your exp?

oh yea 100% owning the tokens / context gets you the best bang for the buck imo.

Thanks @Vaibhav (BAML 🐑) , I 100% agree with the argument to own the loop + context (which why I dislike the response API that openAPI is pushing for) Apology I haven't made my question clear. I was meant to ask more around should we also own the serialisation part of the context transformation into text. - First option is to have the API provider do the serialisation part, and we provide the API a chat history like with the entire (or subset) conversations in the format of the *original chat message [sys, user, assistant, tool, assistant, user...] * -> For the benefit of special token attention - The second option is for us to do our own serialisation, by using token saving format like XML, so we have a maybe just * [sys, user], but the user message contain the history + the new query * -> For the benefit of input token minimisation For projects in my company, people do not do their own serialisation, nor do I (yet). I am just wondering has there been any study or benchmark run on owning the serialisation to cut token will outweight the benefit of the special token treatment. I know that BAML has done one for tool calling, but what about to the chat history context itself?

honestly as with all things LLM, sadly the answer is "It depends". I find that for certain tasks (data processing / transformation) skewing away from

system

user

helps, but if i'm building a chatbot, then sticking closer helps.

Haha yeh I rekon that would be the answer, was trying my luck in case you might have more benchmark like you have with the structured output. This makes completely sense, I would 100% agree that it would be task dependent, hence brings me to your next episode "evals, evals, evals" XD

of course! 🙂