I want to make a translation flow, the llm outputs in english, get's translated and sent to tts and transcription sent to user, the user speaks in a different language but the agent replies in english only and gets translated. I'm using llm_node to do that but after few iteration the llms starts giving output in the other language itself. I'm guessing this is happening due to the entire chat ctx being in the other language, i'm reinforcing the rule of giving output in english at every turn but it's not working. One of the ways i can think of is figure out a way to make the chat ctx be in english itself for assistant message. Any suggestion?

Thanks for reaching out, Dev. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

currently i'm adding reinforcement instruction in user turn completed hook, and translating in llm node because if i translate in tts node the transcription shown to the user is in english itself, how should i do it in a way that transcription shown to the user is in other language but chat ctx is in english?

Thanks — this is a good follow-up. You said you currently: • add reinforcement instructions in

user_turn_completed

, and • translate in

llm_node

because if you translate in

tts_node

the transcription shown to the user becomes English. Goal recap: keep the chat context the LLM sees entirely in English, but show the user the transcription in their original language. Doable — translate at the STT step and keep the original transcription for display. Recommended approach (concise plan) 1. Translate inside

stt_node

(or at STT time) ◦ Have STT produce the original-language transcript for frontend display. ◦ Immediately produce a translated-to-English copy and hand that English text to the chat context/LLM. ◦ Do not pass the original-language text into the chat context. 1. Keep transcription shown to the user in the original language ◦ Use the original STT transcript (pre-translation) for the transcription UI or for

transcription_node

. ◦ Only replace the LLM-visible message with the translated English variant. 1. Continue reinforcing English with a system message each turn ◦ Re-inject or refresh a short system instruction each turn enforcing “Respond only in English.” ◦ This plus English-only chat context prevents drift. Why this order works •

stt_node

controls what the LLM sees; translating there guarantees the chat history stays English. •

transcription_node

(or your frontend) can still display the original-language transcript to the user, so UX is unchanged. • Translating in

llm_node

lets non-English text slip into the context before translation; that causes the drift you’re seeing. Practical implementation sketch • In `stt_node`: a. Get STT result (original_text). b. Send original_text to frontend/transcription stream. c. Translate original_text -> english_text. d. Create an STT event (or message) whose

.text

that the LLM will see is english_text. e. Yield that event into the pipeline so the LLM receives English. Example pseudocode:

async def stt_node(self, audio, model_settings):
    async for event in default_stt(audio, model_settings):
        original_text = event.alternatives[0].text  # e.g., Spanish
        # 1) Send original to frontend/display (preserve it)
        send_to_frontend_transcription(original_text)

        # 2) Translate for LLM
        english_text = await translate_to_english(original_text)

        # 3) Replace event content for the LLM branch ONLY
        event_for_llm = copy(event)
        event_for_llm.alternatives[0].text = english_text

        yield event_for_llm  # LLM sees English

Notes and safeguards • Keep a mapping/metadata linking the displayed original_text to the english_text if you need to correlate UI items with LLM turns. • After the LLM responds in English, store only that English assistant message in the chat context (do not replace it with a translated-to-other-language version). • Post-check: after generation, verify assistant output is English. If not, either re-run with a stronger system instruction or translate the output back to English and store the corrected English version in context. • If you cannot modify

stt_node

directly, you can instead: ◦ Intercept STT output before adding it to chat context, store original for display, and add only the translated text to chat context. ◦ But modifying

stt_node

is cleaner. Answering your exact constraint (why translating in

tts_node

caused wrong transcription) • Translating in

tts_node

changes what you present as the transcription stream (because TTS sits after the LLM output path). That’s why the transcription showed English. Translating earlier (STT) lets you show original language while sending English to the model. Quick questions so I can give code tailored to your setup 1. Can you modify

stt_node

directly in your pipeline? (Yes/No) 2. Which STT provider are you using (native STT, Gladia, etc.)? Some providers return both original and translated text which simplifies this. 3. Are you storing chat history centrally or streaming events to the LLM each turn? (Helps me show the exact place to replace messages.) If you say “Yes” to 1, I’ll provide a ready-to-drop-in

stt_node

snippet for your runtime. If not, tell me your provider and how you currently wire STT -> LLM -> transcription UI and I’ll give the next-best integration approach. Sources: Pipeline nodes and hooks | LiveKit Documentation | Text and transcriptions | LiveKit Documentation | Gladia STT plugin guide | LiveKit Documentation | TTS Translator with Gladia STT | LiveKit Documentation, with interactive elements

i don't want to convert user speech to english, i want the llm output to be in english which it's currently doing but after translation the user should hear and see the other language but the chat ctx should be in english itself for assistant

Thanks — that clarification helps. You want the LLM to always generate English, show the user the assistant reply translated into their language (both TTS audio and visible text), and keep the chat context stored in English so the LLM doesn’t drift. That’s a good design and fully achievable by separating (a) the LLM-visible chat context and (b) the user-facing transcripts/TTS outputs. Here’s a clear architecture and implementation plan you can apply. High-level flow (what to keep separate) • LLM-visible chat context: always English (what you store and send to the LLM). • User-facing transcript/display: original language for user speech; translated language for assistant replies. • TTS: receives the translated assistant reply (user’s language) so the user hears the localized audio. Where to do each step 1. STT (user speech) ◦ Produce original-language transcription (for display to user). ◦ Optionally produce a translated English copy if you want the LLM to understand the content in English. But from your latest message you do not want to replace the displayed transcript with English — keep original transcript visible. 1. LLM input ◦ Always feed the LLM English text. ◦ If the user spoke in non-English, provide a translated-to-English version to the LLM (but do NOT display that to the user). ◦ If the user wrote in English already, just pass it through. 1. LLM output ◦ Ensure the LLM replies in English (reinforce via system prompt and by keeping only English in the chat context). ◦ Store only the English assistant message in the chat context. 1. Translation for user-facing output ◦ Translate the English assistant message into the user’s language for display and TTS. ◦ Show the translated text to the user and send it to TTS to generate audio. Key implementation details and best practices • Keep two parallel streams/records: ◦ Chat context (English-only): used by LLM, persisted to history. ◦ UI/transcription stream (user-language for speech; translated assistant text for replies): shown to user and used for TTS. • Do NOT replace the English assistant message in the chat context with the translated version. Always store the English original as the assistant message. • Reinforce English generation with a short system message each turn, and/or a prompt anchor with examples, to reduce instruction forgetting. • Post-generation check: after the LLM generates a reply, confirm it’s English. If it accidentally outputs non-English, either: ◦ Re-run the LLM with a stronger system message, or ◦ Translate that output back to English and store the corrected English in chat context (but prefer re-call to avoid losing fidelity). Suggested pipeline step-by-step 1. STT node: ◦ Get original_text (user language). Send original_text to frontend transcription display. ◦ Translate original_text -> english_user_text (if user speech is non-English). ◦ Send english_user_text to the LLM input pipeline (but not to the frontend). 1. LLM node: ◦ LLM receives chat context where all assistant messages are English and the latest user message is english_user_text. ◦ Ensure system prompt enforces "always respond in English". ◦ LLM generates english_assistant_text. 1. After LLM: ◦ Persist english_assistant_text to chat context (only English). ◦ Translate english_assistant_text -> translated_assistant_text (user language). ◦ Send translated_assistant_text to TTS node and to frontend display for the assistant reply. ◦ Additionally, optionally display english_assistant_text in a developer/debug view if you want the user to see LLM original too. Why this prevents drift • The LLM never sees translated (non-English) assistant messages; all assistant messages in chat history remain English. Only the user-facing layer presents translations, so the model’s conditioning remains English-centric. Edge cases and extra notes • If conversation length is large, periodically summarize older content into English-only summaries to keep context size reasonable while preserving coherence. • If you cannot modify the STT node directly, intercept STT output before adding to chat context: keep original for UI + add translated English to context. • If you’re using an STT provider that returns both original and translated text (e.g., Gladia with in-flight translation), use the translated field for LLM input and original for UI. Small checklist you can follow now • [ ] Ensure your pipeline stores assistant messages only in English. • [ ] Translate user speech to English only for feeding LLM, but display original speech text to the user. • [ ] Translate LLM English reply into user language for display and TTS, but do not replace English in context. • [ ] Re-insert/refresh a system message each turn: “Always reply in English.” • [ ] Post-check generated assistant language and correct if needed. If you want, I can provide a compact code snippet for your exact hook (stt_node + llm_node + tts_node) once you tell me: 1) Can you modify the

stt_node

? (Yes/No) 2) Which STT provider are you using? (native, Gladia, other) 3) Do you currently store the assistant message in the chat context immediately after LLM generation, or do you mutate it later? (store immediately / mutate later) Tell me those three answers and I’ll give a targeted snippet you can drop into your pipeline. , with interactive elements

i'm using sarvam for translation, provide a short code for the pipeline

Thanks — good info. Below is a short, practical pipeline snippet that uses Sarvam for translations. It keeps: • chat context (what the LLM sees) in English, • user transcription shown in the original language, • assistant reply stored in English but translated to the user’s language for display and TTS. This is pseudocode/Python-style and focused on the three nodes:

stt_node

,

llm_node

(minimal), and the post-LLM translation + TTS step. Adapt variable names and integrations to your codebase.

# Helper: Sarvam translate function (awaitable)
async def sarvam_translate(text: str, target_lang: str) -> str:
    # replace with your Sarvam call
    return await sarvam_client.translate(text, target=target_lang)

# 1) STT node: send original transcript to frontend, feed English to LLM
async def stt_node(audio_stream, model_settings):
    async for stt_event in default_stt(audio_stream, model_settings):
        original_text = stt_event.alternatives[0].text  # user language
        user_lang = detect_language(original_text)  # optional, or pass in session

        # 1a. Send original_text to frontend transcription UI
        send_to_frontend_transcription(original_text, lang=user_lang)

        # 1b. Produce English text for LLM input (if user language != en)
        if user_lang != "en":
            english_user_text = await sarvam_translate(original_text, target_lang="en")
        else:
            english_user_text = original_text

        # 1c. Yield an STT event whose text is the English text so the LLM sees English
        event_for_llm = copy_of(stt_event)
        event_for_llm.alternatives[0].text = english_user_text
        yield event_for_llm

# 2) LLM node: ensure system prompt enforces English and store only English assistant output
async def llm_node(chat_ctx, tools, model_settings):
    # Ensure a short system message exists each turn
    enforce_system = {
        "role": "system",
        "content": "Always respond only in English. Do not output any other language."
    }
    # Prepend or update system message for the model call
    chat_ctx.prepend_system_message(enforce_system)

    # Call default LLM processing (streamed or not)
    async for chunk in Agent.default.llm_node(chat_ctx, tools, model_settings):
        yield chunk

    # After generation: assume final assistant_text is available
    assistant_english_text = chat_ctx.latest_assistant_text  # or capture from result

    # Persist only the English assistant message into chat context
    chat_ctx.add_message(role="assistant", content=[assistant_english_text])

    # 3) Translate assistant reply for user-facing output and send to TTS/frontend
    user_lang = chat_ctx.session.user_language  # obtain from session or detection
    if user_lang != "en":
        translated_for_user = await sarvam_translate(assistant_english_text, target_lang=user_lang)
    else:
        translated_for_user = assistant_english_text

    # Send translated text to frontend (visible) and to TTS node
    send_to_frontend_assistant_reply(translated_for_user, lang=user_lang)
    send_to_tts_node(translated_for_user, lang=user_lang)

Quick notes and tips • Do NOT replace the assistant English text in

chat_ctx

with the translated version. Always store the English assistant message. • Keep a mapping (metadata) linking the stored English assistant message to the translated text shown to the user if you need to correlate events. • If your STT already returns language metadata or both original+translated fields, use original for display and translated-for-LLM for chat ctx. • Add a post-generation language check: if the LLM ever outputs non-English, re-call the model with a stronger system message or translate its output back to English and store that corrected English in context. If you want, I can convert this to a drop-in snippet for your exact framework (show me the function signatures you currently use for

stt_node

,

llm_node

, and your TTS sender). , with interactive elements