https://livekit.io logo
Join Slack
Powered by
Runtime model doesn't follow instruction in genera...
# ask-ai
q
Runtime model doesn't follow instruction in generateReply(), why ?
t
Thanks for reaching out, Heorhi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m sorry you’re seeing that. I’m going to help diagnose why your runtime model isn’t following instructions in 
generateReply()
. A few quick clarifying questions so I can narrow this down: 1. Which runtime model/plugin are you using (e.g., OpenAI Realtime, another realtime model, or a non-realtime LLM)? 2. Are you calling 
generateReply()
with 
instructions
or with 
user_input
? Please paste the exact prompt or a minimal redacted example. 3. Is the model producing any output at all, or an unrelated/incorrect reply? Paste a short example of the model’s actual output and the expected output. 4. Do you have any chat history or prior context loaded before calling 
generateReply()
? If yes, roughly how large/long is that history? 5. Are you using speech/audio synthesis (agent speaks the reply) or text only? 6. Any error logs, warnings, or timing/latency issues you noticed? What I can check/suggest next (based on your answers): • If you’re using a realtime model: realtime models can sometimes deviate from scripted instructions (especially with long history or speech contexts). I can show ways to strengthen instruction-following once I know the model. • If instructions are being passed but not recorded: confirm you’re sending them in the 
instructions
parameter in the expected shape (I can show the exact payload once you share which SDK/version you’re using). • If the reply diverges due to prior chat history: I can suggest ways to reset context, prepend stronger system prompts, or truncate history. • If speech output is involved: there are extra failure modes for scripted audio — I can suggest an alternative flow (text-only generation then synthesize) to improve fidelity. Share the answers and a small code/payload sample and I’ll provide concrete fixes and an example payload you can apply immediately. Sources: Agent speech and audio | LiveKit docs | Realtime model plugins | LiveKit Docs
q
I call generateReply with instructions. i'm using gpt-realtime. I have no chat befor generateReply but have some context,
t
Thanks — that helps. You're using 
gpt-realtime
, calling 
generateReply()
with 
instructions
, and there’s no prior chat but you do have some context. Here’s why the model may not follow the instructions and what you can try next. Why this happens • Realtime models like 
gpt-realtime
can prioritize persistent system/context instructions or conversational behavior over one-off 
generateReply()
instructions. • 
instructions
passed to 
generateReply()
are transient and may be overridden by system-level instructions or context messages already loaded. • Realtime models have limited guarantees for strictly following scripted output (especially for speech); they may reinterpret or rephrase instructions. Concrete fixes to try • Use 
session.say()
for exact scripted output (bypasses LLM and produces deterministic speech). • Move your important instruction into the chat context as a system message so it takes precedence: a. Add a system message like: "For the next reply, respond in one short sentence." b. Then call 
generate_reply()
. • If you must keep instructions transient, ensure the model has no conflicting system/context directives (audit and simplify any context that says “always elaborate” etc.). • If speech fidelity is required, generate text with 
generateReply()
(or other text-only flow), confirm the text, then synthesize it separately — this reduces realtime model reinterpretation. Quick troubleshooting info I may need if this persists • The exact 
instructions
string you pass (redact any secrets). • A sample of the model’s actual output vs. expected output. • Whether your context includes any system-style rules and how you injected it. Tell me the 
instructions
text and a short example output and I’ll suggest the exact payload or a context update you can apply. Sources: Agent speech and audio | LiveKit docs | Realtime model plugins | LiveKit Docs | Pipeline nodes and hooks | LiveKit docs
a
i'd suggest you try passing system_instructions + generate reply instructions appended
t
Thanks — that’s a good suggestion. Appending transient 
generateReply()
instructions to system-level instructions can improve adherence if done correctly. Below is a concise, practical approach and an example you can try. Why this helps • System (system-level) instructions are persistent and take precedence. Appending the 
generateReply()
instruction to them ensures the model sees the directive as higher priority. • If you need exact output, prefer 
session.say()
or a two-step flow (generate text → synthesize). How to implement 1. Create or update a temporary system message that combines your persistent system rules with the one-off directive for the next reply. 2. Add that system message to the chat context (so it’s treated as higher priority). 3. Call 
generate_reply()
without relying only on its transient 
instructions
. Example (pseudo-code)
Copy code
# Build combined system instruction
combined_sys = (
    "You are a concise helpful assistant. "
    "For the next reply only, respond in one short sentence and do not add extra explanations."
)

# Inject into chat context as a system message
session.chat_ctx.add_message(role="system", content=combined_sys)

# Now call generate_reply (no need to pass the same instruction again)
await session.generate_reply(user_input="Explain OAuth briefly.")
Alternative (keep transient + reinforce) • Pass the same short instruction both as a system message and again in 
generate_reply(instructions=...)
to reinforce it. If you want, paste your actual system instruction and the one-off instruction you’d like appended; I’ll merge them and return the exact payload you can use.
2 Views
Open in Slack
PreviousNext
Powered by