My LLM responses seem to be more hallucinating in a self-hosted environment. Why is that happening?

Thanks for reaching out, Varun. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

How do I make these changes for a groq llm?

Thanks — I’ll walk you through the concrete changes to reduce hallucinations for a Groq LLM in your self-hosted setup, and then I’ll ask a couple quick questions so I can tailor any follow-ups. What to change (recommended settings) • Lower temperature ◦ Set

temperature

between 0.1–0.5 (start at 0.3). Lower values make responses more deterministic and reduce inventing facts. • Increase max output tokens if needed ◦ Ensure

max_output_tokens

(or

max_completion_tokens

) is high enough so the model can complete reasoning. Too-low limits can cause truncated or odd answers. • Force tool usage when external facts are required ◦ Set

tool_choice

to

"required"

if you use retrieval, DB lookups, or search tools. This prevents the model from guessing when data exists in a tool. • Disable parallel tool calls unless necessary ◦ Parallel reasoning can sometimes increase hallucination risk. Use sequential tool calls when you need deterministic, stepwise grounding. • Tighten stop tokens and system prompts ◦ Use clear system instructions that require the model to say “I don’t know” or call a tool when uncertain. Explicit refusal instructions reduce fabrication. • Improve context management ◦ Prune conversation history by relevance (keep essential facts and recent turns) rather than bluntly trimming tokens. Preserve user instructions and any grounding facts the model needs. Example configuration snippets • Python (LiveKit AgentSession style)

from livekit.plugins import groq

session = AgentSession(
    llm=groq.LLM(
        model="your-groq-model",
        temperature=0.3,
        max_output_tokens=1024,   # adjust higher/lower based on your needs
        tool_choice="required",
        parallel_tool_calls=False
    ),
)

• TypeScript / JS (AgentSession style)

const session = new voice.AgentSession({
    llm: groq.LLM.withGroq({
        model: "your-groq-model",
        temperature: 0.3,
        max_output_tokens: 1024,
        tool_choice: "required",
        parallel_tool_calls: false
    }),
});

Operational recommendations • Design tools to return complete, ready-to-use answers so the LLM doesn’t need to synthesize facts. • Add short filler speech like “Let me check that” when calling tools so the agent flows naturally while grounding answers. • Consider a verification step: run responses through a lightweight verifier or moderation LLM before presenting to the user if hallucinations persist. • Monitor logs for token-limit or timeout errors (e.g., MAX_TOKENS) and for high latency between your agents and the Groq model — latency and resource constraints can indirectly degrade output quality. Quick questions to make this more specific • Which Groq model name/version are you using? • What are your current

temperature

,

max_output_tokens

, and

tool_choice

settings? • How are you sending conversation history to the model (full transcript, last N turns, token-based pruning)? • Are your agents hosted in the same region as your Groq model endpoint, and what are typical round-trip latencies? Share those details and I’ll give exact values and a step-by-step plan tailored to your deployment. Sources: Groq LLM plugin guide | LiveKit Documentation | The ReAct Pattern for Voice Agents and How AI Agents Think, Act, and Respon... | LLM-Powered Content Filter | LiveKit Documentation, with interactive elements

model and temperature: "model": "openai/gpt-oss-120b", "temperature": 0.2 this is the only configuration we are using

Thanks — that’s helpful. You’re using

openai/gpt-oss-120b

with

temperature: 0.2

and nothing else configured. I’ll start by acknowledging that 0.2 is already low (good), then list targeted changes and checks that will most likely reduce hallucinations for this model in a self‑hosted setup, plus a few short follow-ups so I can narrow recommendations. Immediate changes and checks to try • Add or raise

max_output_tokens

◦ Ensure the model isn’t being cut off mid-reasoning. Set

max_output_tokens

to a value like 1024–2048 and test. If responses are truncated or odd, raise this further. • Set explicit

stop

tokens and clearer system prompts ◦ Add stop tokens or a firm system prompt that instructs the model to respond only when confident and to say “I don’t know” or call a tool when unsure. This reduces invented facts. • Enable tool usage / grounding ◦ If you have access to retrieval, DB lookups, or search tools, configure the LLM to prefer tools for factual queries (set

tool_choice

to

"required"

or the equivalent). Have tools return complete, factual answers the model can echo rather than synthesize from memory. • Check context-window handling ◦ Confirm how much history you send. Keep essential system instructions and the last few user/assistant turns plus any grounding facts. Avoid sending long raw transcripts that force early pruning of key facts. • Add a verification/response-filter step ◦ Run model outputs through a small verifier (a rule check or lightweight LLM classifier) to detect likely hallucinations before presenting them. • Monitor resource and latency metrics ◦ Verify your agent servers aren’t CPU/RAM constrained and that latency to the model endpoint is low. Resource starvation or high RTTs can surface as inconsistent or incomplete outputs. Suggested example configuration to start with

{
  "model": "openai/gpt-oss-120b",
  "temperature": 0.2,
  "max_output_tokens": 1536,
  "tool_choice": "required",
  "parallel_tool_calls": false,
  "stop": [
    "\n\n"
  ]
}

Practical prompt/system rules to reduce fabrication • System prompt: “Answer only when supported by given facts. If you don’t know or cannot verify, reply ‘I don’t know’ and request a tool call.” • Encourage concise factual answers and avoid speculative language. Why these changes matter (brief) • Low temperature helps, but hallucinations often come from missing grounding data or truncated reasoning —

max_output_tokens

, tool grounding, and context management address that. • For large models, forcing tools and verifying outputs are frequently more effective than temperature tweaks alone. Quick follow-up questions so I can refine this • Do you currently set any

max_input_tokens

/ request token limits or see MAX_TOKENS errors in logs? • How do you send conversation history today (full transcript, last N messages, token-based trimming)? • Do you have a retrieval/search tool or database that the model can call for facts? If so, how are tools currently wired into the agent flow? • Where are your agents hosted relative to the model endpoint (same region or cross-region)? Any observed latency numbers? If you answer those, I’ll give a concise, step-by-step change plan you can apply in your deployment and tests to measure improvement. , with interactive elements

What are the default values that are set in livekit cloud?

Thanks — I’m glad you asked for the LiveKit Cloud defaults so you can compare. I’ll answer what’s known, point out what isn’t available, and ask one quick clarifying question so I can tailor any next steps. What LiveKit Cloud defaults are known • For OpenAI-compatible LLMs used through LiveKit plugins, the default model is

gpt-4.1

and the default

temperature

is

0.8

. • For the Groq plugin, the plugin defaults are

model = llama-3.3-70b-versatile

and

temperature = 1.0

. • These plugin defaults come from the respective LLM provider plugins and are the same whether you run in LiveKit Cloud or self-hosted environments. What is not available / not documented • LiveKit Cloud does not publish a single global list of every default parameter (for example, exact default values for

max_output_tokens

,

top_p

,

tool_choice

, or stop tokens applied by Cloud). Those values are set by the individual LLM plugins or by per-session configuration in your AgentSession code. • Any default overrides LiveKit Cloud might apply for hosted inference endpoints are not documented in the sources I can access. Quick clarifying question • Which plugin are you comparing against in LiveKit Cloud for this deployment — are you comparing your self-hosted

openai/gpt-oss-120b

to a LiveKit Cloud session configured with an OpenAI-like plugin or to a Groq/other plugin? (This helps me give the most relevant default comparisons and tuning advice.) If you want, I can: • Show side-by-side recommended configuration values to match LiveKit Cloud behavior for the plugin you name. • Provide a concrete change/test plan to reduce hallucinations for

openai/gpt-oss-120b

based on those defaults. Sources: Agents on LiveKit Cloud | Deploy voice AI on global infrastructure | LiveKi... | Amazon Bedrock LLM plugin guide | LiveKit Documentation | Anthropic Claude LLM plugin guide | LiveKit Documentation | Groq LLM plugin guide | LiveKit Documentation | Google Gemini LLM | LiveKit Documentation, with interactive elements

I understand all of this, but why doesn't the same prompt lead to hallucination in cloud while it does in a self hosted setup?

You’re right to be puzzled — same prompt, different behavior is frustrating. Below are the most likely reasons the exact same prompt would produce hallucinations in your self‑hosted environment but not in LiveKit Cloud, plus focused checks you can run to pinpoint the cause. Top reasons this can happen • Subtle configuration differences ◦ LiveKit Cloud or its plugin defaults (model, temperature, stop tokens, tool settings) can differ from your self-hosted settings. Even small differences (e.g., temperature 0.2 vs 0.3, implicit top_p, or different stop tokens) change generation behavior. • Different model versions or provider implementations ◦ Cloud may use a slightly different model build, patch, or provider implementation than your self‑hosted runtime. Those differences can affect hallucination propensity. • Missing grounding/tools or tool integration differences ◦ Cloud sessions may have retrieval, search, or tool integrations enabled or tuned (or require tool usage) that ground answers. If your self‑hosted flow doesn’t call the same tools or return the same tool responses, the model will guess. • Context management and what you actually send as input ◦ Cloud may be preserving or formatting the system prompt, instructions, or relevant facts differently. Your self‑hosted agent might be truncating or pruning critical context before sending the prompt, causing the model to lack necessary facts and invent them. • Token limits and truncated reasoning ◦ If your self‑hosted environment applies smaller

max_output_tokens

or hits

MAX_TOKENS

errors, the model can produce incomplete or odd answers that appear like hallucination. Cloud may allow larger token outputs. • Resource constraints and timing (CPU, memory, latency) ◦ Agent resource starvation or high RTTs between your agent and the model endpoint can cause retries, truncated requests, or timeouts that change outputs. Cloud hosting tends to provide consistent resources and regional proximity to model endpoints. • Guardrails and verification layers in Cloud ◦ LiveKit Cloud or the Cloud-configured pipelines may include verification, moderation, or post-processing steps that filter or modify hallucinated content before it reaches the user. Your self-hosted pipeline may not include these checks. Quick checks to run (fast diagnostics) • Compare the full request payloads the cloud session and your self-hosted session send to the LLM: model name, temperature, top_p, stop tokens,

max_output_tokens

, system prompt text, and conversation history length. • Check logs for MAX_TOKENS, timeout, or retry errors in your self-hosted setup. • Verify whether any retrieval/search/tools are called in Cloud for that prompt and whether those tools are available/used in your self-hosted flow. • Measure round‑trip latency from your self‑hosted agent to the model endpoint; compare to Cloud latency. • Temporarily replicate the Cloud environment values: match model version, temperature, stop tokens, and token limits exactly in your self‑hosted config and test the prompt again. If you want, I can give a short step‑by‑step checklist to compare the cloud and self‑hosted request/response flows. To make it precise, please share: • A redacted example of the exact request payload you send (system prompt + user prompt + config fields). • Any logs showing token/timeout errors or notable latencies. • Whether Cloud uses any retrieval/tool calls for that prompt (if you know). Give me one of those items and I’ll produce a concise, actionable diff of what to change in your self-hosted setup to match Cloud behavior. , with interactive elements