I have a set of questions, each modeled as a Task within a TaskGroup. After completing 4–5 tasks, the agent asks a question that isn’t included in either the agent’s system instructions or the task definitions. What might be causing this?

Thanks for reaching out, Suman. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I marked the taskgroup summarization as false. What else can i try ? It always hallucinates a question that is not listed as a task.

Thanks — I hear you: you disabled summarization and the agent still generates an unexpected question. That’s frustrating. Let’s walk through the next, targeted steps to find and stop the hallucination. Quick checklist — try these next • Lock down agent/system instructions ◦ Make the agent’s system prompt explicit and minimal. Remove any broad goal statements such as “ensure the user’s intent is satisfied” that could cause the model to invent follow-ups. ◦ If you have multiple instruction layers (system, assistant, task descriptions), ensure they don’t conflict or leave an open-ended objective. • Make task exit criteria explicit ◦ For each task, add a concise completion condition in the task description (for example: “Task complete when X data fields are returned and user confirms Y”). ◦ Add an explicit “do not ask additional questions beyond this task” clause where you want strictly no follow-ups. • Tighten model temperature and decoding settings ◦ Lower the model temperature (e.g., to 0.0–0.3) to reduce creative/unexpected outputs. ◦ Use deterministic decoding where possible (top_p, beam settings) to reduce hallucinations. • Remove or gate general-purpose clarification tools ◦ Ensure no broadly-named “clarify” or “ask_user” tools are available to the model inside the TaskGroup. ◦ If you need a clarification tool, add strict guardrails and explicit invocation conditions so it’s only callable when a specific flag is present. • Constrain the allowed actions after task completion ◦ After a task is marked completed, explicitly set the agent’s next possible actions to either: 1) proceed to the next task id, or 2) stop and return the final summary. Prevent free-form messaging if you need linear behavior. • Add an internal “allowed prompts” list ◦ Provide the model an explicit list of allowed question templates or allowed next-turn intents. If the model’s next output is not one of those templates, treat it as invalid and retry/generate again with stronger constraints. How to diagnose what’s triggering the question 1. Enable detailed logging for the turn that produced the hallucination: ◦ Log the exact input prompt the model received (system + conversation + task context + tools). ◦ Log any tool calls, model metadata, and the model’s full raw output. 1. Reproduce with a minimal test case: ◦ Strip to the smallest TaskGroup with 2–3 tasks and no external tools. See whether the hallucination still appears. ◦ If it disappears, incrementally add back instructions/tools until it reappears to isolate the trigger. 1. Inspect for implicit user signals and hidden context ◦ Check whether earlier user messages or task outputs contain ambiguous phrases like “anything else?” or “tell me more” that the model could interpret as permission to ask new questions. 1. Try forcing the next action ◦ After a task completes, programmatically insert a short assistant message that says “No further clarification required. Next action: proceed to task X” before the model is prompted again. If you want, I can give specific changes based on your configuration • Share (redact any secrets) the following and I’ll point to the exact lines to change: ◦ The agent/system prompt text ◦ One or two task descriptions (the ones before the hallucination occurs) ◦ Model settings (temperature, top_p, any repetition penalties) ◦ Any tools available to the agent in the TaskGroup Next step suggestion If you send those snippets, I’ll review them and give exact edits you can apply (phrasing for “do not ask follow-ups,” recommended temperature, and any tool gating) — that usually stops the model from inventing extra questions.

how to enable detailed logging ?

Thanks — you want step-by-step instructions to enable detailed logging so you can trace why the agent in your TaskGroup is asking an unexpected question. Below are practical options you can use depending on your environment, plus quick diagnostic steps to capture the exact turn that produces the hallucination. 1) Enable DEBUG-level logs when starting the agent • Python (production):

uv run agent.py start --log-level=DEBUG

• Python (dev):

uv run agent.py dev --log-level=DEBUG

• Node.js:

pnpm run start --log-level=debug

This increases verbosity for the agent runtime so you’ll see model prompts, tool calls, and internal events. 2) Use verbose test-time output for evals • For test runs with the evaluation framework:

LIVEKIT_EVALS_VERBOSE=1 uv run pytest -s -o log_cli=true <your-test-file>

This prints full RunResult events (LLM messages, function/tool calls, judgments) useful for replaying the exact flow. 3) Turn on Agent Observability (project/dashboard) • If enabled in your LiveKit project, Agent Observability provides: ◦ Conversation transcripts and turn-by-turn traces ◦ Timing and metadata for LLM, TTS, and tool calls ◦ Logs and traces you can inspect after a session completes Note: Observability requires a supported SDK version and that the feature is enabled in project settings. It shows session-level traces (not necessarily real-time). 4) Forward logs to an external monitoring service (recommended for production) • Configure your project to forward logs to Datadog, CloudWatch, Sentry, or similar. • This gives persistent searchable logs and easier correlation across runs (recommended over relying solely on short-term agent logs). 5) Add explicit debug logs in your code • Python example:

import logging
  logger = logging.getLogger("myAgent")
  logger.setLevel(logging.DEBUG)
  logger.debug(f"TaskGroup chat_ctx before prompt: {chat_ctx}")

• Node.js example:

import { log } from '@livekit/agents';
  log().debug('TaskGroup chat_ctx before prompt:', chat_ctx);

Log the exact prompt you send the model (system + task context + prior turns) and any tool inputs/outputs. Include identifiers like session ID, task id, and worker id. 6) Capture model input/output and tool calls for the problematic turn • Ensure logs include: ◦ Full assembled prompt passed to the model (system + assistant + task context) ◦ Any tool invocation and its input/response ◦ Model raw output (including tokens/metadata if available) • This will show whether the model invented the question, or if a tool call or earlier message triggered it. 7) Reproduce with a minimal test case and logging • Create a small TaskGroup with 2–3 tasks and no external tools. • Run with DEBUG logs. If hallucination disappears, reintroduce instructions/tools incrementally until it reappears to isolate the cause. Quick diagnostic checklist • Log the pre-prompt that led to the hallucinated question (most important). • Confirm no general-purpose “clarify” tool is available during the TaskGroup run. • Verify model settings (temperature, top_p); set temperature to 0.0–0.3 to reduce creative outputs during tests. If you want, share (redact secrets) the log excerpt for the turn where the hallucination appears: • The full prompt the model received • The model response that contains the unexpected question • Any tool call entries around that turn I’ll review the log lines and point to the exact trigger (missing context, tool invocation, or instruction wording) and suggest the precise change to stop those extra questions. Sources: Where to find your Agent Logs | LiveKit | Testing and evaluation | LiveKit Documentation | Streamline troubleshooting with Agent Observability | Job lifecycle | LiveKit Documentation

where do i set the temperature

Good question — you set the model

temperature

when you create the LLM used by your AgentSession (the LLM instance your agent uses to generate responses). Lower values make outputs more deterministic and reduce hallucinations. Examples: • Python

from livekit.agents import AgentSession
from livekit.plugins import openai

session = AgentSession(
    llm=openai.LLM(
        model="gpt-4o-mini",
        temperature=0.3,  # set here
    ),
    # other config...
)

• Node.js / TypeScript

import * as openai from '@livekit/agents-plugin-openai';

const session = new voice.AgentSession({
  llm: openai.LLM({
    model: "gpt-4o-mini",
    temperature: 0.3, // set here
  }),
  // other config...
});

Recommended values • 0.0–0.3: very deterministic — best for preventing unexpected questions. • 0.3–0.5: some flexibility but still controlled. • >0.7: more creative (not recommended when you need strict, linear behavior). Extra tips • Combine a low temperature with explicit, narrow system/agent instructions and strict task exit criteria to prevent the model from inventing follow-ups. • If you’re still seeing hallucinated questions after lowering temperature, enable DEBUG logs and capture the exact prompt the model received for the offending turn — I can help review it if you paste the redacted prompt and response. Sources: OpenAI LLM plugin guide | LiveKit Documentation | Tasks and task groups | LiveKit Documentation

An example, I don't have a diabetes question. However, I get "have you ever been diagnosed with diabetes for example" TaskGroupResult(task_results={'intro_task': True, 'hippa_task': HippaTaskResult(date_of_birth='1990-07-18', zip_code='12345'), 'asthma_task': False, 'copd_task': COPDTaskResult(copd=False), 'depression_task': DepressionTaskResult(depression=False), 'anxiety_task': AnxietyTaskResult(anxiety=False), 'cancer_task': CancerTaskResult(cancer=False), 'dementia_task': DementiaTaskResult(dementia=False), 'heart_disease_task': HeartDiseaseTaskResult(heart_disease=False)})

Thanks — I see the exact problem: after several completed tasks your TaskGroup produced an out-of-scope question (“have you ever been diagnosed with diabetes for example”) even though diabetes is not one of the defined tasks and your TaskGroupResult shows diabetes-related tasks aren’t present. I know that’s frustrating. Let’s walk through targeted causes and steps you can take to stop this specific hallucination. Why this likely happened • The LLM inferred a broader health-screening intent and probed a related condition (diabetes) even though it wasn’t a defined task. The model can generate follow-ups when it thinks a higher-level goal (complete a comprehensive health check) hasn’t been satisfied. • The hallucination can appear during task transitions, turn detection, or an implicit regression to a prior intent — the model may decide to ask one general catch-all health question if it believes coverage is incomplete. • A loose system/agent instruction like “ensure all relevant conditions are checked” or any ambiguous task descriptions gives the model license to ask extra medical questions. Targeted steps to stop it now 1. Capture the exact prompt that produced the diabetes question ◦ Enable DEBUG logging for the offending session and find the assembled prompt (system + conversation + task context) immediately before that model response. That will show which instruction or context caused the probe. If you want, paste the redacted prompt and response and I’ll point to the trigger. 1. Add an explicit “do not ask outside the task list” guard ◦ In the agent/system prompt and each task description add a short, explicit rule: ▪︎ Example phrasing: “Do not ask about conditions not listed as tasks in this TaskGroup. If a condition is not on the task list, do not ask follow-up questions about it.” ◦ Put this rule close to the top of the system prompt so it’s prioritized. 1. Make task completion and exit criteria strict ◦ For each task, include a clear completion sentence: “Task complete when [X fields] are filled. After completion, do not ask additional questions unless the user explicitly requests them.” ◦ After the final task, insert an explicit assistant turn: “All tasks complete. Do not ask additional screening questions. Return results.” This prevents the LLM from inventing a next turn. 1. Remove or gate general-purpose clarification helpers/tools ◦ Ensure there’s no broadly-named tool (e.g.,

clarify_intent

,

ask_user

) available during the TaskGroup run or add a guard that only allows it when a specific flag is present. 1. Lower temperature and force determinism for tests ◦ Set the LLM temperature to 0.0–0.3 for test runs to reduce creative probing. Reproduce the flow with temperature = 0.0 to see if the diabetes question still appears. 1. Constrain allowed next actions or allowed question templates ◦ Provide the LLM with an explicit list of allowable next-turn outputs (or templates). If the model’s candidate output is not on that list, reject/regen. This enforces linear behavior. 1. Reproduce with a minimal TaskGroup and log ◦ Create a tiny TaskGroup with only the tasks you expect. Run and inspect logs. If the hallucination disappears, incrementally reintroduce instructions/tools until it reappears — this isolates the offending instruction or tool. Quick diagnostic checklist (high priority) • Find the log entry for the turn that asked about diabetes. Look at the assembled prompt. That single step usually reveals whether the model inferred a broad goal or a tool was invoked. • Confirm system prompt wording doesn’t include global goals like “ensure the user’s intent is fully satisfied by probing all related conditions.” • Confirm no summarization reintroduced broader language (you already turned summarization off, which helps). If you want, I can do a focused review • Share (redact secrets) these items and I’ll point to the exact change to stop the diabetes question: ◦ The agent/system prompt ◦ The task descriptions for the tasks just before the hallucination ◦ The assembled prompt and model response from the debug log for that turn ◦ Current temperature and tool list Send those snippets and I’ll mark the exact phrase(s) that caused the model to probe and provide the minimal edits to eliminate the extra question. Sources: Tasks and task groups | LiveKit Documentation | How to Test Your Agent Using Another Agent | LiveKit | Prompting guide | LiveKit Documentation | Azure OpenAI LLM plugin guide | LiveKit Documentation

How to do 3 ?

You’re asking how to implement explicit task completion and exit criteria (step 3 from earlier) so the agent stops asking unrelated questions after tasks complete. Below are concise, practical steps with code examples and exact phrasing you can drop into your tasks and agent logic. 1) Make task instructions extremely explicit • In every task’s

instructions

, state the single allowed purpose, required fields, and a strict “do not ask anything else” rule. • Example phrasing to include near the top of each task: ◦ “Your ONLY job is to [collect X / ask Y].” ◦ “Ask at most once (or at most twice) for missing information.” ◦ “If the user refuses, record refusal and STOP.” ◦ “DO NOT ask about conditions or topics not listed as tasks in this TaskGroup.” Python example (task definition):

class CollectDOBTask(AgentTask[DOBResult]):
    def __init__(self):
        super().__init__(
            instructions="""
            Task: Collect the user's date of birth (MM-DD-YYYY).

            - Ask exactly: 'Please provide your date of birth (MM-DD-YYYY).'
            - If user gives a valid DOB, call set_dob(date) and STOP.
            - If user provides incomplete DOB, ask once for the missing part and STOP.
            - If user refuses or says 'I don't want to provide', call set_dob('REFUSED') and STOP.
            - DO NOT ask about any other health conditions or additional screening questions.
            """,
        )

2) Immediately complete the task in your tool/handler • When the required data or refusal is received, call

self.complete(result)

(or your framework’s equivalent) right away so control returns to the TaskGroup and the model won’t continue probing. Python example (tool that saves and completes):

@function_tool()
async def set_dob(self, ctx: RunContext, date_of_birth: str):
    # Save whatever persistence you need...
    self.complete(DOBResult(date_of_birth=date_of_birth))
    # Do not send any extra assistant messages here.

3) Use strict

description

text when adding tasks to the TaskGroup • The

description

is visible to the LLM as meta-guidance. Make it explicit about non-regression. • Example:

task_group.add(
    lambda: CollectDOBTask(),
    id="dob",
    description="Collect DOB only. After DOB is set (or refused), DO NOT return to this task or ask unrelated questions."
)

4) After a task completes, programmatically control the next turn • Don’t rely on the model to decide whether to ask more questions. Explicitly instruct the agent what to do next (e.g., proceed to next task or finish). • Example after awaiting TaskGroup:

results = await task_group
# Immediately produce a bounded reply that does NOT invite extra screening
await session.generate_reply(instructions="All required screening tasks complete. Do not ask additional health questions.")

5) Add parent-agent guardrails in the system prompt • Top-level rules prevent the model from inventing topics across the whole session. • Example lines to add to the system prompt: ◦ “Never ask about topics not defined as tasks for this session.” ◦ “After all tasks are completed, do not ask follow-up screening questions unless explicitly requested by the user.” 6) Combine with runtime controls • Set model temperature low (0.0–0.3) during tests to reduce creativity. • Remove or gate any broad “clarify” tools during this TaskGroup run. 7) Test with logging and minimal reproduction • Run a minimal TaskGroup (2–3 tasks) with DEBUG logging enabled and temperature = 0.0 to confirm no extraneous questions appear. • If an unexpected question still appears, capture the assembled prompt that produced it and inspect which instruction line or tool call preceded it. Checklist summary (short): • Add explicit “DO NOT ask X” lines in each task’s instructions. • Call

self.complete(...)

immediately when done. • Use strict

description

when adding tasks to the group. • Make the parent agent explicitly forbid extra questions after completion. • Lower temperature and disable broad clarification tools for testing. If you want, paste (redact any secrets) one task’s current

instructions

and the surrounding system prompt and I’ll rewrite each to the minimal, strict wording that stops extra probes (I can also point out exactly where to call

self.complete()

in your code). Sources: Prompting guide | LiveKit Documentation | Tasks and task groups | LiveKit Documentation | Survey Calling Agent | LiveKit Documentation

It looks like its happening between two tasks transitions

Thanks — that’s useful to know. If the hallucinated question appears specifically during task transitions, the most effective fixes focus on how you handle the handoff between tasks and what the model sees at that boundary. Below are targeted, step-by-step actions you can take (with code examples and exact phrasing) to stop the agent from inventing questions between tasks. 1) Ensure tasks complete immediately and silently at transition • When a task finishes, call the task-completion API (

self.complete(...)

or equivalent) as soon as you have the result, and avoid sending any extra assistant message from inside the task. • This prevents the LLM from remaining “active” in task scope and inventing follow-ups. Example (Python tool that completes):

@function_tool()
async def set_task_result(self, ctx: RunContext, result_value: str):
    # persist result...
    self.complete(TaskResult(value=result_value))
    # do NOT generate any further assistant messages here

2) Insert an explicit deterministic transition message • Right after a task completes and control returns to your agent code, programmatically inject a short assistant-turn that instructs the model what to do next (and what not to do). This replaces an open-ended model continuation with a constrained, explicit instruction. Example:

# After await task_group or after the specific task completes:
await session.generate_reply(
    instructions="Task X is complete. DO NOT ask any additional screening or unrelated questions. Proceed to Task Y."
)

3) Add explicit “no new topics” guard in both task and transition prompts • In each task's

instructions

and in the transition message, include a line like: ◦ “Do not ask about conditions or topics that are not listed as tasks in this TaskGroup.” ◦ “If the next task is defined, proceed directly — do not ask additional screening questions.” 4) Use TaskGroup

description

to discourage re-entry • When adding tasks, set

description

text that tells the LLM not to regress:

task_group.add(lambda: SomeTask(), id="some", description="Collect X only. After completion do not return or ask unrelated questions.")

5) Gate or remove general clarification tools during transitions • If you expose a general-purpose clarification tool or allow open tool invocation across tasks, temporarily remove it or add a guard so it is only callable when a specific explicit flag exists. Broad clarification tools are a common source of unexpected questions. 6) Force deterministic model behavior at the boundary • For the model call that occurs at task handoff, set temperature to 0.0 (or very low) so it produces deterministic, directive-following text instead of inventing probes. • If you can set model parameters per-call, apply low temperature for the transition prompt only. 7) Shorten and prioritize system-level guardrails • Put the “do not ask outside the task list” rule at the very top of the system prompt so it has priority when the model composes responses during transitions. 8) Log the exact handoff prompt to debug remaining cases • Capture and inspect the exact assembled prompt at the moment the hallucination occurs (system + conversation + task context). If you still see the diabetes question after applying the above, paste the redacted pre-prompt and model response and I’ll point to the exact offending line. Minimal reproduction strategy (to isolate cause) 1. Create a tiny TaskGroup with two tasks (A -> B). 2. In both tasks, add strict “ask only X; do not ask anything else” instructions and immediate

self.complete()

in the tool. 3. Run with temperature = 0.0 and DEBUG logging. 4. If the hallucination still appears, inspect the exact handoff prompt; that will reveal whether the model inferred a broader goal or a tool/regression was triggered. If you’d like, share these three redacted items for the transition where the diabetes question occurs: • The top of your system prompt (first ~6–10 lines) • The two task

instructions

and

description

fields (task before and after transition) • The assembled prompt and model response from the debug log for that transition I’ll point to the exact phrase or missing guard that’s causing the model to probe and provide the minimal edit to prevent it.