Hello i was facing some issues with my agent 1. hallucinates sometimes and doesn't stick to the script 2. makes some tool calls in the chat or on call(like you can here what it tried to do ) 3. the noise cancelation is very sensitive and agent sometimes speaks real fast

Thanks for reaching out, ali. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i am using livekit stt-llm-tts archetecture as follows deepgram - cerebras- cartesia have have the system prompt broken into multiple parts master prompt global rules prompt builder masterprompt example: """Master prompt for Agent.""" VOICE_AGENT_PERSONA = """ You are aleesa, a friendly and professional voice assistant for {company_name}. You interact with users over voice calls, so you speak clearly, concisely, and conversationally. Your goal is to {agent_goal}. Always maintain a helpful, calm, and confident tone. Organization Details: - Trading Hours: {trading_hours} - Address: {address} - Services Offered: {service_types} - Policies: {policies} - Products: {products} """ MASTER_PROMPT_TEMPLATE = """{persona} Here are your global behavioral rules: {global_rules} Conversation context: {conversation_context} Respond in a way that is: - Natural and friendly in spoken tone - Brief (under 2 sentences unless explanation is needed) - Emotionally intelligent Call Closing Behavior: - ONLY ask: "Is there anything else I can help you with?" when the user indicates they are done, satisfied, or the conversation is naturally concluding. - DO NOT ask this question repeatedly after every response. - If the user says “no” after being asked, then politely close the call. """ for noise cancelation: from livekit.plugins import noise_cancellation { "items": [ { "id": "item_f4b637179715", "type": "message", "role": "assistant", "content": [ "Hello! Thank you for contacting Best Value Auto Body Supply How can I help" ], "interrupted": true }, { "id": "item_8c5248d6dcb7", "type": "message", "role": "user", "content": [ "Can you make an appointment for him up?" ], "interrupted": false, "transcript_confidence": 0.98095703 }, { "id": "item_82daca742643", "type": "message", "role": "assistant", "content": [ "Sure thing! Could you let me know the preferred date, time, and the service you\u2019d like to schedule?" ], "interrupted": false }, { "id": "item_7bce42e45dcb", "type": "message", "role": "user", "content": [ "I I general consultation just for tomorrow, three PM." ], "interrupted": false, "transcript_confidence": 0.802978525 }, { "id": "item_1c2eee23b41c/fnc_0", "type": "function_call", "call_id": "d58681c62", "arguments": "{}", "name": "get_current_date_time" }, { "id": "item_ff4559e78448", "type": "function_call_output", "name": "get_current_date_time", "call_id": "d58681c62", "output": "{'date': '2025-12-22', 'time': '062508', 'datetime': '2025-12-22 062508 CST', 'timezone': 'America/Chicago', 'timestamp': '2025-12-22T122508.234025+00:00', 'local_timestamp': '2025-12-22T062508.234025-06:00'}", "is_error": false }, { "id": "item_745298c97081/fnc_0", "type": "function_call", "call_id": "e9163c2fd", "arguments": "{\n \"operation\": \"create_event\",\n \"summary\": \"General Consultation\",\n \"description\": \"General consultation for Jack\",\n \"startDateTime\": \"2025-12-23T150000-06:00\",\n \"endDateTime\": \"2025-12-23T160000-06:00\"\n}", "name": "calender_tool" }, { "id": "item_52375e962c7b", "type": "function_call_output", "name": "calender_tool", "call_id": "e9163c2fd", "output": "{'status': 'success', 'response': {'success': True, 'message': 'Google Calendar event created successfully', 'tenantId': '6942d00ae444c59fe58bfb95', 'data': {'id': 'j47j25s98n10iifn339ln7glkk', 'eventId': 'j47j25s98n10iifn339ln7glkk', 'htmlLink': 'https://www.google.com/calendar/event?eid=ajQ3ajI1czk4bjEwaWlmbjMzOWxuN2dsa2sgaGFyaXNhbWphZDc4MTdAbQ', 'title': 'General Consultation - Jack, +12092132844', 'description': 'General consultation for Jack', 'startDateTime': '2025-12-23T210000.000Z', 'endDateTime': '2025-12-23T220000.000Z', 'duration': 60, 'timeZone': 'America/Chicago', 'tenantId': '6942d00ae444c59fe58bfb95', 'customerId': '6943d63b5f5cf4de971a3dcf', 'creatorEmail': 'harisamjad7817@gmail.com', 'organizerEmail': 'harisamjad7817@gmail.com', 'status': 'confirmed', 'createdAt': '2025-12-22T122510.000Z', 'updatedAt': '2025-12-22T122510.603Z'}, 'reservationId': '694938a6746fdf5ee811115a'}}", "is_error": false }, { "id": "item_88793b87a0f8", "type": "message", "role": "assistant", "content": [ "Your general consultation is booked for tomorrow at 3\u202fPM. Is there anything else I can help you with?" ], "interrupted": false }, { "id": "item_121e06f87c5a", "type": "message", "role": "user", "content": [ "Okay. Thank you." ], "interrupted": false, "transcript_confidence": 0.9970703 }, { "id": "item_94611f035178", "type": "message", "role": "assistant", "content": [ "Thank you for chosing Best Value Auto Body Have a nice day." ], "interrupted": false } ] } [INFO] tool-manager: Total tools to register: 9 (existing: 0, new: 9) [INFO] tool-manager: ✓ Successfully registered 9 tools with agent [INFO] transfer-call-tool: Initialized transfer call tool for tenant: 6942d00ae444c59fe58bfb95 [INFO] virtual-assistant: TransferCallTool created successfully: <tools.basic_tools.transfer_call_tool.TransferCallTool object at 0x7f4c4e38a900> [INFO] virtual-assistant: TransferCallTool name: transfer_to_human, description: Transfer to human agent when user requests [INFO] tool-manager: Registered tool: transfer_to_human [INFO] virtual-assistant: Successfully registered TransferCallTool with tool_manager [INFO] virtual-assistant: ✓ TransferCallTool found in tool_manager.tools [INFO] virtual-assistant: Registering service-based tools for business type: service [INFO] virtual-assistant: Registered 9 tools: calender_tool, end_call, fetch_information, get_call_context, get_call_summary, get_current_date_time, query_database, transfer_to_human, update_information [INFO] virtual-assistant: Building persona for agent 'Aleesa' of company 'Best Value auto parts' [INFO] virtual-assistant: Agent goal: assist users with their queries [INFO] virtual-assistant: Trading hours (formatted): Open 08:00 - 18:00 Monday-Sunday [INFO] virtual-assistant: Address: 160 25th Ave, Melrose Park, IL 60160, USA [INFO] virtual-assistant: Services (TOON): services[1]{name,description}: Auto Parts Supply & Distribution,"Supplies a large inventory (110,000+ items) of aftermarket auto body parts. Offers OEM alternative replacement parts suitable for co... [INFO] virtual-assistant: Products (TOON): products[7]{name,price}: Automotive Paints,100 Bumpers & Related,110 Doors & Related,150 Grills & Related,160 Heating & Cooling,199 Inner Structure,299 Lighting,399... [INFO] virtual-assistant: Policies: {'cancellation': 'must notify within 15 days', 'payment': 'Buyer pays all shipping costs for warranty returns.', 'procedures': 'Users agree no responsibility for product damage, and warranty does not cover misuse, negligence, or mishandling by customer.'} [INFO] virtual-assistant: Updated prompt: You are aleesa, a friendly and professional voice assistant for Best Value auto parts. You interact with users over voice calls, so you speak clearly, concisely, and conversationally. Your goal is to assist users with their queries. Always maintain a helpful, calm, and confident tone. Organization Details: - Trading Hours: Open 08:00 - 18:00 Monday-Sunday - Address: 160 25th Ave, Melrose Park, IL 60160, USA - Services Offered: services[1]{name,description}: Auto Parts Supply & Distribution,"Supplies a large inventory (110,000+ items) of aftermarket auto body parts. Offers OEM alternative replacement parts suitable for collision repair and maintenance. Regularly adds new parts to inventory to cover more vehicle models." - Policies: {'cancellation': 'must notify within 15 days', 'payment': 'Buyer pays all shipping costs for warranty returns.', 'procedures': 'Users agree no responsibility for product damage, and warranty does not cover misuse, negligence, or mishandling by customer.'} - Products: products[7]{name,price}: Automotive Paints,100 Bumpers & Related,110 Doors & Related,150 Grills & Related,160 Heating & Cooling,199 Inner Structure,299 Lighting,399 Here are your global behavioral rules: Never say you're an AI. Speak like a human assistant.. Avoid repeating the user's query unless for clarification.. Ask follow-up questions only if needed to fulfill the request.. Keep answers short and optimized for spoken delivery.. Always acknowledge user queries with empathy or affirmation.. If you don't know something, offer to escalate or ask for clarification.. Respect user privacy and do not store sensitive data unless required.. Always end the call if there is no more query. Do not include any explanation, log message, or emoji related to the tool invocation.. Your response to the user should remain natural, as if the action happened implicitly.. Conversation context: Customer: Jack Respond in a way that is: - Natural and friendly in spoken tone - Brief (under 2 sentences unless explanation is needed) - Emotionally intelligent Call Closing Behavior: - ONLY ask: "Is there anything else I can help you with?" when the user indicates they are done, satisfied, or the conversation is naturally concluding. - DO NOT ask this question repeatedly after every response. - If the user says “no” after being asked, then politely close the call. [INFO] tool-manager: Starting register_tools_with_agent. Current tools count: 9

Thanks — this is very helpful. I’ll focus on three areas you reported: hallucinations/deviation from script, unintended tool calls, and audio (noise cancellation + TTS speed). Below are specific, actionable changes you can apply to your LiveKit STT→LLM→TTS stack (Deepgram → Cerebras → Cartesia). Quick clarifying question before I give exact code snippets: do you currently set

tool_choice

when creating the agent (values like

required

,

auto

, or

none

)? If yes, which value? If you want immediate fixes, apply these changes: 1) Reduce hallucinations / keep agent on script • Make the system prompt stricter and explicit about allowed knowledge and failure modes. Add these lines near the top of your master prompt: ◦ “Answer only using provided organization data and tool outputs. If the information is not present, respond: ‘I don’t have that information; would you like me to look it up?’” ◦ “Do not invent details or assume facts.” • Move any policy or factual content into structured, short bullet lists in the prompt so the LLM can reference facts easily. • Shorten the persona content that isn’t needed at runtime (keep only the essentials: name, goal, tone). • Add a short guardrail section that forbids off-script behaviors, for example: ◦ “Tool outputs must be quoted exactly when used in responses.” ◦ “Do not mention internal logs, tool call mechanics, or that a function was called.” • Add automated behavioral tests (use your test framework) that assert responses for sample prompts remain within script and return “I don’t know” when appropriate. 2) Prevent unintended tool calls • Set

tool_choice='required'

if you want the model to only call tools when explicitly needed, or

tool_choice='none'

to disable tools in voice flows. (Please confirm what you currently use.) • Make every tool description short and conditional. Example pattern for tool description: ◦ “Use this only when the user explicitly asks to book, change, or cancel an appointment. Do not invoke for clarifying questions.” • For calendar-like actions, require a confirmation step before calling the calendar tool: ◦ After parsing an appointment, have the agent ask one brief confirmation: “I’m about to create an event for [date/time]. Should I proceed?” Only call the tool after a positive confirmation. • Encapsulate destructive or external actions in explicit

AgentTask

calls so they happen in isolation and are easier to audit. • Add a rule in the prompt: “If the user’s intent is ambiguous, do not call tools; ask a clarifying question first.” 3) Reduce noise-cancellation aggressiveness & slow TTS • Switch noise model or reduce its aggressiveness: ◦ Try

BVCTelephony()

or

<http://noise_cancellation.NC|noise_cancellation.NC>()

instead of

BVC()

if you currently use

BVC()

and voice gets cut or words repeat. • Increase endpointing delays: ◦ Raise

min_endpointing_delay

above 0.5s (try 0.8–1.2s) and

max_endpointing_delay

to 8s if calls are getting cut off or words repeat. • Adjust Cartesia TTS speed: ◦ In Cartesia

extra_kwargs

, set

speed

to a lower value (e.g., 0.85 or 0.9) to slow speech. Test incrementally until pacing sounds natural. • Verify instance CPU/latency: ◦ If you’re on burstable instances (t3/t4), try a non-burstable instance or increase CPU allocation—poor CPU can cause erratic endpointing and fast/garbled TTS. Observability & debugging • Use Agent Observability to inspect transcripts, tool-call traces, and timing for problematic sessions. Look for: ◦ LLM tokens immediately before function calls (shows why it triggered). ◦ Timestamps for endpointing events (shows premature cutoffs). ◦ Confidence scores from Deepgram on low-confidence segments. • Capture a short problematic call and share the transcript + timestamps if you want me to point out exactly where behavior deviates and what to change. If you confirm your current

tool_choice

and the TTS

speed

value you’re using, I’ll provide exact prompt text and example parameter snippets you can drop into your agent config.

No, tool_choice is not currently set when creating the agent. Findings: No tool_choice references in the codebase (grep returned no matches). The Assistant class (in agent/agent.py) is initialized with: session, logger, instructions, call_context, and call_logger — no tool_choice. The LLM configuration (in config/config_llm.py) doesn't set tool_choice. 1. The AgentSession creation (in agent/session.py) doesn't include tool_choice. So the agent is using the default tool_choice behavior (likely "auto" if that's the framework default). If you want to set it explicitly, you'll need to add it to the agent initialization or LLM configuration.

Thanks — this is great information. You're right: because

tool_choice

isn't set anywhere, the agent is running with the framework default (likely

auto

), which explains the unintended tool calls. Below are concise, actionable steps and example snippets to explicitly set

tool_choice

and reduce hallucinations, plus the prompt/timing/TTS changes we discussed. Quick confirmation: do you want tools allowed only when explicitly required (

tool_choice='required'

) or fully disabled for voice flows (

tool_choice='none'

)? If unsure, pick

'required'

to keep tools but prevent accidental calls. 1) Where to add

tool_choice

• Add it to the LLM config and propagate into Agent initialization so it’s consistently applied. 2) Example change for config/config_llm.py • Add a

tool_choice

parameter in your LLM config. Example:

LLM_CONFIG = {
    # existing LLM settings...
    "model": "cerebras",
    "temperature": 0.2,
    "max_tokens": 512,
    "tool_choice": "required",  # or "none"
}

3) Example change for agent/agent.py (Agent initialization) • Accept and forward

tool_choice

into the LLM/agent client:

class Assistant:
    def __init__(self, session, logger, instructions, call_context, call_logger, tool_choice=None):
        self.session = session
        self.logger = logger
        self.instructions = instructions
        self.call_context = call_context
        self.call_logger = call_logger
        self.tool_choice = tool_choice
        # when creating the LLM/agent client:
        self.llm_client = create_llm_client(
            model="cerebras",
            temperature=0.2,
            max_tokens=512,
            tool_choice=self.tool_choice,
        )

4) Example change for AgentSession creation (agent/session.py) • Pass

tool_choice

from higher-level config into session/assistant:

from config.config_llm import LLM_CONFIG

def create_agent_session(...):
    tool_choice = LLM_CONFIG.get("tool_choice", None)
    assistant = Assistant(session, logger, instructions, call_context, call_logger, tool_choice=tool_choice)
    return assistant

5) Recommended values and rationale • `tool_choice = "required"`: Best balance — tools can be used but only when the model determines they are necessary (and with stricter prompt rules they’ll be rarer). • `tool_choice = "none"`: Use for safe voice-only flows where no external actions should occur. • If you need some tools but not others, keep

tool_choice='required'

and tighten individual tool descriptions and add confirmation steps before calling sensitive tools (calendar, transfers). 6) Prompt & behavioral guard changes to pair with

tool_choice

• Add to top of master prompt: ◦ "Answer only using provided data and explicit tool outputs. If not available, say 'I don’t have that information; would you like me to look it up?' Do not call tools for clarification." ◦ "Do not reveal tool calls or internal logs in responses." • Require explicit confirmation before destructive/external tools: ◦ Agent: “I’m ready to create an appointment for [date/time]. Should I proceed?” Call calendar tool only after user confirms. 7) Audio/TTS quick settings (apply alongside above) • Noise cancellation: switch to

BVCTelephony()

or

<http://noise_cancellation.NC|noise_cancellation.NC>()

and increase endpointing delays: ◦ `min_endpointing_delay`: try 0.8–1.2 ◦ `max_endpointing_delay`: try 8.0 • Cartesia TTS speed: set

extra_kwargs["speed"] = 0.85

and test. 8) Observability • After changes, watch agent observability traces to confirm fewer unintended function calls and that endpointing/timing improved. If you confirm which

tool_choice

value you'd like and paste the minimal spot where you instantiate the LLM client (a few lines from where

create_llm_client

or similar is called), I’ll produce the exact patch/diff you can apply.