Has anyone observed Groq returning

<function=...>

markup instead of structured

tool_calls

with LiveKit tools enabled? Is there a recommended way to prevent or handle this?

Thanks for reaching out, Patrick. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Thanks — we’ll implement a markup parser + tool validation layer and normalize into FunctionCall/FunctionCallOutput. Do you recommend performing that normalization at the LLM stream boundary (vs. a downstream hook like TTS), and re-prompting the LLM with tool output, or is it acceptable to inject the tool result and continue without a second model call?

Good question — both approaches can work; choice depends on correctness needs and latency trade-offs. Recommendation summary: • If tool output affects the LLM’s next reasoning step (e.g., it must interpret or synthesize the tool result), normalize at the LLM stream boundary, inject the tool result into the conversation, and re-prompt the model so it can incorporate the result into its next token generation. This preserves coherent model reasoning and reduces hallucination risk. • If the tool result is a final action or presentation (e.g., TTS, simple lookup, short factual value) and does not require the LLM to re-reason, it’s acceptable to normalize and inject downstream without an immediate second model call to save latency and cost. Implementation tips: • Normalize into `FunctionCall`/`FunctionCallOutput` as early as possible so downstream components see a consistent shape. • At stream boundary, validate tool name/args and reject or sanitize malformed calls before executing. • When re-prompting, include a brief system/user message with the exact

FunctionCallOutput

(structured JSON) so the model can use it reliably. • If you skip a re-prompt, still log the call and attach metadata to the conversation so you can re-run a re-prompt later if needed (for debugging or recovery). • For interactive UX (voice/TTS), prefer re-prompt when follow-up instructions depend on tool results; prefer direct injection when you only need immediate playback. Which workflow do you plan to prioritize — lower latency or stronger model-consistent reasoning? I can provide a short example prompt pattern or code snippet for the approach you pick.

We’ve implemented a parser for Groq’s

<function=...>

markup and normalize into

FunctionCall/FunctionCallOutput

per your guidance. In a voice agent, we want to announce tool execution (e.g. “One moment while I check that”) while the tool runs, to avoid dead air. We observed that: • yielding announce text early from

tts_node()

does not result in audio playback until after the tool completes • calling

session.say()

from inside

tts_node()

causes re-entrant speech and undefined behavior (double TTS / stalled playout) Question: What is the supported LiveKit pattern for speaking a short announcement during long-running tool execution in voice agents? • Is there a recommended hook outside

tts_node()

for this? • Or is early announce + tool execution intentionally serialized in the default pipeline? We want to stay strictly within supported LiveKit semantics.

You're implementing the announcement while a long-running tool runs — good call. Responding to your two questions and confirming the supported LiveKit pattern. Acknowledgement • You're asking whether to announce outside

tts_node()

and whether early announce + tool execution is serialized by default. Good summary of the problem you observed. Recommended, supported pattern • Use

session.generate_reply()

(or

session.say()

) from inside the long-running tool implementation (i.e., in the tool function), not inside

tts_node()

or other pipeline nodes. • Trigger the announcement asynchronously right after starting the long-running work, then continue the work. This plays audio immediately while the tool continues running and avoids re-entrant TTS issues. Why this is the supported approach •

tts_node()

is a synchronous transformation of LLM output and is serialized with the current turn, so early yields there are queued until the tool completes. • Calling

session.say()

from

tts_node()

creates re-entrant speech and undefined behavior. •

session.generate_reply()

is safe to call from tool code and integrates with the speech queue without blocking the tool. Implementation notes / best practices • Start the announcement as an async background task with a tiny delay (e.g.,

await asyncio.sleep(0.1)

or create_task) to avoid race conditions. • Set

allow_interruptions

appropriately so the announcement can or cannot be interrupted by the user. • Check interruption state (e.g.,

run_ctx.speech_handle.interrupted

) in your tool and cancel/clean up if the user interrupts. • Normalize function calls and validate before executing to avoid invalid calls producing announcements. • Log or attach metadata so you can re-run a re-prompt later if you need the model to incorporate the tool output into its next reasoning step. Short example pattern (conceptual) 1. Tool starts → spawn background task that calls

session.generate_reply("One moment while I check that...")

. 2. Tool performs long operation. 3. Tool returns result → normalize into

FunctionCallOutput

, optionally re-prompt model if you need its synthesis. If you want, I can produce a concise example in your preferred language (Python/Node) showing the async announcement + interruption checks. Which language would you like? Sources: Agent speech and audio | LiveKit docs | Pipeline nodes and hooks | LiveKit docs | Tool definition and use | LiveKit docs | External data and RAG | LiveKit Docs

“When calling

session.say()

from within a tool (async task), what is the recommended way to prevent the same text being spoken again via the normal assistant TTS pipeline when we also stream the assistant text? Is there a canonical ‘tool announcement’ mechanism or metadata flag we should attach to suppress downstream TTS for that message?”

Short answer: Use

session.say(..., add_to_chat_ctx=False)

from inside the tool and do not return or stream the same announcement text as part of the assistant’s LLM reply. There is no built-in metadata flag to auto-suppress downstream TTS — suppression is achieved by keeping the announcement out of the chat context and not echoing it in the assistant response. Key points (concise): • Recommended: call

await context.session.say("One moment...", add_to_chat_ctx=False)

(or

generate_reply()

without adding to chat) from the tool’s async task. • Prevent duplication: do not include the exact announcement text in the tool return value or in subsequent LLM-generated assistant text. Return a silent/structured result (or different wording) so the normal TTS pipeline has nothing to re-synthesize. • If you must include the info in the model turn, mark it in your own metadata or wrap it so your downstream TTS step can detect and skip it — LiveKit provides no automatic

skip_tts

flag. • Use interruption checks (

run_ctx.speech_handle.interrupted

) and

allow_interruptions

to control announcement lifecycle and cancellation. If you want, I can produce a short Python or Node example showing

session.say(..., add_to_chat_ctx=False)

plus a safe tool return pattern. Which language? Sources: Agent speech and audio | LiveKit docs | Events and error handling | LiveKit docs | Tool definition and use | LiveKit docs | External data and RAG | LiveKit Docs

Can I ask some further questions?

Absolutely — please go ahead with your questions.

"Is there a preferred or recommended timing pattern for injecting

FunctionCallOutput

and re-prompting the LLM — should this always happen after tool completion, or can it be streamed in chunks during long executions?"

Short answer: inject the final

FunctionCallOutput

and re-prompt the LLM only after the tool completes. Do not stream partial

FunctionCallOutput

into the LLM during execution. Why: • The LiveKit agent architecture expects tools to return a finalized output before the LLM is re-invoked; streaming partial outputs into the chat context breaks consistent state and can confuse model reasoning. • For long-running tasks, provide user-facing progress via speech (e.g.,

session.say(..., add_to_chat_ctx=False)

) or short status messages, but keep those out of the chat context so they don’t become part of the LLM’s input. When to re-prompt vs. skip: • Re-prompt after completion when the LLM needs to interpret, synthesize, or act on the tool’s result (preserves coherence and reduces hallucination). • If the tool result is purely an actionable side-effect or a simple factual value that doesn’t require further LLM reasoning, you may skip an immediate re-prompt and continue without another model call. Practical pattern 1. Start tool; emit async announcement via

session.say(..., add_to_chat_ctx=False)

if needed. 2. Run tool to completion and produce final

FunctionCallOutput

. 3. Inject that

FunctionCallOutput

into chat context and re-prompt the LLM only if the model must incorporate or comment on the result. If you want, I can provide a short code example (Python or Node) showing this pattern. Which language? Sources: Collect recording consent with tasks | LiveKit docs | Agents (Node.js) | livekit/livekit_composite | DeepWiki | External data and RAG | LiveKit docs | Tool definition and use | LiveKit docs

"Are there any internal flags or message metadata conventions (beyond

add_to_chat_ctx=False

) to instruct downstream TTS not to re-speak a specific message? Or do we need to manage this entirely at the application layer?"

Short answer: there are no built-in internal flags beyond

add_to_chat_ctx=False

to automatically suppress downstream TTS. You must manage suppression at the application layer. Recommended application-layer patterns: • Call

session.say(text, add_to_chat_ctx=False)

from the tool for announcements so the text is not added to chat context. • Ensure the tool does NOT return or stream the same announcement text as part of its

FunctionCallOutput

. Return structured data or a different/silent message instead. • Add a small message-level metadata convention you control (e.g.,

{ spoken_via: "tool_announcement" }

) when generating assistant replies, and have your downstream TTS step skip messages with that flag. • If your pipeline logs or archives turns, tag the announcement entries so replays or analytics know they were spoken already. • Use interruption flags (

run_ctx.speech_handle.interrupted

) and

allow_interruptions

to manage lifecycle and avoid replays on retries. If you want, I can give a minimal Python or Node snippet showing: 1)

session.say(..., add_to_chat_ctx=False)

in a tool async task, and 2) how to tag and filter assistant messages in your TTS pipeline. Which language do you prefer?

"What’s the best practice for logging or tagging tool calls (especially tool announcements) for observability or replay in analytics/debug flows? Should we use metadata fields in FunctionCall/Output?"

Yes — you should treat observability as an application-level concern supplemented by LiveKit Agent Observability. Recommended best practices (concise): • Enable LiveKit Agent Observability first — it automatically logs tool calls, arguments, responses, and timing in synchronized traces (primary source for debug/replay). • Add structured logs from inside each tool’s

execute

function: ◦ Include: tool name, invocation id/trace id,

raw_arguments

, normalized `FunctionCall`/`FunctionCallOutput`, start/end timestamps, and any errors. ◦ For announcements, log an explicit event like

{ event: "tool_announcement", text, announced_via: "session.say", add_to_chat_ctx: false }

. • Attach your own minimal metadata for replay/analytics rather than relying on FunctionCall fields (LiveKit does not expose built-in metadata fields for this): ◦ Example keys:

invocation_id

,

spoken_via

,

announcement_flag

,

analytics_tags

. • Correlate traces and logs with a unique invocation/trace id and surface that id into LiveKit traces and your logs for easy replay. • Store both the structured tool outputs (JSON) and a human-readable transcript entry for analytics; keep announcement entries distinct from assistant reply entries. • For replayability: record whether a speech was played live (boolean), its TTS parameters (voice, ssml), and whether it was added to chat context. • Use your logging/telemetry system (structured JSON + traces, or OpenTelemetry) so you can filter for announcement events, tool failures, long latency, and replay scenarios. Why not rely on FunctionCall metadata: • FunctionCall/FunctionCallOutput are for tool semantics; LiveKit doesn’t provide built-in fields that automatically control downstream TTS or analytics. Managing announcement tagging and replay is done at the application/logging layer. If you’d like, I can provide a minimal Python or Node snippet showing: 1) structured logging inside a tool execute function with an invocation id, 2) logging of a

session.say(..., add_to_chat_ctx=False)

announcement event, 3) emission of the final

FunctionCallOutput

and linking to the same trace id. Which language would you prefer? Sources: Tool definition and use | LiveKit docs | examples/frontdesk | livekit/livekit_composite | DeepWiki | examples/drive-thru | livekit/livekit_composite | DeepWiki | External data and RAG | LiveKit docs | Agents (Node.js) | livekit/livekit_composite | DeepWiki

"Is

run_ctx.speech_handle.interrupted

valid only during TTS, or can it be reliably checked within tools that span several seconds? Is there a timeout/retry recommendation when a user interrupts mid-tool-execution?"

Short answer:

run_ctx.speech_handle.interrupted

is valid throughout a tool’s lifecycle — not just during TTS — and you can reliably check it inside long-running tools. Use it frequently to cancel or avoid committing work if the user interrupts. Below are concise best practices and a small pattern you can adopt. Key behaviors • The flag is set when the user interrupts (VAD/STT) and remains true for that SpeechHandle’s lifetime. • It can be polled at any point inside the tool, even seconds after speech started. • The framework does not auto-retry interrupted tools; retry/timeout logic is up to your agent. Recommended pattern 1. Start long-running work as an awaitable task (or several). 2. Spawn or periodically call checks that inspect

run_ctx.speech_handle.interrupted

. 3. Use helper semantics like

wait_if_not_interrupted()

where available to wait safely. 4. On interruption: cancel ongoing tasks, perform minimal cleanup, return

None

or a neutral value so the tool result is ignored by the agent turn, and optionally add a short system/chat note acknowledging the interruption. 5. Offer a controlled retry flow: after interruption, re-prompt the user (or wait for user intent) before re-running the tool. Minimal Python sketch

@function_tool()
async def long_tool(self, run_ctx: RunContext, params):
    task = asyncio.create_task(self._do_long_work(params))

    # Periodically check interruption while waiting
    while not task.done():
        await asyncio.sleep(0.25)
        if run_ctx.speech_handle.interrupted:
            task.cancel()
            # optional: log and return neutral result so LLM ignores it
            return None

    return await task  # final FunctionCallOutput

Alternate with helper • Use

await run_ctx.speech_handle.wait_if_not_interrupted([task])

if available to avoid races and simplify logic. Timeout/retry recommendations • Timeouts: enforce sensible timeouts for external calls (e.g., 10–30s depending on UX). • On interruption: do not auto-retry. Ask the user (via a short prompt) whether to continue or restart the action. • If the tool is idempotent and you want auto-retry, gate it behind user confirmation to avoid surprising behavior. Additional notes • For critical operations that must finish, temporarily disallow interruptions (

run_ctx.disallow_interruptions()

), but use this sparingly because it affects UX. • Log interruption events with invocation/trace IDs so you can inspect cancelled runs in observability traces. If you’d like, I can produce a Node or more complete Python example adding interruption-aware cleanup and logging. Which would you prefer? Sources: Agent speech and audio | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki | Tool definition and use | LiveKit docs | Turns overview | LiveKit docs

"Are there recommended prompt patterns for Groq (or other models) to consistently emit

<function>

markup when calling tools? Should we include special system messages or few-shot examples?"

Yes — you can increase consistency, and the recommended approach is to use explicit system instructions + a couple of few‑shot examples + precise tool schemas. Concrete checklist and a minimal system prompt pattern you can copy: 1) System instruction (must) • Tell the model to output ONLY raw function-call markup when invoking tools, give exact markup format you expect. Example: System: You must use function calls to invoke tools. When calling a tool, output exactly:

<function=TOOL_NAME>{JSON_ARGS}</function>

Do not output any explanatory prose or extra characters around the markup. 2) Few‑shot examples (highly recommended) • Provide 1–2 short examples (user → expected

<function=...>

output) showing real inputs and exact markup JSON. Keep examples minimal and representative. 3) Precise tool schemas / descriptions (required) • Define each tool with a clear name, one-line description, and parameter schema/types. Prefer strict JSON schema or typed parameter definitions so the model knows argument names/types. 4) Enforce format constraints • In the system prompt state: “If you cannot call a tool, reply in plain text explaining why; otherwise output only the function markup.” This reduces mixed prose + markup outputs. 5) Normalization + validation (must) • Always validate and normalize model output (parse markup, JSON-decode args, validate against tool schema). Models may still vary, so parsing + fallback is required. 6) Testing & iterative refinement • Test a variety of prompts and examples; if you see deviations, add a counterexample to the system prompt or tighten examples. • Use LiveKit’s testing harness / observability to inspect outputs and iterate. 7) Optional stability techniques • Add a short guard token rule in system prompt like: “Do not include spaces between

>

and

{

,” or exact spacing rules if you need deterministic parsing. • If Groq sometimes uses other delimiters, include alternative accepted patterns in your normalization code rather than widening system prompt complexity. Minimal prompt template to start with • System: (as above) + one few‑shot example pair. • Then user message(s) as normal. Notes • There’s no guaranteed way to force 100% compliance; models differ. Prompting + schemas + few‑shot examples greatly increase consistency, but always implement parser + validation. • If strict structured function calling is critical, prefer models/tools that support standard structured

tool_calls

natively; otherwise the above pattern plus robust normalization is the supported path. If you want, I can draft a ready-to-use system prompt and two few-shot examples tailored to your tool names and parameter schemas. Would you like that? Sources: Groq LLM plugin guide | LiveKit docs | How to test your agent using another agent | Tool definition and use | LiveKit docs | LLM Output Replacement | LiveKit docs | Testing and evaluation | LiveKit docs