Thanks — I’m looking at the hang you described when preemptive generation is used with structured outputs. I’ll highlight the most likely suspects and give targeted troubleshooting steps you can run now to narrow it down. What I’m responding to: preemptive generation causing the agent / custom LLM node to hang when using structured outputs. Likely causes and why they matter • Structured-output parsing waiting for complete JSON: If your parser expects valid JSON before proceeding, partial/interim chunks from preemptive streaming can leave the parser waiting indefinitely and block the pipeline. • Preemptive-generation trigger mismatch: Preemptive generation starts when the agent isn’t draining and no audio is playing. If the LLM’s streaming format or timing differs for structured outputs, the agent may not detect completion and the pipeline can stall. • Stream chunk shape differences: Code that mutates or reads

chunk.delta.content

assumes a ChatChunk shape. Structured outputs may arrive as different shapes (raw strings, nested JSON), causing silent failures or exceptions that hang the generator. • Buffer flush logic tied to punctuation: If flush logic only flushes on sentence-ending punctuation, JSON-only structured payloads may never trigger a flush. • Missing/insufficient error handling: Network or parse errors during streaming may be unhandled, leaving futures or async generators unresolved. • Stale preemptive state: If the

_preemptive_generation

flag or activity isn’t cancelled/updated correctly, the agent can keep waiting on an old preemptive task. Steps to reproduce/confirm (quick checks) 1. Disable preemptive_generation temporarily and confirm the hang disappears. If it does, preemptive flow is the culprit. 2. Log every incoming chunk in your llm_node (dump raw chunk shape and content) to see if you get partial JSON, string yields, or unexpected chunk types. 3. Reproduce with a minimal prompt that requests a small structured JSON so you can observe streaming token-by-token. Concrete fixes and mitigations 1. Make parsing resilient to partial JSON ◦ Incrementally accumulate chunks and try non-blocking parse attempts; only block for a short, configurable timeout waiting for completion. ◦ On parse failure, do not block forever — fallback to returning the last valid partial content or signal an incomplete-JSON handler. 1. Change flush triggers for structured output ◦ Don’t rely only on punctuation. Use token-count thresholds, explicit end-of-structured-output markers, or time-based flushes. 1. Validate chunk structure before mutating ◦ Check for presence of

delta

and

delta.content

(or string yields) before in-place modification. Log an error and skip mutation if the shape differs. 1. Improve error handling around the stream ◦ Wrap stream processing in try/except. Ensure all exit paths resolve any awaiting futures and clear preemptive flags. 1. Ensure preemptive state is cancelled/updated ◦ When a preemptive task is interrupted or completes, explicitly clear

_preemptive_generation

and related activity state so no stale waiters remain. 1. Confirm compatibility of response format with streaming ◦ Verify the LLM supports streaming structured responses as you expect. If the model returns full JSON only at the end, preemptive streaming assumptions must be adjusted. Diagnostic logs to collect (if you want me to help further) • A sequence of raw chunk logs from start of preemptive generation to hang (timestamped). • Whether chunks are ChatChunk objects or plain strings, and any parse exceptions raised. • The state of `_preemptive_generation`/AgentActivity when the hang occurs. • The llm_node override snippet that mutates or inspects

chunk.delta.content

. If you’d like, paste a short sample of the raw chunk logs or the llm_node override code you’re using and I’ll point out exact lines to change and precise try/except placement. Sources: Tool definition and use | LiveKit Documentation | examples/voice_agents | livekit/livekit_composite | DeepWiki | LLM-Powered Content Filter | LiveKit Documentation | LLM Output Replacement | LiveKit Documentation | Modifying LLM output before TTS | LiveKit Documentation