can an agent handoff be done between agents of different providers? for example, main agent runs OpenAI Realtime but we have a handoff to an agent running Grok Voice?

Thanks for reaching out, Sebastián. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

how to override the agent session model in the handoff agent?

You're asking how to override the agent session model (LLM / TTS / STT) for the agent that receives a handoff. You do this by configuring the new agent's models at instantiation—i.e., in the receiving agent's constructor—and returning that agent from the handoff. The receiving agent runs with its own plugins and model settings independent of the handing-off agent. Quick examples: 1) Python

class BillingAgent(Agent):
    def __init__(self, chat_ctx=None):
        super().__init__(
            instructions="Billing specialist",
            llm=xai.realtime.RealtimeModel(voice="Ara"),   # override LLM/provider
            tts=cartesia.TTS(model="sonic-3", voice="billing-voice")
        )
        if chat_ctx:
            self.chat_ctx = chat_ctx  # preserve history if desired

class TriageAgent(Agent):
    @function_tool()
    async def transfer_to_billing(self):
        return BillingAgent(chat_ctx=self.chat_ctx), "Transferring to billing"

2) Node.js / TypeScript

class BillingAgent extends voice.Agent {
  constructor(opts = {}) {
    super({
      instructions: "Billing specialist",
      llm: new GrokRealtime({ model: "grok-voice-model" }), // different provider
      tts: new CartesiaTTS({ model: "sonic-3", voice: "billing-voice" }),
      ...opts
    });
  }
}

class TriageAgent extends voice.Agent {
  constructor() {
    super({
      tools: {
        transferToBilling: llm.tool({
          execute: async () => {
            return llm.handoff({ agent: new BillingAgent({ chat_ctx: this.chat_ctx }) , returns: "Transferring to billing" });
          }
        })
      }
    });
  }
}

Notes: • Set the LLM/TTS/STT in the receiving agent's constructor—there is no separate post-handoff “session model override.” • Pass

chat_ctx

when creating the new agent if you want to preserve conversation history. • Each agent can use different providers/models (OpenAI, Grok, etc.) because plugins are per-agent. If you want, paste your current agent classes and I’ll show the exact changes to override the model for your handoff. Sources: Agents and handoffs | LiveKit Documentation | The Handoff Pattern for Voice Agents That Replaces IVR Menus | LiveKit, with interactive elements

handoff has a last message spoken by the current model as if it was the to-be handoff model, can we change that? is like after a tool call of the handoff, the next response is from the previous agent

Thanks — you're asking why the final spoken message before a handoff sounds like it's from the to-be-handoff model and whether you can make that last message come from the new agent instead. This is expected behavior and cannot be changed: the current agent completes its turn (including generating and speaking its reply) before the handoff takes effect. The new agent only begins on the next turn. What you can do: • Have the current agent return a short, explicit transfer line (e.g., "Connecting you to billing now...") so the user expects the transition. • Preserve context by passing

chat_ctx

when instantiating the new agent so it can continue the conversation smoothly. • Let the new agent speak first on its turn via an

on_enter

greeting (or immediate prompt) so its voice/model appears promptly after the transfer. • If the originating agent’s speech is problematic, return a minimal transfer message (brief text) to reduce confusion. If you want, paste the handoff tool/agent code you’re using and I’ll show the exact place to add a transfer line,

chat_ctx

passing, or an

on_enter

greeting. Sources: Agents and handoffs | LiveKit Documentation | Function tool definition | LiveKit Documentation | The Handoff Pattern for Voice Agents That Replaces IVR Menus | LiveKit, with interactive elements

I'm getting errors on OpenAI realtime about missing or duplicate item ids

Sorry you’re hitting that — I can help diagnose it. A few quick questions to narrow this down: • Can you paste the full error text (exact message and any error codes)? • Where/when does it occur? (on connect, during streaming, on handoff, when sending a chunked event, etc.) • Are you using LiveKit Agents with OpenAI Realtime or calling the OpenAI Realtime API directly? Which SDK/language and version? • Do you create any custom items/events with IDs (e.g., for streaming messages, audio frames, or tools)? If so, show the code that generates those IDs. • Has this started after a specific change (new agent, handoff, client update)? Post those details and a minimal code snippet of the failing call and I’ll point out likely causes and exact fixes. , with interactive elements

i get livekit.agents.llm.realtime.RealtimeError: update_chat_ctx timed out. when transferring from grok to openai back with this issue:

OpenAI Realtime API returned an error: RealtimeError(message="Invalid 'item.id': string too long. Expected
                                             a string with maximum length 32, but got a string with length 36 instead.", type='invalid_request_error',
                                             code='string_above_max_length', event_id='chat_ctx_create_8596967c1c44', param='item.id')

Thanks — I see two related issues: the OpenAI Realtime error complaining the

item.id

is too long (36 chars vs max 32), and

update_chat_ctx timed out

during the handoff. Both are causing the handoff to fail. What to do next (quick fixes and checks): 1. Trim/normalize IDs • Ensure any IDs you send to OpenAI Realtime (chat context item ids, event ids) are <= 32 characters. • If you generate UUIDs or long strings, truncate or hash them to a 32-char safe value before sending (e.g., take a SHA256 hex and use the first 32 chars). • Example pattern:

id = sha256(original_id)[:32]

or

id = original_id[:32]

. 1. Locate where the long ID is created • Search your handoff code and

chat_ctx

serialization for item/event id generation (look for fields named

id

,

item.id

,

event_id

,

chat_ctx_create_*

). • If you pass

chat_ctx

between agents, inspect entries in `chat_ctx`—one of those item ids is exceeding 32 chars. 1. Fix duplicate IDs (if present) • Ensure IDs used for distinct items are unique after truncation (use a stable hash + short suffix, or include a timestamp). • Avoid reusing the same id for new items. 1. Address the

update_chat_ctx timed out

error • That timeout can happen when OpenAI rejects the update due to invalid payload (like too-long ids) or if the update is retried. • Fixing the invalid id usually resolves the timeout; after correcting IDs, retry the handoff and the update call should succeed. • If timeouts persist after fixing IDs, capture the full request payload and timing to see if payload size or network latency is the cause. If you want, paste the code snippet that builds/sends the

chat_ctx

or the handoff call (the place that creates

item.id

), and I’ll show exactly where to truncate or replace the id and how to ensure uniqueness. , with interactive elements

but im not doing anything, is it an issue between openai, xai and livekit chat ctx?

Thanks — you provided the error showing OpenAI Realtime rejected an

item.id

because it was 36 chars (limit is 32), and

update_chat_ctx timed out

during the handoff. What’s happening • OpenAI Realtime requires item IDs ≤ 32 characters. • LiveKit (and common UUIDs) often use 36-char IDs. When LiveKit sends chat context items with IDs longer than 32 chars, OpenAI rejects the update which causes the handoff/update to fail and timeout. • This is a compatibility/validation mismatch (not an arbitrary cross-provider bug). Fixing the IDs resolves the OpenAI error and the timeout. Immediate fixes you can apply • Truncate or replace offending IDs before sending chat context to OpenAI: ◦ Truncate:

id = original_id[:32]

◦ Stable hash:

id = sha256(original_id).hexdigest()[:32]

(keeps uniqueness) • Ensure uniqueness after truncation (append a short timestamp or suffix if needed). • Apply this when constructing or copying

chat_ctx

for the handoff so all

item.id

and event ids you send to the Realtime API are ≤ 32 chars. If you want, paste the small snippet that builds/sends your

chat_ctx

(or tell me whether you’re using Python or Node.js) and I’ll show the exact place to truncate/hash IDs and a short snippet you can drop in. Sources: Chat context | LiveKit Documentation | OpenAI Realtime API plugin guide | LiveKit Documentation | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Agents and handoffs | LiveKit Documentation | The Handoff Pattern for Voice Agents That Replaces IVR Menus | LiveKit, with interactive elements

what's this element: chat_ctx_create_8596967c1c44

Good question — that element is an internal event identifier LiveKit generates when creating/updating the chat context. •

chat_ctx_create_8596967c1c44

is the event_id LiveKit attached to the chat-context creation/update sent to the LLM provider (it helps trace the chat_ctx update request). • OpenAI Realtime enforces a maximum item id length of 32 characters; that event_id (36 chars) is what's being rejected in your error. • To fix this, ensure LiveKit sends event/item ids ≤ 32 chars when calling the OpenAI Realtime API (truncate or replace with a stable 32-char hash before the handoff). If you want, share whether you’re using the Python or Node SDK (or paste the small snippet that triggers the handoff) and I’ll show the exact place to truncate/hash that event id. , with interactive elements

i'm using python sdk:

OpenAI Realtime API returned an error: RealtimeError(message="Invalid 'item.id': string too long. Expected
                                             a string with maximum length 32, but got a string with length 36 instead.", type='invalid_request_error',
                                             code='string_above_max_length', event_id='chat_ctx_create_7b8051a0f2d4', param='item.id')

Thanks — I’ll show a quick Python SDK fix and where to apply it. Why this happens (brief) • LiveKit generates event IDs like

chat_ctx_create_7b8051a0f2d4

(36 chars). • OpenAI Realtime requires

item.id

strings ≤ 32 chars, so the event_id is rejected and the chat_ctx update/handoff times out. Quick fix (truncate or stable-hash event/item ids before sending to OpenAI Realtime) 1) Truncate IDs (simple) • Before calling the handoff or sending chat_ctx to the Realtime API, ensure each

item.id

and any

event_id

is at most 32 chars:

def safe_id(orig: str) -> str:
    return orig[:32]

2) Stable hash (preserves uniqueness) • Preferred if you need uniqueness across long originals:

import hashlib

def safe_id(orig: str) -> str:
    return hashlib.sha256(orig.encode()).hexdigest()[:32]

Where to apply it • When you create or copy

chat_ctx

for the new agent (the place you pass

chat_ctx

into the handoff), map over

chat_ctx.items

and replace

item.id

and any

event_id

fields with

safe_id(...)

before sending to the OpenAI Realtime plugin. • Also sanitize any generated UUIDs or tool/event IDs you pass to providers. If you paste the small snippet that does your handoff or shows how you build/send

chat_ctx

(the Python code that returns the new Agent or calls

session.update_agent()

), I’ll return the exact patched snippet with the sanitization applied. , with interactive elements

can you help me with that please?

chat_ctx=self.chat_ctx.copy(
                    exclude_instructions=True,
                    exclude_config_update=True,
                    exclude_empty_message=True,
                ),

Thanks — I’ll show a minimal patch that sanitizes IDs when you copy

chat_ctx

before handing off. 1) Add a helper to produce safe IDs (stable hash keeps uniqueness):

import hashlib

def safe_id(orig: str) -> str:
    return hashlib.sha256(orig.encode()).hexdigest()[:32]

2) Sanitize the copied chat context items before passing into the new agent:

new_ctx = self.chat_ctx.copy(
    exclude_instructions=True,
    exclude_config_update=True,
    exclude_empty_message=True,
)

# Replace item ids and any event_id fields with safe ids
for item in new_ctx.items:
    if getattr(item, "id", None):
        item.id = safe_id(item.id)
    if getattr(item, "event_id", None):
        item.event_id = safe_id(item.event_id)
    # sanitize nested ids if present (function_call, references, etc.)
    if getattr(item, "function_call", None) and getattr(item.function_call, "id", None):
        item.function_call.id = safe_id(item.function_call.id)

3) Return the new agent with the sanitized context:

return BillingAgent(chat_ctx=new_ctx), "Transferring to billing"

If you paste the structure of a typical

item

(its fields), I can adjust the sanitization loop to cover any nested id fields you have. , with interactive elements

mi main session linked with the original OpenAI realtime is trying to remote update with the xai conversation items, why is that?

You're seeing the original OpenAI Realtime session try to update with xAI (Grok) conversation items because LiveKit preserves and attempts to synchronize the shared chat context across the session during handoff. Key points and what to check/fix: • Why it happens ◦ LiveKit sends the current session's

chat_ctx

(all items/events) to the LLM provider handling the next update. If the session still has an active OpenAI Realtime connection, LiveKit will try to push the updated chat context (including items produced by the xAI agent) back to that OpenAI session. ◦ Handoffs do not retroactively replace the provider mid-turn: the originating agent (OpenAI) finishes its turn and LiveKit processes chat context updates afterwards. That can cause OpenAI to receive items created by the xAI agent if context merging/sanitization isn't handled prior to the update. • Likely causes in your flow ◦ You passed the raw

chat_ctx

(containing xAI-created items) back into an OpenAI-managed update without sanitizing or remapping IDs/fields. ◦ The OpenAI session was still considered the active session owner when LiveKit attempted

update_chat_ctx

. ◦ Race/timing: the new agent created items quickly (or concurrently) and LiveKit attempted to update OpenAI before the handoff lifecycle completed. • Immediate actions to stop OpenAI from receiving xAI items a. Ensure you pass a cleaned/filtered

chat_ctx

to the new agent instead of blindly copying the entire shared context. For handoffs, copy and then filter out provider-specific items or items with IDs not compatible with the target provider. b. Sanitize item IDs (we previously discussed truncation/hash to <=32 chars) before any provider update. c. Explicitly complete/close the originating agent's turn before creating provider-specific items. Let the new agent run its

on_enter

or first turn only after the handoff completes. d. Use

chat_ctx.copy(exclude_instructions=True, ...)

then remove or remap items you don't want forwarded to OpenAI: • Remove items where

item.source

or a metadata flag identifies them as xAI-created. • Or keep only user messages and essential function-call history for the provider update. a. Add a short delay or await the handoff confirmation path so LiveKit does not race to update the old session with newly created items. • Example pattern (conceptual) ◦ When handing off from Grok -> OpenAI: i. NewAgent = OpenAIAgent(chat_ctx = sanitized_copy) ii. Ensure sanitized_copy contains only the items OpenAI should see (user messages, stable ids ≤32, no xAI-only events) iii. Return NewAgent so LiveKit switches session ownership, then let NewAgent call OpenAI. • Debug tips ◦ Log

chat_ctx.items

and their `source`/metadata right before any provider update to see which items are being sent. ◦ Trace the handoff lifecycle events to confirm which agent is active when

update_chat_ctx

runs. ◦ Verify you’re not calling provider-specific updates from callbacks that run before the handoff finalizes. If you share the small snippet that performs the handoff and any code that mutates

chat_ctx

after the handoff (especially where provider updates occur), I’ll show exactly where to filter/sanitize items so OpenAI will not receive xAI conversation items. , with interactive elements

and when going from OpenAI to Grok

You're asking how to prevent the original OpenAI session from attempting to update with Grok (xAI) conversation items when handing off from OpenAI → Grok. Below is a concise explanation and a Python pattern to close/sanitize the context and perform a clean handoff. Why this happens • LiveKit keeps a shared

chat_ctx

. If the OpenAI session is still considered active when LiveKit pushes an update, it can receive items created by the Grok agent. • OpenAI rejects items with invalid IDs (e.g., >32 chars), causing update failures/timeouts. • The solution is: ensure the handoff finalizes the OpenAI turn, sanitize/filter the

chat_ctx

(IDs and provider-specific items), then instantiate the Grok agent with the cleaned context. Minimal Python pattern to fix (sanitize, close origin turn, then handoff)

import hashlib

def safe_id(orig: str) -> str:
    return hashlib.sha256(orig.encode()).hexdigest()[:32]

def sanitize_chat_ctx(ctx):
    new_ctx = ctx.copy(
        exclude_instructions=True,
        exclude_config_update=True,
        exclude_empty_message=True,
    )
    filtered_items = []
    for item in new_ctx.items:
        # drop provider-specific items if marked (adjust condition to your metadata)
        if getattr(item, "provider", None) == "xai":
            continue
        # sanitize obvious id fields
        if getattr(item, "id", None):
            item.id = safe_id(item.id)
        if getattr(item, "event_id", None):
            item.event_id = safe_id(item.event_id)
        # sanitize nested function_call ids if present
        if getattr(item, "function_call", None) and getattr(item.function_call, "id", None):
            item.function_call.id = safe_id(item.function_call.id)
        filtered_items.append(item)
    new_ctx.items = filtered_items
    return new_ctx

# Call this in your handoff tool/handler
# 1) ensure current agent completes its turn (return a short transfer message)
# 2) sanitize context and create new agent
new_ctx = sanitize_chat_ctx(self.chat_ctx)

# instantiate Grok agent with cleaned context
return GrokBillingAgent(chat_ctx=new_ctx), "Connecting you to billing now..."

Practical points • Ensure the originating agent returns a short transfer message (so OpenAI finishes its turn) before LiveKit attempts provider updates. • Filter out items that are explicitly created by the other provider (use a metadata flag like

item.provider

or

item.source

), keeping only user messages and required history. • Sanitize all IDs (item.id, event_id, nested ids) to ≤32 chars using

safe_id

. • Log

chat_ctx.items

right before provider updates to verify what will be sent. • If you still see timeouts after sanitizing IDs, confirm the OpenAI session is closed/finished and check for race conditions or retries. If you paste the exact structure of one

item

and where `provider`/`source` metadata lives in your objects, I’ll adapt the sanitize loop precisely for your schema. , with interactive elements

how could we keep the or forward the user request or intention from OpenAI to Grok so user doesn't have to repeat itself

Good question — you can keep and forward the user's intent so they don't need to repeat themselves. The core idea is to pass a sanitized, focused slice of the chat context (the user's recent request + minimal necessary history) into the new Grok agent so it can continue immediately. What to do (concise steps) 1. End the OpenAI agent's turn with a short transfer message so handoff completes. 2. Build a minimal context containing: ◦ The user’s last message(s) that express intent ◦ Any relevant prior messages or function-call results needed for context ◦ Optional system/instruction items required by the Grok agent 1. Sanitize IDs and provider-specific items (truncate/hash IDs to ≤32 chars; remove OpenAI/xAI-specific events you don’t want forwarded). 2. Instantiate the Grok agent with that sanitized chat_ctx so it can respond immediately (use

on_enter

or a prompt to surface the forwarded intent). Minimal Python pattern (drop into your handoff tool)

import hashlib

def safe_id(orig: str) -> str:
    return hashlib.sha256(orig.encode()).hexdigest()[:32]

def make_forward_ctx(ctx, keep_last_n=1):
    # copy and trim down to the last N user messages + essentials
    new_ctx = ctx.copy(exclude_instructions=True, exclude_config_update=True, exclude_empty_message=True)
    # keep only user messages and needed metadata
    user_items = [it for it in new_ctx.items if it.role == "user"]
    keep = user_items[-keep_last_n:]
    # optionally include a short system prompt for Grok
    system_item = next((it for it in new_ctx.items if it.role == "system"), None)
    new_items = ([system_item] if system_item else []) + keep
    # sanitize ids
    for it in new_items:
        if getattr(it, "id", None):
            it.id = safe_id(it.id)
        if getattr(it, "event_id", None):
            it.event_id = safe_id(it.event_id)
        if getattr(it, "function_call", None) and getattr(it.function_call, "id", None):
            it.function_call.id = safe_id(item.function_call.id)
    new_ctx.items = new_items
    return new_ctx

# usage inside handoff
transfer_text = "Connecting you to our specialist — one moment."
# ensure OpenAI finishes its turn by returning a short transfer message
new_ctx = make_forward_ctx(self.chat_ctx, keep_last_n=1)
return GrokAgent(chat_ctx=new_ctx), transfer_text

Practical tips • Keep the forwarded context minimal (user intent + required facts) to avoid size/validation issues and to make model behavior predictable. • Use a small system instruction tailored for Grok so it knows it’s continuing a transferred request. • Log the items you forward to confirm the right message is being passed. • Sanitize IDs to avoid OpenAI length limits and provider incompatibilities. If you paste a sample

chat_ctx.items

(one user item + one system item), I’ll adapt the helper to your exact schema and show the precise sanitized payload you should send. , with interactive elements