FrameProcessor on AudioInputOptions.noise_cancellation works with Realtime models?

Thanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

for self hosting approach what should we use?

You're asking what to use for noise cancellation when self‑hosting. A few important facts and a couple quick questions so I can give precise guidance: Facts: • The enhanced noise cancellation plugin used by LiveKit Cloud must be removed when self‑hosting. • For self‑hosted deployments you should use your own plugins/AI providers and set up a custom deployment (Kubernetes/Docker). • Krisp noise cancellation is supported for telephony/SIP use cases and can be enabled via API for trunks/participants. Quick questions: 1. Are you self‑hosting for WebRTC rooms, SIP/telephony, or both? 2. Are you using Realtime models (bidirectional audio) or an STT→LLM→TTS pipeline? 3. What's your deployment environment (Kubernetes, Docker Compose, cloud provider)? Answer those and I’ll recommend the exact components and config/snippet to use. Sources: Turns overview | LiveKit Documentation | Voice AI quickstart | LiveKit Documentation | Python SDKs | livekit/livekit_composite | DeepWiki | Agents telephony integration | LiveKit Documentation, with interactive elements

sip telephony and for testing locally i am using livekit playground open ai realtime model docker compose and kubernetes

Thanks — you’re building a self‑hosted setup for SIP telephony, testing locally with LiveKit Playground, using the OpenAI Realtime model, and deploying via Docker Compose / Kubernetes. Below are focused, actionable recommendations and next steps. What to use (high level) • Remove LiveKit Cloud’s enhanced noise‑cancellation plugin from your agent code when self‑hosting. • For SIP telephony: use Krisp for noise cancellation if you need telephony-grade noise suppression; Krisp can be enabled via API when creating SIP trunks/participants. • Use your own plugins/providers for AI (OpenAI Realtime) in the self‑hosted agent framework. • Test locally with LiveKit Playground, then deploy the same agent to Docker Compose for quick verification and Kubernetes for production. Concrete next steps 1. Local testing (LiveKit Playground) ◦ Connect your SIP trunk/participant through the Playground and route audio to your local self‑hosted agent. ◦ Ensure agent code does not include the LiveKit Cloud enhanced noise plugin. 1. Agent configuration (use OpenAI Realtime) ◦ Configure the agent to use OpenAI Realtime as your model provider inside your agent’s plugins configuration. ◦ Ensure the agent session/room options enable audio processing settings you need (noise cancellation handled either via Krisp or your own audio processing plugin). 1. Enabling Krisp for SIP ◦ When creating SIP trunks or participants, enable Krisp via the relevant API option so telephony audio gets Krisp processing. ◦ Keep Krisp usage tied to telephony/SIP participants where required. 1. Deployment ◦ Docker Compose: deploy the agent and LiveKit services locally for testing. Keep the same plugin/config layout used in Playground. ◦ Kubernetes: create manifests/Helm charts for the agent, LiveKit server, and any sidecars (STT/TTS) you need. Remove cloud-only plugins and include your provider credentials as Kubernetes secrets. What I need to give exact config/snippets • A sample of your current agent config or Docker Compose/K8s manifest (redact secrets). • How you currently create SIP trunks/participants (API calls or UI) so I can show the exact place to enable Krisp and where to remove the cloud plugin. If you want, paste your agent config or compose/k8s files (remove secrets) and I’ll produce the exact edits/snippets to: • Remove the cloud noise plugin • Wire OpenAI Realtime into your agent plugins • Enable Krisp for SIP participants • Provide Docker Compose/Kubernetes examples for deployment., with interactive elements

session = AgentSession[CallState]( llm=openai.realtime.RealtimeModel.with_azure( voice="alloy", modalities=["audio", "text"], turn_detection=None, input_audio_noise_reduction="far_field", azure_endpoint=deployment.base_url or "", api_key=deployment.api_key, azure_deployment=deployment.azure_deployment_name, api_version=deployment.api_version or "2024-10-01-preview" ), vad=ctx.proc.userdata["vad"], userdata=state, ivr_detection=True, user_away_timeout=15.0, aec_warmup_duration=3.0, turn_handling={ "interruption": { "enabled": True, "min_duration": 0.8, "min_words": 2, }, }, max_tool_steps=5, ) what will be gemini equivalent and required imports?

Thanks — I’m responding to your request to convert the OpenAI Realtime

AgentSession

snippet to the Gemini Live API equivalent and list required imports. Required imports (example Python)

from livekit import agents
from livekit.agents import AgentSession
from livekit.plugins import noise_cancellation, google  # google contains the Gemini Live plugin

Gemini-equivalent AgentSession

session = AgentSession[CallState](
    llm=google.realtime.RealtimeModel(
        model="gemini-2.5-flash-native-audio-preview-12-2025",  # choose appropriate Gemini model
        voice="Puck",  # pick a supported Gemini voice
        modalities=["AUDIO", "TEXT"],          # Gemini uses these modality names
        turn_detection=None,                   # disable server-side turn detection if desired
        # Gemini does not support OpenAI's `input_audio_noise_reduction` parameter
        api_key=deployment.api_key,            # or use Vertex AI credentials instead
        vertexai=False,                        # set True and provide project/location if using Vertex AI
    ),
    vad=ctx.proc.userdata["vad"],
    userdata=state,
    ivr_detection=True,
    user_away_timeout=15.0,
    aec_warmup_duration=3.0,
    turn_handling={
        "interruption": {
            "enabled": True,
            "min_duration": 0.8,
            "min_words": 2,
        },
    },
    max_tool_steps=5,
)

Notes specific to noise reduction and SIP telephony •

input_audio_noise_reduction

is OpenAI-specific and is not available for Gemini. Use LiveKit-side noise cancellation instead. • For SIP telephony, use `BVCTelephony`/Krisp options via

AudioInputOptions

(or enable Krisp when creating trunks/participants) so telephony audio gets proper suppression. If you want, paste your agent config or compose/k8s files (redact secrets) and I’ll produce exact edits/snippets to: • remove cloud-only noise plugins, • wire Gemini Live API credentials, • enable telephony noise cancellation (BVCTelephony or Krisp) in your self-hosted setup. Sources: Azure OpenAI Realtime API and LiveKit | LiveKit Documentation | Turns overview | LiveKit Documentation | Using Noise Cancellation for SIP and WebRTC Participants | Gemini Live API plugin | LiveKit Documentation | Realtime model metrics | LiveKit Documentation, with interactive elements

livekit.…ns.google Gemini Realtime API error: 1007 None. Request contains an invalid argument. Traceback (most recent call last): File "/usr/local/lib/python3.11/site-… line 1107, in connect raw_response = await ws.recv(decode=False) ^^^^^^^^^^^^^… File "/usr/local/lib/python3.11/site-… line 322, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.Connection… received 1007 (invalid frame payload data) Request contains an invalid argument.; then sent 1007 (invalid frame payload data) Request contains an invalid argument. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-… line 829, in _main_task async with self._client.aio.live.connect( File "/usr/local/lib/python3.11/conte… line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-… line 1117, in connect errors.APIError.raise_error(… reason, None) File "/usr/local/lib/python3.11/site-… line 188, in raise_error raise cls(status_code, response_json, response) google.genai.errors.APIError: 1007 None. Request contains an invalid argument. {"room": "voice_assistant_room_1", "customer_id": "cust_001", "campaign": "camp_001", "trunk_id": "ST_MJixyuHRmi5a", "pid": 256, "job_id": "AJ_twjvFiDVVij5", "room_id": "RM_DdeiDdgnhJ7D"} Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/google/genai/live.py", line 1107, in connect raw_response = await ws.recv(decode=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/connection.py", line 322, in recv raise self.protocol.close_exc from self.recv_exc websockets.exceptions.ConnectionClosedError: received 1007 (invalid frame payload data) Request contains an invalid argument.; then sent 1007 (invalid frame payload data) Request contains an invalid argument. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/livekit/plugins/google/realtime/realtim e_api.py", line 829, in _main_task async with self._client.aio.live.connect( File "/usr/local/lib/python3.11/contextlib.py", line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/google/genai/live.py", line 1117, in connect errors.APIError.raise_error(code, reason, None) File "/usr/local/lib/python3.11/site-packages/google/genai/errors.py", line 188, in raise_error raise cls(status_code, response_json, response) google.genai.errors.APIError: 1007 None. Request contains an invalid argument. 133502.049 ERROR events event.component_error {"source": "RealtimeModel", "error": "type='realtime_model_error' timestamp=1776173702.0469296 label='livekit.plugins.google.realtime.realtime_api.RealtimeModel' error=APIError('1007 None. Request contains an invalid argument.')

Thanks — I’m responding to your Gemini Realtime API 1007 error ("Request contains an invalid argument") from the LiveKit Google plugin. Below are the most likely causes, focused fixes, and two quick checks I need from you. Most likely causes & fixes • Using an incompatible Gemini model (Gemini 3.1 variants have known limitations). Use a compatible model such as a Gemini 2.5 flash native-audio model (e.g.,

gemini-2.5-flash-native-audio-preview-12-2025

). • Passing unsupported Realtime options. Remove any Gemini 3.1-only options like

enable_affective_dialog

,

proactivity

, or other server-side features not accepted by the model. • Sending

input_audio_noise_reduction

or other OpenAI-specific params to Gemini. Gemini does not accept that parameter — handle noise reduction on the LiveKit side (BVCTelephony/Krisp). • Missing or incorrect auth/config for Gemini: ◦ If using API key: ensure

api_key

is set and exported into the container as

GOOGLE_API_KEY

. ◦ If using Vertex AI: set

vertexai=True

, supply

project

and

location

, and mount

GOOGLE_APPLICATION_CREDENTIALS

. • WebSocket payload framing issues due to invalid request fields cause the 1007 close. Cleaning the model config (above) usually resolves it. Quick actionable changes to try 1. Change model to a Gemini 2.5 audio model and remove OpenAI-specific fields: ◦ Remove

input_audio_noise_reduction

. ◦ Remove any

enable_affective_dialog

/

proactivity

/ nonstandard turn-detection flags. 1. Ensure auth: ◦ For API key mode: set

api_key=deployment.api_key

and export

GOOGLE_API_KEY

in your container. ◦ For Vertex AI: set

vertexai=True

, add

project

and

location

, and mount service account JSON via

GOOGLE_APPLICATION_CREDENTIALS

. 1. Re-run and watch the plugin logs for the first WebSocket connect attempt (the invalid field is typically shown before the close). What I need to pinpoint this • The exact

google.realtime.RealtimeModel(...)

kwargs you’re passing (paste the snippet). • Which Gemini model string you used. • Whether you’re authenticating via API key or Vertex AI (and whether

GOOGLE_API_KEY

or

GOOGLE_APPLICATION_CREDENTIALS

is set in the container). Share those three items and I’ll give the precise corrected snippet and any container/env changes. Sources: Gemini Live API plugin | LiveKit Documentation, with interactive elements

Thanks — I’m responding to the Gemini 1007 error and the RealtimeModel snippet you posted. Problems in your snippet • You set

api_key="key"

but didn’t show the environment/auth wiring; Gemini expects a valid API key or Vertex credentials. • WebSocket 1007 usually means an invalid argument was sent to Gemini. Common causes are unsupported model options or mismatched modality/field names. Corrected minimal snippet (use this pattern)

llm = google.realtime.RealtimeModel(
    model="gemini-2.5-flash-native-audio-preview-12-2025",
    voice="Puck",
    modalities=["AUDIO", "TEXT"],
    api_key=deployment.api_key,    # use a real key from env/secret
    vertexai=False,                # set True + project/location if using Vertex
)

Actionable checks to fix 1007 • Ensure the model string is compatible (avoid Gemini 3.1 variants). • Remove any unsupported kwargs (e.g.,

input_audio_noise_reduction

,

enable_affective_dialog

,

proactivity

, or other non-Gemini fields). • Confirm auth: ◦ If using API key mode: set

api_key

and export it as

GOOGLE_API_KEY

inside the container. ◦ If using Vertex AI: set

vertexai=True

, provide

project

and

location

, and mount service account JSON as

GOOGLE_APPLICATION_CREDENTIALS

. • Verify

modalities

values match expected casing ("AUDIO","TEXT"). • Inspect the plugin logs for the first connect attempt — the invalid field is often logged just before the close. Quick ask so I can give an exact fix 1. Are you using API key mode or Vertex AI? 2. Paste the full kwargs you pass into

google.realtime.RealtimeModel(...)

(redact secrets). Send those and I’ll return the exact corrected snippet + the env/compose or k8s changes you need. , with interactive elements