Anyone has guidance on usng Krisp with self hosted Livekit agents, by accessing the raw audio frames and adding the Krisp processing before pushing it to the room?

Thanks for reaching out, Sanskriti. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Python

Great — since you’re using Python, here’s a focused guide and example for both common flows. I’ll start with the recommended plugin approach, then show how to manually access and publish frames if you need custom processing. Recommended: use LiveKit’s noise_cancellation plugin (simpler, optimized) 1. Install plugin: add

livekit-plugins-noise-cancellation

to your environment. 2. Wrap incoming tracks with

AudioStream(..., noise_cancellation=...)

so frames are processed automatically. 3. Forward processed frames to STT or publish a new local track. Example — process remote audio with Krisp/BVC and forward to STT (Python):

from livekit import rtc
from livekit.agents import JobContext, WorkerOptions, cli
from livekit.plugins import noise_cancellation, openai
from livekit.agents.voice import room_io

@room_io.room.on("track_subscribed")
async def on_track_subscribed(track: rtc.RemoteAudioTrack, participant: rtc.RemoteParticipant):
    # Wrap the remote track with Background Voice Cancellation
    stream = rtc.AudioStream(track, noise_cancellation=noise_cancellation.BVC())

    # Stream processed frames into STT or any downstream consumer
    stt_stream = openai.STT().stream()
    async for frame_event in stream:
        stt_stream.push_frame(frame_event.frame)
    await stt_stream.end_input()

async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=rtc.AutoSubscribe.AUDIO_ONLY)
    await ctx.room.participant.wait_for_participant()

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Notes: • Use

NC()

for standard noise cancellation,

BVC()

for background voice cancellation, or

BVCTelephony()

if telephony-tuned. • Don’t enable noise cancellation both in frontend and agent — avoid double-processing. • Plugin handles raw PCM frames and model downloads; ensure the agent environment has network access. Advanced: manual frame access + publish processed audio Use this if you must run an external Krisp binary or perform custom DSP: 1. Create an

AudioStream

without the plugin to read raw frames. 2. Create an

AudioSource

and a

LocalAudioTrack

. 3. Process each

frame.data

(raw PCM int16) with your custom routine and capture into the source. Example skeleton:

stream = rtc.AudioStream(track)  # no plugin
source = rtc.AudioSource(track.sample_rate, track.num_channels)
processed_track = rtc.LocalAudioTrack.create_audio_track("cleaned", source)
await ctx.room.local_participant.publish_track(processed_track)

async for frame_event in stream:
    raw = frame_event.frame.data        # raw PCM int16 bytes
    cleaned = your_process(raw)        # call into Krisp binary/lib -> returns PCM bytes
    await source.capture_frame(cleaned)

Questions to finalize the snippet for your environment: • Are you processing remote tracks (like above) or local microphone input before publish? • Do you want to use the LiveKit plugin or an external Krisp library/binary for processing? Tell me which flow you want and whether you prefer plugin vs external Krisp; I’ll provide the exact install commands and a tailored snippet.

Capture and use seperaye Kisp as its self hosted

Thanks — you want to capture raw frames in a Python LiveKit agent, send them to a separate self‑hosted Krisp instance for processing, then publish the cleaned audio to the room. Below is a concise, runnable pattern and key considerations. 1) High‑level flow • Subscribe to the remote/local track and read raw PCM frames from an AudioStream (no noise plugin). • Send frames to your self‑hosted Krisp service (over IPC, socket, or HTTP) and receive cleaned PCM frames. • Feed cleaned frames into an AudioSource and publish a LocalAudioTrack. 2) Minimal example (skeleton)

from livekit import rtc
from livekit.agents import JobContext, WorkerOptions, cli
from livekit.agents.voice import room_io

async def process_with_krisp(raw_pcm_bytes) -> bytes:
    # Implement sending raw_pcm_bytes to your self-hosted Krisp and return cleaned PCM bytes.
    # Example: send over a TCP socket or HTTP request and return response.content
    pass

@room_io.room.on("track_subscribed")
async def on_track_subscribed(track: rtc.RemoteAudioTrack, participant: rtc.RemoteParticipant):
    # Read raw frames (no LiveKit noise plugin)
    stream = rtc.AudioStream(track)

    # Create a source and local track to publish cleaned audio
    source = rtc.AudioSource(track.sample_rate, track.num_channels)
    cleaned_track = rtc.LocalAudioTrack.create_audio_track("krisp_cleaned", source)
    await room_io.room.local_participant.publish_track(cleaned_track)

    async for frame_event in stream:
        raw_bytes = frame_event.frame.data          # raw PCM int16 bytes
        cleaned_bytes = await process_with_krisp(raw_bytes)
        await source.capture_frame(cleaned_bytes)

3) Key implementation details • Frame format:

frame.data

is raw PCM (int16). Preserve sample rate and channel count when sending to Krisp and when capturing back into

AudioSource

. • Latency: Batch frames if your Krisp endpoint benefits from larger chunks to reduce round‑trip overhead. Aim to keep end‑to‑end latency low for real‑time use. • Threading/async: Use async I/O or a worker pool so Krisp processing doesn’t block the audio loop. • Buffering & jitter: Implement a small jitter buffer to smooth variable processing time; drop frames if backlog grows too large to avoid unbounded latency. • Avoid double processing: Ensure no frontend or other agents are also applying noise cancellation. • Encoding: Keep PCM raw during processing; only encode if required by transport. If your Krisp service expects floats, convert appropriately. 4) Reliability & perf tips • Measure throughput: frames/sec and processing time per chunk; tune chunk size accordingly. • Backpressure: if Krisp is overloaded, either fallback to passthrough or apply lighter local filtering. • Resource sizing: Krisp models can be CPU/GPU intensive—monitor CPU and memory and scale workers accordingly. If you’d like, tell me: • how your Krisp service accepts audio (HTTP chunk, TCP stream, gRPC, filesystem, etc.), and • whether you’re processing local mic capture or remote participant tracks. I’ll produce a complete code example wired to that transport (e.g., HTTP POST chunk or TCP socket) and include recommended chunking and buffering parameters.