https://livekit.io logo
Join Slack
Powered by
Hello, Can someone please guide me on how to integ...
# ask-ai
f
Hello, Can someone please guide me on how to integrate a custom LLM with LiveKit? I’m looking to use the bitext/Mistral-7B-Insurance model and want to understand if it’s possible to connect this model with LiveKit for building a voice agent in the insurance domain. Any guidance, documentation, or examples would be greatly appreciated. Thanks in advance!
t
Thanks for reaching out, Nikhil. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I can help with that. You're asking how to connect bitext/Mistral-7B-Insurance to LiveKit to build a voice agent. I can walk you through two practical approaches and help pick the right one for your setup. Which of these matches how you’ll host the model? • Your model is exposed via an OpenAI-compatible HTTP API (supports /chat/completions and streaming). • Your model is exposed via a custom API or SDK (non-OpenAI format) and needs custom call/stream handling. • You plan to run the model locally (self-hosted) and can wrap it with an API. If you tell me which option fits, I’ll provide the exact code snippets and configuration steps (STT/TT S, VAD, keys, streaming tips) for integrating with LiveKit. Sources: Using Custom LLM Providers with the Agent | Live Conversations with AI using ChatGPT and WebRTC | Large language models (LLM) | LiveKit Docs
f
Yes I am calling this model through hugging face api
t
Great — thanks for confirming you're using the Hugging Face API. Below are two practical options to integrate bitext/Mistral-7B-Insurance with LiveKit for a voice agent, plus a concrete example you can drop into an agent by overriding 
llm_node
. Which option you choose depends on whether you need streaming from Hugging Face. Hugging Face supports streaming via the inference/websocket or streaming responses from certain endpoints — if you can stream, the UX will be much smoother (lower latency). If not, you can still use non-streaming responses but will have slightly higher TTS/turn latency. Option A — Use an OpenAI-compatible wrapper (if you have one) • Wrap your Hugging Face model behind an OpenAI-compatible endpoint (or use a proxy that translates /chat/completions) so you can reuse LiveKit’s OpenAI plugin by pointing its 
base_url
to your wrapper. • Advantages: Minimal LiveKit code changes; leverages existing streaming handling if the wrapper supports it. • Use this if you already have or can deploy a small proxy service. Option B — Implement a custom 
llm_node
that calls Hugging Face directly (recommended if you call Hugging Face API) • Fully controlled approach: call the Hugging Face inference API (REST or streaming WebSocket), yield streamed chunks as LiveKit expects, and handle auth, retries, and tokenization specifics yourself. • This is the approach below with sample code. Sample: custom 
llm_node
calling Hugging Face inference (async, streaming-friendly) • Goals covered: ◦ Authenticate with Hugging Face API key ◦ Send conversation prompt as needed ◦ Stream tokens/chunks back into LiveKit by yielding 
llm.ChatChunk
objects ◦ Handle non-streaming fallback Example (Python-ish pseudocode for an Agent subclass):
Copy code
import aiohttp
import asyncio
from livekit.agents import Agent, llm

HUGGINGFACE_API_KEY = "hf_XXXXXXXX"

class InsuranceAgent(Agent):
    def __init__(self):
        super().__init__(instructions="You are an insurance expert assistant.")

    async def llm_node(self, chat_ctx, tools, model_settings=None):
        # Build the prompt from chat_ctx.messages (adjust formatting to your model)
        prompt = ""
        for msg in chat_ctx.messages:
            role = msg.role  # e.g., "user" or "assistant"
            content = msg.content
            prompt += f"{role}: {content}\n"
        prompt += "assistant:"

        # Hugging Face inference endpoint for the model
        url = "<https://api-inference.huggingface.co/models/bitext/Mistral-7B-Insurance>"

        headers = {
            "Authorization": f"Bearer {HUGGINGFACE_API_KEY}",
            "Accept": "text/event-stream"  # request streaming (if supported)
        }

        # Try streaming via SSE (event-stream). If HF doesn't stream, fall back.
        try:
            async with aiohttp.ClientSession() as session:
                async with <http://session.post|session.post>(
                    url,
                    headers=headers,
                    json={"inputs": prompt, "parameters": {"max_new_tokens": 512, "temperature": 0.2}},
                    timeout=120
                ) as resp:
                    # If HF returns SSE/chunked streaming, process incrementally
                    if resp.status == 200 and resp.content_type == "text/event-stream":
                        async for raw_chunk in resp.content:
                            chunk_text = raw_chunk.decode(errors="ignore").strip()
                            if not chunk_text:
                                continue
                            # Parse SSE lines to extract token/text (HF SSE format may vary)
                            # Simple heuristic: yield each non-empty line as a ChatChunk
                            yield llm.ChatChunk(text=chunk_text)
                    else:
                        # Non-streaming response: read whole body
                        data = await resp.json()
                        # HF response format: either dict with 'generated_text' or list
                        text = ""
                        if isinstance(data, dict) and "generated_text" in data:
                            text = data["generated_text"]
                        elif isinstance(data, list) and len(data) > 0 and "generated_text" in data[0]:
                            text = data[0]["generated_text"]
                        else:
                            # fallback: stringify
                            text = str(data)
                        yield llm.ChatChunk(text=text)
        except Exception as e:
            # On error, yield an error message as a chunk or raise
            yield llm.ChatChunk(text=f"[LLM error] {str(e)}")
Practical tips and considerations • Streaming: If you can use Hugging Face's streaming (SSE/websocket) endpoint, parse the event stream and yield tokens quickly to avoid TTS latency. If not, return the full text and let TTS play after generation. • Prompt formatting: Format prompts to the model’s expected chat or text-instruction format. Test locally to ensure quality in insurance domain. • Safety & filtering: Add policy/safety checks after generation before sending to users (especially in insurance). • STT / TTS: Configure LiveKit’s STT and TTS nodes (or external services) to transcribe user audio and synthesize the LLM output. Keep chunk sizes reasonable for TTS to avoid many tiny utterances. • Retries & timeouts: Hugging Face inference can be slower for large models; implement retries, exponential backoff, and timeouts. • Rate limits & batching: Respect Hugging Face rate limits; consider batching or a small queue if you expect many simultaneous users. • Local wrap: If latency is critical, consider self-hosting the model and exposing a streaming-compatible endpoint (or use a local wrapper like vLLM) then call it as above. Next step Would you like: • A full runnable example that includes STT/TTS node wiring and LiveKit session setup? • A version of 
llm_node
that uses Hugging Face websocket streaming (I can produce a websocket example if you confirm you can use the HF WebSocket SSE endpoint)? • Help formatting prompts for insurance — sample instruction + prompt templates? Tell me which follow-up you want and I’ll provide the code and config.
f
Copy code
noise = noise_cancellation.BackgroundVoiceCancellation()

Cannot find reference 'BackgroundVoiceCancellation' in '__init__.py' 

ai_session = AgentSession(
        stt=stt,
        llm=llm,
        tts=tts,
        vad=vad,
        turn_detection="vad",
        noise_cancellation=noise
    )
Unexpected argument noise_cancellation
t
Thanks — I see two problems you hit: 
BackgroundVoiceCancellation
isn't available, and 
AgentSession
doesn't accept a 
noise_cancellation
parameter. Here's the correct approach and a minimal example. Short answer • 
BackgroundVoiceCancellation
is not a valid class. Use 
noise_cancellation.BVC()
or 
noise_cancellation.BVCTelephony()
instead. • Configure noise cancellation via 
room_options
(inside 
AudioInputOptions
) when starting the session — do not pass 
noise_cancellation
to 
AgentSession
constructor. Fix (minimal example)
Copy code
from livekit.agents import AgentSession, room_io
from livekit.plugins import noise_cancellation, silero  # or other VAD

session = AgentSession(
    stt=stt,
    llm=llm,
    tts=tts,
    vad=silero.VAD.load(),
    turn_detection="vad",
)

await session.start(
    room=ctx.room,
    agent=agent,
    room_options=room_io.RoomOptions(
        audio_input=room_io.AudioInputOptions(
            noise_cancellation=noise_cancellation.BVC(),  # or BVCTelephony()
        ),
    ),
)
Notes and tips • Use 
BVC()
for general background cancellation, 
BVCTelephony()
for telephony-optimized cancellation. • Keep 
AgentSession
args limited to stt/llm/tts/vad/turn_detection — audio noise settings go in 
room_options
. • If you still get import errors, ensure your installed LiveKit package version includes 
noise_cancellation
plugin (upgrade to the latest release). Would you like a websocket/SSE streaming example for your Hugging Face LLM or a runnable repo example that ties STT → llm_node → TTS with this noise setup? Sources: Voice AI quickstart | LiveKit docs | Google AI and LiveKit | LiveKit Docs | Agent session | LiveKit docs
f
livekit.agents._exceptions.APIStatusError: Error code: 403 - {'error': 'This authentication method does not have sufficient permissions to call Inference Providers on behalf of user Nikhil8087'} (status_code=403, request_id=Root=1-694f94a1-5b4873f11d29ccfd122fadef, body=This authentication method does not have sufficient permissions to call Inference Providers on behalf of user Nikhil8087, retryable=True)
t
Thanks — I see the 403: your token/auth method lacks permission to call Inference Providers on behalf of user Nikhil8087. I can help resolve this; a couple quick questions so I give exact steps: 1. How are you generating the token/API key used by your agent? (server-side JWT using SDK, API key from dashboard, or a third‑party proxy?) 2. Are you using a short-lived JWT created with a LiveKit API key/secret, or a static API key from the dashboard? What to check next (do these while I wait for your answers): • Ensure the token includes the proper grants for inference usage. Tokens must be created server-side with the LiveKit API key/secret and include the appropriate permissions (the server-side JWT flow is required to act on behalf of a user). • If you use a dashboard/static key, verify that key has inference/agent-related permissions enabled in your LiveKit project settings. • If you create JWTs yourself, make sure the claims include the necessary room/user grants (the server-side flow with the API secret is required to call Inference Providers on behalf of a user). If you prefer, paste the code snippet (redact secrets) you use to create the token and the exact client call that triggers the 403 — I’ll point out which claim or setting to change and provide a corrected token example. Sources: Server-side | livekit/livekit_composite | DeepWiki | Using Custom LLM Providers with the Agent | Agent session | LiveKit docs
Open in Slack
PreviousNext
Powered by