I want to build voice insurance assistance agent. Tell me how can I build production ready agent. With custom llm from hugging face using livekit

Thanks for reaching out, Nikhil. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I want to use Hugging face inference APi. Python is language and front-end intergartion

Great — thanks for confirming. I'll give a focused, production-ready plan using the Hugging Face Inference API, Python backend (LiveKit Agents), and front-end integration. I’ll include architecture, concrete Python code snippets (OpenAI-compatible HF endpoint approach), Docker/deploy notes, STT/TTS recommendations, observability/testing checklist, and front-end integration tips. If any of these should be expanded first, tell me which: • Minimal end-to-end example (code + config) • Full Dockerfile + deployment steps for LiveKit Cloud • STT/TTS/VAD and turn-detection config and provider recommendations • Front-end integration with @livekit/components-react (voice UX + signaling) Which do you want me to deliver first?

Minimal end to end code with configuration.

Thanks — below is a minimal, end-to-end example to run a production-ready voice insurance assistant using the Hugging Face Inference API, LiveKit Agents in Python, and a simple React front end. It uses the OpenAI-compatible plugin approach so the HF inference endpoint can be used directly. Files included: • .env — credentials • requirements / install command • agent.py — minimal LiveKit Agents Python backend • token-server.py — lightweight token generator for frontend • frontend snippet (React) — join room & play agent audio • Dockerfile notes and run commands .env

LIVEKIT_URL=wss://<your-project>.livekit.cloud
LIVEKIT_API_KEY=<your_livekit_api_key>
LIVEKIT_API_SECRET=<your_livekit_api_secret>
HUGGINGFACE_API_KEY=<your_hf_api_key>
HUGGINGFACE_MODEL=<your-hf-model-id>    # e.g. your-username/your-model

Install (local dev)

uv add "livekit-agents[openai]~=1.3" python-dotenv
pip install python-dotenv

agent.py (minimal)

# agent.py
import os
import asyncio
from dotenv import load_dotenv
from livekit.agents import cli, AgentSession, Agent, JobContext
from livekit.plugins import openai, inference, silero

load_dotenv()

HF_API_KEY = os.getenv("HUGGINGFACE_API_KEY")
HF_MODEL = os.getenv("HUGGINGFACE_MODEL")
LIVEKIT_URL = os.getenv("LIVEKIT_URL")

class InsuranceAgent(Agent):
    def __init__(self):
        super().__init__(instructions="You are a helpful insurance assistant. Be concise and professional.")

    async def on_enter(self):
        # optional: auto-greet when session starts
        await self.session.say("Hello, I'm your insurance assistant. How can I help today?")

async def entrypoint(ctx: JobContext):
    await ctx.connect()  # wait for participant audio to arrive

    # Configure LLM to use Hugging Face Inference API (OpenAI-compatible base URL)
    llm = openai.LLM(
        model=HF_MODEL,
        api_key=HF_API_KEY,
        base_url="<https://api-inference.huggingface.co/v1>"
    )

    # Choose STT/TTS (use inference wrappers or specific providers)
    stt = inference.STT(model="deepgram/nova-3-general")          # replace if you have another STT
    tts = inference.TTS(model="cartesia/sonic-3")                # replace with preferred TTS

    # Optional: load VAD for better voice activity handling
    vad = silero.VAD.load()

    session = AgentSession(
        ctx=ctx,
        stt=stt,
        llm=llm,
        tts=tts,
        vad=vad
    )

    # attach session to agent and start
    agent = InsuranceAgent()
    await session.start(agent=agent, room=ctx.room)

if __name__ == "__main__":
    # Run the agent worker
    cli.run_app(entrypoint_fnc=entrypoint)

token-server.py (simple Flask token endpoint)

# token-server.py
from flask import Flask, jsonify
import os
from livekit_server_sdk import AccessToken, RoomGrant

app = Flask(__name__)
API_KEY = os.getenv("LIVEKIT_API_KEY")
API_SECRET = os.getenv("LIVEKIT_API_SECRET")

@app.route("/token/<identity>")
def token(identity):
    at = AccessToken(API_KEY, API_SECRET, identity=identity)
    grant = RoomGrant(room="insurance-room", room_join=True)
    at.add_grant(grant)
    return jsonify(token=at.to_jwt())

if __name__ == "__main__":
    app.run(port=3001)

Frontend (React minimal)

// VoiceAgent.jsx
import React, { useEffect, useRef } from "react";
import { Room } from "livekit-client";

export default function VoiceAgent({ token, url }) {
  const audioRef = useRef(null);

  useEffect(() => {
    let room = new Room();
    let connected = false;

    async function connect() {
      await room.connect(url, token, { audio: true, video: false });
      room.localParticipant.setMicrophoneEnabled(true);
      room.on("trackPublished", (pub, participant) => {
        // subscribe to remote audio tracks (agent audio)
        pub.setSubscribed(true);
      });

      room.participants.forEach(p => {
        p.audioTracks.forEach(t => {
          const track = t.track;
          if (track && track.kind === "audio") {
            const el = audioRef.current;
            const stream = new MediaStream();
            stream.addTrack(track.mediaStreamTrack);
            el.srcObject = stream;
            el.play();
          }
        });
      });

      room.on("participantConnected", p => {
        p.on("trackSubscribed", (track) => {
          if (track.kind === "audio") {
            const el = audioRef.current;
            const stream = new MediaStream();
            stream.addTrack(track.mediaStreamTrack);
            el.srcObject = stream;
            el.play();
          }
        });
      });

      connected = true;
    }

    connect();

    return () => {
      if (connected) room.disconnect();
    };
  }, [token, url]);

  return <audio ref={audioRef} autoPlay />;
}

How to run locally 1. Start token server: ◦ export LIVEKIT_API_KEY/SECRET ◦ python token-server.py 1. Start agent (developer mode, ensures it connects to LiveKit): ◦ export LIVEKIT_URL and HF env vars ◦ python agent.py dev 1. Frontend: fetch token from /token/<identity> and render VoiceAgent with token + LIVEKIT_URL. Docker notes (production) • Use multi-stage build, pre-download models if self-hosting; run as non-root user. • Ensure environment variables are injected via your orchestration (Kubernetes, ECS). • Use secure secret storage for HF and LiveKit keys, enable HTTPS, and rate-limit HF calls. Recommendations and production tips (brief) • Use a managed STT/TTS (Deepgram, AssemblyAI, ElevenLabs) for stability and quality. • Enable VAD and turn-detection to avoid interruptions. • Implement retries and a fallback prompt if HF inference fails or rate-limited. • Collect metrics: TTFB, TTFT, token usage, error rates. • Ensure PII handling: redact/store transcripts according to policy. If you want, I can next: • Provide a full Dockerfile and livekit.toml for LiveKit Cloud deployment, • Replace inference.STT/inference.TTS with concrete provider configs, • Show exact request/response mapping for HF chat completions (prompt templates). Which of those would you like next? Sources: Models | LiveKit Docs | Improved End-of-Turn Model Cuts Voice AI Interruptions 39% | LiveKit turn detector plugin | LiveKit docs

Give me production ready dockerfile for self hosted livekit

Below is a production-ready, multi-stage Dockerfile tailored for a self-hosted LiveKit Agents Python app that uses the Hugging Face Inference API. It follows best practices: small runtime image, non-root user, dependency locking, optional model pre-download step (for self-hosted models — commented), and healthcheck. I also include a minimal docker-compose.yml, env guidance, and a few deployment notes (TLS, scaling, secrets, monitoring). Dockerfile

# Stage 1: build / install dependencies
FROM <http://ghcr.io/astral-sh/uv:python3.13-bookworm-slim|ghcr.io/astral-sh/uv:python3.13-bookworm-slim> AS builder

# Create app user and workdir
ARG APP_USER=livekit
ARG APP_UID=10001
RUN adduser --disabled-password --gecos "" --uid ${APP_UID} ${APP_USER}

WORKDIR /app

# Copy lockfiles first for reproducible installs
COPY pyproject.toml poetry.lock ./

# Install dependencies into a wheel/cache to speed later builds
RUN uv sync --locked

# Copy application code
COPY . .

# (Optional) Pre-download models / assets (uncomment if self-hosting HF models or VAD/TTS files)
# ENV HF_MODEL_REPO="your-local-model-repo"
# RUN --mount=type=cache,target=/root/.cache/huggingface \
#     python -c "from transformers import AutoTokenizer, AutoModelForCausalLM; AutoTokenizer.from_pretrained('$HF_MODEL_REPO'); AutoModelForCausalLM.from_pretrained('$HF_MODEL_REPO')"

# Make sure app files are owned by non-root user
RUN chown -R ${APP_USER}:${APP_USER} /app

# Stage 2: runtime
FROM <http://ghcr.io/astral-sh/uv:python3.13-bookworm-slim|ghcr.io/astral-sh/uv:python3.13-bookworm-slim> AS runtime

ARG APP_USER=livekit
ARG APP_UID=10001
RUN adduser --disabled-password --gecos "" --uid ${APP_UID} ${APP_USER}

WORKDIR /app

# Copy installed deps and application from builder
COPY --from=builder /usr/local /usr/local
COPY --from=builder /app /app

# Ensure non-root
USER ${APP_USER}

ENV PYTHONUNBUFFERED=1
ENV PATH="/home/${APP_USER}/.local/bin:${PATH}"

# Expose port the app binds to (adjust if different)
EXPOSE 8080

# Healthcheck (adjust path/port to match your health endpoint)
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f <http://localhost:8080/health> || exit 1

# Final entry (use your app start command; `uv run` works with the uv base image)
CMD ["uv", "run", "agent:app", "--host", "0.0.0.0", "--port", "8080"]

docker-compose.yml (minimal, for self-hosted deployment)

version: "3.8"
services:
  livekit-agent:
    build: .
    image: livekit/insurance-agent:latest
    restart: unless-stopped
    environment:
      - LIVEKIT_URL=<wss://your-livekit-host>
      - LIVEKIT_API_KEY=${LIVEKIT_API_KEY}
      - LIVEKIT_API_SECRET=${LIVEKIT_API_SECRET}
      - HUGGINGFACE_API_KEY=${HUGGINGFACE_API_KEY}
      - HUGGINGFACE_MODEL=${HUGGINGFACE_MODEL}
      - UV_WORKERS=2
    ports:
      - "8080:8080"
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Environment / secrets guidance • Put secrets in your orchestration secret store (Kubernetes Secrets, Docker Swarm secrets, HashiCorp Vault) — do not bake them into images or plain .env files in production. • Required env vars: ◦ LIVEKIT_API_KEY, LIVEKIT_API_SECRET, LIVEKIT_URL ◦ HUGGINGFACE_API_KEY, HUGGINGFACE_MODEL ◦ Optional: STT/TTS provider keys, TELEPHONY credentials, REGION, SENTRY_DSN, PROMETHEUS config • Limit HF API exposure: if using HF Inference API, keep requests proxied and rate-limited from your backend. Deployment & scaling notes • Use a process supervisor / container orchestrator (Kubernetes, ECS, Nomad) rather than docker-compose for production. • Run multiple replicas behind a load balancer. Ensure sticky sessions or handle room assignment centrally if required. • Use horizontal autoscaling based on CPU, memory, and custom metrics (TTFB, token usage). • If self-hosting heavy ML models, provision GPU nodes and ensure the image includes CUDA/runtime layers (this Dockerfile uses CPU Python base). • For TLS: terminate TLS at the ingress/load balancer (NGINX ingress, ALB) — do not rely on container-level TLS for certificates. • Monitoring & logs: push metrics to Prometheus/Grafana; attach structured logs to ELK/Datadog; add health and readiness probes. • Security: run as non-root (done), drop unnecessary capabilities, enable resource limits, and run vulnerability scans on images. If you want, I can: • Provide a Kubernetes Deployment + Service + HPA + Secret manifests for this image. • Add GPU-enabled Dockerfile variants and instructions to pre-download large HF models into a shared volume. • Produce a complete livekit.toml and uv/worker tuning for optimal concurrency. Which of these should I provide next?

I want to use eleven labs and assembly ai. Also reduce background noise. And capture correct user transcript

Thanks — I’ll give a concise, production-ready Python + Docker setup that uses AssemblyAI for STT, ElevenLabs for TTS, enables noise reduction, and captures accurate transcripts. I’ll include environment vars, the agent code, updated Dockerfile snippets, and notes for reliable transcription. 1) .env (required)

LIVEKIT_URL=wss://<your-livekit-host>
LIVEKIT_API_KEY=<livekit_api_key>
LIVEKIT_API_SECRET=<livekit_api_secret>

HUGGINGFACE_API_KEY=<hf_api_key>
HUGGINGFACE_MODEL=<hf_model_id>

ASSEMBLYAI_API_KEY=<assemblyai_key>
ELEVENLABS_API_KEY=<elevenlabs_key>

LIVEKIT_AGENT_PORT=8080

2) Minimal production-ready agent (agent.py)

# agent.py
import os
from dotenv import load_dotenv
from livekit.agents import cli, AgentSession, Agent, JobContext
from livekit.plugins import openai, inference, assemblyai, elevenlabs, noise_cancellation, silero

load_dotenv()

HF_API_KEY = os.getenv("HUGGINGFACE_API_KEY")
HF_MODEL = os.getenv("HUGGINGFACE_MODEL")
ASSEMBLYAI_KEY = os.getenv("ASSEMBLYAI_API_KEY")
ELEVENLABS_KEY = os.getenv("ELEVENLABS_API_KEY")

class InsuranceAgent(Agent):
    def __init__(self):
        super().__init__(instructions="You are a helpful insurance assistant. Be concise and professional.")

    async def on_enter(self):
        await self.session.say("Hello — I'm your insurance assistant. How can I help?")

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    # LLM via Hugging Face Inference API (OpenAI-compatible plugin)
    llm = openai.LLM(model=HF_MODEL, api_key=HF_API_KEY, base_url="<https://api-inference.huggingface.co/v1>")

    # AssemblyAI STT plugin (streaming preferred)
    stt = assemblyai.STT(api_key=ASSEMBLYAI_KEY, model="assemblyai_standard", streaming=True)

    # ElevenLabs TTS plugin
    tts = elevenlabs.TTS(api_key=ELEVENLABS_KEY, voice="alloy")  # choose voice id as needed

    # Noise reduction: use built-in background noise cancellation module
    # and optional Silero VAD for turn detection
    noise = noise_cancellation.BackgroundVoiceCancellation()
    vad = silero.VAD.load()

    session = AgentSession(
        ctx=ctx,
        stt=stt,
        llm=llm,
        tts=tts,
        vad=vad,
        noise_cancellation=noise,
    )

    agent = InsuranceAgent()
    await session.start(agent=agent, room=ctx.room)

if __name__ == "__main__":
    cli.run_app(entrypoint_fnc=entrypoint)

Notes on this code: •

assemblyai.STT(...)

and

elevenlabs.TTS(...)

use LiveKit plugin interfaces — ensure

livekit-agents[assemblyai,elevenlabs,openai]

is installed. •

noise_cancellation.BackgroundVoiceCancellation()

enables background suppression before STT. • VAD + turn-detection ensures the session captures full user turns for accurate transcripts. 3) Install dependencies

uv add "livekit-agents[openai,assemblyai,elevenlabs]~=1.3" python-dotenv
pip install python-dotenv

4) Production Dockerfile adjustments (add to the Dockerfile you already have) • Ensure required system libs (ffmpeg) are present for audio handling. • Install plugins via uv sync and run as non-root. Example additions to Dockerfile (runtime stage)

# install ffmpeg for audio processing
RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg && rm -rf /var/lib/apt/lists/*

# copy app and environment
COPY --from=builder /app /app
WORKDIR /app
USER 10001

EXPOSE ${LIVEKIT_AGENT_PORT:-8080}
ENV PATH="/home/livekit/.local/bin:${PATH}"
CMD ["uv", "run", "agent:app", "--host", "0.0.0.0", "--port", "8080"]

5) docker-compose (snip) — include env and resource limits

services:
  livekit-agent:
    image: livekit/insurance-agent:latest
    environment:
      - LIVEKIT_URL=${LIVEKIT_URL}
      - LIVEKIT_API_KEY=${LIVEKIT_API_KEY}
      - LIVEKIT_API_SECRET=${LIVEKIT_API_SECRET}
      - ASSEMBLYAI_API_KEY=${ASSEMBLYAI_API_KEY}
      - ELEVENLABS_API_KEY=${ELEVENLABS_API_KEY}
      - HUGGINGFACE_API_KEY=${HUGGINGFACE_API_KEY}
      - HUGGINGFACE_MODEL=${HUGGINGFACE_MODEL}
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G

6) Runtime and reliability recommendations (actions to implement) • Use AssemblyAI streaming to get low-latency partial transcripts; only commit final transcript after end-of-turn. • Use VAD + turn-detection to detect end of user utterance so transcripts capture full sentences. • Apply

noise_cancellation.BackgroundVoiceCancellation()

before sending audio to AssemblyAI to reduce background noise. • Implement retry/backoff for HF and AssemblyAI API calls and a fallback short response when LLM/STT fails. • Log transcripts and tokens securely; redact PII according to your compliance needs. • Monitor metrics: transcription latency, STT error rate, TTS latency, LLM TTFB. 7) Transcript accuracy tips • Send preprocessed (noise-reduced, single-speaker) audio to AssemblyAI. • Use AssemblyAI features (language model selection, profanity filter off, enhanced punctuation) via the plugin options if available. • Capture partial results and reassemble final transcript at end-of-turn; keep timestamps for QA and analytics. If you want, I can now: • Provide the exact Kubernetes manifests (Deployment, Service, HPA, Secrets) for this image, • Produce a tested Dockerfile (full) with ffmpeg and plugin installation, • Show exact AssemblyAI plugin parameters for best transcription accuracy and ElevenLabs voice configuration. Which next? Sources: ElevenLabs TTS plugin guide | LiveKit docs | AssemblyAI plugin guide | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki | Pipeline nodes and hooks | LiveKit docs | Agents (Node.js) | livekit/livekit_composite | DeepWiki