i'm exploring the option of self hosting TTS, STT and LLM, is it possible with livekit?

Thanks for reaching out, Aman. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i'm looking for model which can support hindi, english and hinglish

Thanks — you're asking for models that handle Hindi, English, and Hinglish for self-hosting with LiveKit. I can help — here’s a concise recommendation and next step options. Quick recommendations • STT (Hinglish/code-switching): try Whisper (self-hosted) or Gladia configured for code-switching. Both can handle English+Hindi mixtures; Gladia has explicit code-switching features. • TTS (Hindi + mixed output): ElevenLabs multilingual models are a good starting; if you need fully open-source, consider Tacotron/Coqui TTS variants and fine-tune or use transliteration for mixed text. • LLM (Hinglish text processing): use an OpenAI‑compatible LLM you can self-host (e.g., a locally hosted GPT-style model with multilingual training data). GPT-family models handle Hinglish well when exposed to mixed-language prompts. • Turn detection / voice activity: LiveKit’s MultilingualModel supports Hindi and mixed-language conversations and is suitable for interruption/turn-detection. What I need to give exact setup steps • Which components do you want to self-host first (STT, TTS, LLM — or all three)? • Any preferred models/runtimes (Whisper, OpenFL, LlamaX, Coqui, ElevenLabs, etc.) or open-source vs commercial preference? • Will your model endpoints expose OpenAI‑compatible APIs, or do you need help wrapping custom endpoints? Tell me those details and I’ll provide the specific Agent plugin configs and example config snippets to connect your self-hosted endpoints to LiveKit. Sources: TTS Translator with Gladia STT | LiveKit docs | Pipeline Translator Agent | LiveKit docs | Improved End-of-Turn Model Cuts Voice AI Interruptions 39% | TTS Metrics | LiveKit docs

how to use self hosted whisper for stt

Thanks — I’ll walk you through using a self‑hosted Whisper instance for STT with LiveKit Agents and provide a minimal example. What I’m addressing: how to connect a self‑hosted Whisper server (whisper.cpp, faster‑whisper, or any OpenAI‑compatible wrapper) to LiveKit Agents so audio from a room is transcribed. Prerequisites • A running Whisper server that accepts HTTP requests (e.g., whisper.cpp server, whisper-serving, or faster-whisper) — note the transcription endpoint URL and any auth headers. • Python environment with LiveKit Agents and Whisper plugin:

pip install "livekit-agents[whisper]"

Steps (high level) 1. Start your Whisper server and confirm the transcription endpoint (example: `http://localhost:8000/v1/audio/transcriptions`). 2. Create a Whisper STT plugin instance that points to your endpoint. 3. Create an AgentSession with that STT instance and start the session in your LiveKit room. 4. Optionally add VAD (Silero) for better turn-detection and to reduce unnecessary transcriptions. Minimal Python example

import asyncio
from livekit.agents import AgentServer, AgentSession, JobContext
from livekit.plugins import whisper

server = AgentServer()

# Point to your self-hosted Whisper endpoint
stt = whisper.WhisperSTT(endpoint="<http://localhost:8000/v1/audio/transcriptions>")

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(stt=stt)

    @session.on("user_input_transcribed")
    def on_transcript(transcript):
        if transcript.is_final:
            print("Final transcript:", transcript.text)

    await session.start(room=ctx.room)
    await ctx.connect()

if __name__ == "__main__":
    server.run()

Notes and tips • Endpoint shape: if your server uses a non‑standard API, extend the STT base class or wrap requests so the plugin can POST audio and parse responses. • Streaming/latency: use a Whisper server that supports streaming or chunked requests for lower latency; quantized models or GPU inference reduce latency. • Language and model: choose a Whisper model size appropriate for accuracy vs CPU/GPU cost. Larger models give better Hindi/Hinglish accuracy. • Authentication: if your Whisper server requires auth, ensure your wrapper adds headers (you may need to modify the plugin instance to pass auth). • VAD: adding Silero VAD improves efficiency by sending audio only when voice is detected. If you want, tell me: • which Whisper server you’re running (whisper.cpp, faster-whisper, or a specific wrapper), • the endpoint URL and whether it needs auth, and I’ll give the exact config snippet or a small custom STT adapter you can drop into your Agent. Sources: An open source stack for real-time multimodal AI | Agents Framework | livekit/livekit_composite | DeepWiki | Transcriber | LiveKit docs | STT Metrics | LiveKit docs | LiveKit Docs