I want a best approach for handling my room for ou...
# ask-ai
l
I want a best approach for handling my room for outbound caller. first when SIP particiapant connect to the our agent and communicating each other. then if manager wants to intrupt the call from frontend (he wants to speak directly to the SIP participant) then our agent only listens to the each participant and transcript the speech. how to do this with python
t
Thanks for reaching out, Rajadurai. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a clear and useful goal. I’m responding to your question about how to handle an outbound SIP call where: (1) a SIP participant and an agent are talking, (2) a manager can interrupt from the frontend and speak directly to the SIP participant, and (3) the agent then stops speaking and only listens and transcribes. Below is a concise, practical Python-first approach you can implement with LiveKit. High-level design (what you’ll build) • One LiveKit room holds the SIP participant, the agent service, and optionally the manager (frontend). • The agent runs as a program (AgentSession) that by default participates actively (STT + TTS). • When the manager requests intervention, the manager publishes an audio track into the same room and signals the agent to enter “intervention/transcription-only” mode. • In transcription-only mode the agent: disables outgoing audio (no TTS), keeps STT on, subscribes to the SIP participant and manager audio tracks, and writes per-participant transcripts. Key behaviors you must implement • Outbound SIP dialing into a LiveKit room. • Agent audio control: toggle output (speak) and input (listen). • Manager interrupt signaling (via RPC or data messages). • Per-participant subscription so transcripts are tagged with speaker identity. Concrete steps + Python patterns 1) Create room and dial outbound SIP • Use LiveKit Server SDK to create/ensure the room and dial the SIP target so a SIP participant joins that room. • When dialing, give the SIP side an identity (e.g., "caller_<id>") so transcripts can be attributed. 2) Start your AgentSession as a participant in the room • AgentSession should: ◦ Subscribe to audio tracks published in the room. ◦ Run STT (Deepgram/Whisper) on incoming audio frames. ◦ Publish audio output when acting (TTS) and disable it when in transcription mode. 3) Manager joins from frontend and signals intervention • Manager joins via WebRTC and publishes an audio track when they want to speak. • Use an RPC method or a data channel message to tell the agent to switch modes: ◦ Example RPC name:
enter_intervention_mode
(payload: { manager_id, target_participant }) • The manager audio is then audible to the SIP participant (because they’re in the same room). 4) Agent: implement mode switching (active ↔️ transcription-only) • Maintain a small state machine in your agent: ◦ active: publish TTS/audio; respond; normal conversation logic ◦ transcription_only: disable TTS output, keep STT running and log transcripts • Implementation patterns: ◦ Disable output publication: stop or mute agent’s audio track (so agent is silent). ◦ Keep input subscription: ensure you’re subscribed to audio tracks for SIP participant and manager. ◦ Tag transcripts using participant.identity. Minimal Python pseudo-flow for switching modes 1. On RPC `enter_intervention_mode`: ◦ agent.set_mode("transcription_only") ◦ agent.stop_publishing_audio() # mute / remove published audio track ◦ agent.ensure_subscriptions_to_all_audio_tracks() ◦ start transcribing frames and attach participant identity to each transcript 1. On RPC `exit_intervention_mode`: ◦ agent.set_mode("active") ◦ agent.start_publishing_audio() # re-enable TTS / audio output 5) Subscribing & transcribing per-participant audio • When a track is published: ◦ Subscribe and read audio frames from the track ◦ Send frames to your STT client tagged with the publisher’s identity • This lets you keep separate transcripts for the SIP participant and manager. Example subscription/transcription pattern (conceptual) • On track published: ◦ if track.kind == audio: ▪︎ subscription = await publication.subscribe() ▪︎ open an audio stream from subscription ▪︎ for each frame: send to STT, attach participant.identity to the resulting text What to watch out for • Latency: real-time transcription requires small chunk sizes and low-latency STT. • Echo and mixing: when manager speaks directly to SIP participant, ensure agent is muted to avoid feedback or duplicate speech. • Permissions: ensure the manager’s published audio is routed to SIP participant (room-based publication is usually enough). • Speaker attribution: rely on participant.identity assigned when publishing the track. Useful implementation notes (how this maps to LiveKit concepts) • Dialing SIP uses the server-side SIP/participant APIs to create the SIP participant in a room. • Mode switching is done by muting/unpublishing the agent’s output track or toggling
audio_enabled
. • Keep STT active in both modes; only stop any TTS or agent voice output during intervention. Would you like one of these next steps? • A full end-to-end Python code sample showing: room creation, dialing SIP, agent service that toggles modes, manager RPC handler, and per-participant transcription logging. • A focused snippet showing how to mute/unmute agent audio tracks and subscribe to participant audio tracks in Python. • Guidance on choosing an STT provider (Deepgram vs Whisper) for low-latency streaming. Tell me which next step you prefer and any details (STT provider, existing code, how manager will signal) and I’ll prepare the exact code you need. Sources: How to build an agent with speech-to-text input and text-only output | Live... | Agent speech and audio | LiveKit Documentation | Job lifecycle | LiveKit Documentation
l
give full python code
t
Thanks — I see you asked for a full end-to-end Python code example to run an outbound SIP call with LiveKit where: (1) a SIP participant and an agent converse, (2) a manager can interrupt from the frontend and speak directly to the SIP participant, and (3) the agent goes silent and transcribes both participants. Below is a concise, runnable example and notes to help you adapt it to your environment. Important: replace placeholders (API keys, SIP trunk ID, phone number) with your values before running. Full example (single-file) • This sample uses the LiveKit Python SDK patterns and shows: ◦ creating a room and making an outbound SIP call, ◦ running an agent session that toggles between active (TTS+STT) and transcription-only (STT-only) modes, ◦ a simple RPC/data signal handler a manager can call to trigger the intervention mode, ◦ per-participant transcription logging.
Copy code
# file: outbound_sip_agent.py
import os
import asyncio
import logging
from typing import Dict
from dataclasses import dataclass

from dotenv import load_dotenv

# LiveKit imports (install with `pip install livekit`)
from livekit import api
from livekit.cluster import LivekitCluster  # optional depending on SDK version
from livekit.agents import AgentSession  # conceptual; adapt to your livekit-agents package
from livekit.rtc import ParticipantKind, TrackKind

load_dotenv()
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
logger = logging.getLogger("outbound_sip_agent")

# --- Configuration (edit these) ---
LIVEKIT_API_KEY = os.getenv("LIVEKIT_API_KEY")
LIVEKIT_API_SECRET = os.getenv("LIVEKIT_API_SECRET")
LIVEKIT_URL = os.getenv("LIVEKIT_URL")  # e.g. <https://your.livekit.cloud>
SIP_TRUNK_ID = os.getenv("SIP_TRUNK_ID") or "ST_YOUR_OUTBOUND_TRUNK_ID"
CALLER_NUMBER = os.getenv("CALLER_NUMBER") or "+15105550123"
ROOM_NAME = os.getenv("ROOM_NAME") or f"outbound-call-{int(asyncio.get_event_loop().time())}"
# -------------------------------

@dataclass
class ParticipantState:
    identity: str
    kind: ParticipantKind
    is_manager: bool = False

class AgentController:
    def __init__(self, room_name: str):
        self.room_name = room_name
        self.session = None  # AgentSession analog
        self.participants: Dict[str, ParticipantState] = {}
        self.in_intervention = False

    async def start(self):
        # Create a LiveKit API client
        self.lkapi = api.LiveKitAPI()  # adapt constructor if SDK requires url/key/secret
        <http://logger.info|logger.info>("LiveKit API client created")

        # Create room (idempotent)
        try:
            await self.lkapi.create_room(self.room_name)
            <http://logger.info|logger.info>(f"Room created or exists: {self.room_name}")
        except Exception:
            <http://logger.info|logger.info>("Room creation skipped or failed (may already exist)")

        # Dial outbound SIP into the room
        try:
            sip_req = {
                "room_name": self.room_name,
                "sip_trunk_id": SIP_TRUNK_ID,
                "sip_call_to": CALLER_NUMBER,
                "participant_identity": f"sip_{CALLER_NUMBER.replace('+','').replace('-','')}",
                # Depending on SDK, parameters may differ; adapt accordingly.
            }
            sip_participant = await self.lkapi.create_sip_participant(**sip_req)
            <http://logger.info|logger.info>(f"Dialed SIP participant, participant sid: {getattr(sip_participant,'sid', 'unknown')}")
        except Exception as e:
            logger.exception("Failed to create SIP participant (check SIP_TRUNK_ID, credentials): %s", e)
            return

        # Start agent session (conceptual: create AgentSession with STT, TTS, VAD)
        # Replace with your actual AgentSession initialization from your livekit-agents package
        self.session = AgentSession(
            room_name=self.room_name,
            stt_provider="deepgram",    # placeholder; instantiate your STT class
            tts_provider="elevenlabs",  # placeholder; instantiate your TTS class
        )
        # Register callbacks for incoming transcripts and participant events
        self.session.on_participant_published(self._on_participant_published)
        self.session.on_transcript(self._on_transcript)
        self.session.on_rpc(self._on_rpc)

        # Start session (connect to room, publish agent audio if needed)
        await self.session.start()
        <http://logger.info|logger.info>("Agent session started in active (speak+listen) mode")

        # Agent initial greeting (TTS spoken into room)
        await self.session.speak("Hello, this is your service agent. How can I help you today?")

        # Keep running until cancelled
        try:
            while True:
                await asyncio.sleep(1)
        except asyncio.CancelledError:
            <http://logger.info|logger.info>("Shutting down agent session")
        finally:
            await self.session.stop()
            await self.lkapi.aclose()

    async def _on_participant_published(self, participant, publication):
        # Track participants and subscribe to audio for transcription
        identity = participant.identity
        kind = participant.kind
        self.participants[identity] = ParticipantState(identity=identity, kind=kind, is_manager=(kind != ParticipantKind.PARTICIPANT_KIND_SIP))
        <http://logger.info|logger.info>(f"Participant published: {identity} (kind={kind})")

        # Ensure session subscribes to audio track so we can transcribe
        if publication.track_kind == TrackKind.KIND_AUDIO:
            await self.session.subscribe_to_track(participant, publication)

    async def _on_transcript(self, transcript):
        # Transcript object expected to contain: text, participant_identity, is_final
        identity = getattr(transcript, "participant_identity", "unknown")
        text = transcript.text.strip()
        if not text:
            return

        tag = "Manager" if self.participants.get(identity, None) and self.participants[identity].is_manager else "Customer"
        <http://logger.info|logger.info>(f"[{tag}] {identity}: {text}")

        # If in intervention mode, keep transcribing. If not, agent can respond when appropriate.
        if not self.in_intervention and transcript.is_final and tag == "Customer":
            # Example: generate a short reply using LLM -> TTS
            reply = await self.session.generate_reply_for_text(text)
            if reply:
                await self.session.speak(reply)

    async def _on_rpc(self, rpc_name, payload, from_participant):
        # Simple RPC handler that toggles intervention mode from frontend manager
        <http://logger.info|logger.info>(f"RPC received: {rpc_name} from {from_participant}")
        if rpc_name == "enter_intervention_mode":
            await self.enter_intervention_mode(from_participant, payload)
        elif rpc_name == "exit_intervention_mode":
            await self.exit_intervention_mode(from_participant, payload)

    async def enter_intervention_mode(self, manager_identity, payload):
        if self.in_intervention:
            return
        <http://logger.info|logger.info>(f"Manager {manager_identity} requested intervention: {payload}")
        self.in_intervention = True

        # Stop any agent TTS/speaking and ensure agent remains only listening
        await self.session.interrupt()                # stop current speech actions
        await self.session.set_output_enabled(False)  # mute agent's published audio
        await self.session.set_input_enabled(True)    # keep listening

        # Optionally update agent instructions to "listen-only"
        await self.session.update_system_message("You are now in listening mode. Do not speak; only transcribe.")

    async def exit_intervention_mode(self, manager_identity, payload):
        if not self.in_intervention:
            return
        <http://logger.info|logger.info>(f"Manager {manager_identity} exiting intervention: {payload}")
        self.in_intervention = False

        # Re-enable agent output so it can speak again
        await self.session.set_output_enabled(True)
        await self.session.update_system_message("You may resume speaking to the caller.")

# --- Run the controller ---
async def run():
    controller = AgentController(ROOM_NAME)
    await controller.start()

if __name__ == "__main__":
    try:
        asyncio.run(run())
    except KeyboardInterrupt:
        <http://logger.info|logger.info>("Stopped by user")
How this maps to your requirements • Outbound SIP call: the code calls the LiveKit API to create a SIP participant in the room (check the actual API parameters in your LiveKit SDK—method names may differ). • Manager interrupt: manager triggers RPC
enter_intervention_mode
(this should be called from your frontend using LiveKit Data or RPC API). The agent reacts by muting its audio output and remaining listening-only. • Agent transcription-only: STT remains enabled; transcripts are logged with participant identity and manager/customer tags. • Agent resumes after manager exits intervention: manager triggers
exit_intervention_mode
(or you implement an automatic timeout/VAD-based end-of-speech detection). Practical steps to complete and run this example 1. Install packages: ◦ pip install livekit python-dotenv ◦ Also install any STT/TTS/agent packages you use (e.g., livekit-agents, deepgram client, whisper, ElevenLabs SDK). 1. Fill environment variables: ◦ LIVEKIT_API_KEY, LIVEKIT_API_SECRET, LIVEKIT_URL, SIP_TRUNK_ID, CALLER_NUMBER 1. Adapt AgentSession: ◦ The sample uses a conceptual
AgentSession
API. Replace it with the actual classes/functions from your
livekit-agents
package and your chosen STT/TTS providers. Key Sources: Agents telephony integration | LiveKit Documentation | Outbound Calling Script | LiveKit Documentation | Making calls overview | LiveKit Documentation | examples/voice_agents | livekit/livekit_composite | DeepWiki | Text and transcriptions | LiveKit Documentation