does the stt node only run when theres vad detected or all the time when the livekit agent is active?

Thanks for reaching out, Sujal. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

is it possible to send data to the livekit agent that is not directly from speech? I want another model that is an audio noise classification model that pairs alongside my elevenlabs tts model to understand noises that are happening additionally, i have another source of information that i would like to pass to the agent that is async to the audio and is only sent sparsely. think of it like additional context for the agent.

Thanks — I’m responding to your two questions: (1) can you send non-speech audio (noise classification) alongside speech, and (2) can you send sparse, async context to the agent. Short answer: Yes to both. How to do each (high level) • Non-speech audio / noise classification ◦ Publish the noise audio as a separate audio track (or subscribe to the same track and read raw frames). ◦ In the agent, wrap the track with

rtc.AudioStream

and run your noise-classification model on incoming `AudioFrame`s in parallel with the STT pipeline. ◦ This runs independently of VAD/STT, so you can classify environmental sounds continuously or on whatever cadence you need. • Sparse, asynchronous context ◦ Send context via a byte stream, a text/data channel topic, or by invoking a custom RPC on the agent. ◦ On the agent, register a byte-stream handler, listen for the specific text/data topic, or register an RPC method to receive and integrate the data into the agent’s state or chat context. ◦ Use these updates to influence the agent’s decisions, prompts, or future responses. If you want, I can: • Provide a short code example (agent-side) for subscribing to raw audio frames and calling a noise classifier. • Show a snippet for registering a byte-stream handler or RPC method and how to merge the received context into the agent’s chat state. Which snippet would be most helpful: noise-classifier audio processing, byte-stream/RPC receive, or both? Sources: Capturing the First Few Seconds of a Call for Processing | Vision | LiveKit docs | External data and RAG | LiveKit docs | Text and transcriptions | LiveKit docs | Job lifecycle | LiveKit docs

can you give me the specific pages on the livekit website that talk about this?

Thanks — I’m responding to your request for the specific LiveKit docs pages that describe (1) processing raw audio tracks / sending non-speech audio, and (2) sending asynchronous/sparse context (byte streams, RPC, text streams). Relevant LiveKit docs pages: • Processing raw media tracks (capturing audio frames / AudioStream): https://docs.livekit.io/agents/server/job/#processing-raw-media-tracks • Passing data to a job (job metadata / room metadata / participant attributes): https://docs.livekit.io/agents/server/job/#passing-data-to-a-job • Frontend → Agent RPC / external data: https://docs.livekit.io/agents/logic/external-data/ • Receiving text streams / non-transcription text topics: https://docs.livekit.io/agents/multimodality/text/#receiving-text-streams • STT node / standalone STT usage: https://docs.livekit.io/agents/build/nodes/#stt_node • Byte streams (send arbitrary binary/text data): https://docs.livekit.io/transport/data/byte-streams/ • Playing/publishing custom audio from an agent: https://docs.livekit.io/recipes/playing_audio/ Would you like a short code snippet for: (A) subscribing to raw audio frames and running a noise classifier, or (B) registering a byte-stream handler / RPC method to receive sparse context? Sources: Capturing the First Few Seconds of a Call for Processing | Job lifecycle | LiveKit docs | External data and RAG | LiveKit docs | Speech-to-text (STT) models | LiveKit Docs | Playing Audio | LiveKit docs

You seem to be getting confused about LiveKit being used between participants and LiveKit being used with an agent. Because I want to pre-process the audio being captured by the agent for potential noises, and then for passing data I wanted to be sent to an agent not another participant

Thanks — I see the distinction you’re making: you want to preprocess audio inside the agent (not route it to other participants) and send sparse, async data directly to the agent (not to other participants). That’s clear — yes, both are possible. Quick plan and exact options: 1) Preprocess audio inside the agent (noise classification) • Have the agent subscribe to the incoming participant audio track(s) and wrap the track with

rtc.AudioStream

to access `AudioFrame`s inside the agent process. • Run your noise-classification model on those frames in parallel with the STT node (VAD+STT remains gated by VAD; your classifier can run continuously or on your own cadence). • Use the classification outputs to update the agent’s internal state (e.g., session context, flags) so the agent’s logic/LLM can incorporate noise info without sending it to other participants. Docs: • Processing raw media tracks / AudioStream: https://docs.livekit.io/agents/server/job/#processing-raw-media-tracks • STT node (for how STT integrates with VAD): https://docs.livekit.io/agents/build/nodes/#stt_node 2) Sending sparse, asynchronous data directly to the agent (not to participants) Options you can use from your external system to push context into the agent runtime: • Byte streams: send arbitrary JSON/binary to a named byte-stream topic; register a byte-stream handler in the agent to receive it. ◦ Docs: https://docs.livekit.io/transport/data/byte-streams/ • Agent RPC methods: register an RPC on the agent (e.g.,

provide_context

) and have your external service/frontend call

perform_rpc

to push updates. ◦ Docs: https://docs.livekit.io/agents/logic/external-data/ • Text/data streams targeted at the agent (using a non-transcription topic) and have the agent listen for that topic. ◦ Docs: https://docs.livekit.io/agents/multimodality/text/#receiving-text-streams • Job metadata / room/participant attributes for initial or occasional context. ◦ Docs: https://docs.livekit.io/agents/server/job/#passing-data-to-a-job Which would you prefer? I can provide a concise agent-side code snippet for: • subscribing to raw audio frames and calling a noise-classifier, or • registering a byte-stream handler / RPC method and merging received context into the agent’s chat/session state. Pick one and I’ll send the snippet.