🚢

livekit-agents

0.10.2 release 🚢 Heads up, there's a new version of

livekit-agents

available! v0.10.2 includes the following updates: • Added

VAD.update_options

to the Silero plugin, which you can use to update the VAD options after the VAD object has been created. • Included various fixes related to address cases where

VoicePipelineAgent

was getting stuck. • Included some additional VAD-related fixes. Please reach out if you run into any issues or have any other feedback!

Hy everyone! I'm running a free workshop on Friday Nov 1st (this Friday) showing how to build a fullstack Advanced Voice web app with the OpenAI Realtime API. It's an hour long, will be interactive (as time permits) and we'll start from the basics and build all the way from 0 to a finished app that talks with users using Advanced Voice. Backend will be Python, frontend will be in React — but once you're finished it's very easy to switch either piece out for a different technology. There's only a few spots left, but I figured I'd post here in case anyone here who hasn't started building yet wanted a live crash course. Here's the registration link: https://forms.gle/SAQjdnQXFLoEsLXt7 Hope to see some of you on Friday!

🚢

livekit-agents

0.11.0 release 🚢 We've just released

livekit-agents

0.11.0 which is full of treats! Inside you’ll find the much-requested addition of more detailed timing metrics as each stage of the

VoicePipelineAgent

and much more. Highlights: • Introduced detailed time metrics (time to first byte/token of STT/LLM/TTS, duration) and usage metrics, including an example for calculating the cost of a conversation. • Added

TTS.update_options

- useful for making runtime changes, such as changing voices. • Added the ability to get the human and agent transcript for the

MultimodalAgent

class. • Fixed CPU load calculation using the system usage instead of the cgroup usage, addressing autoscaling issues for some developers. Check out the full changelog for livekit-agents and each plugin for a detailed rundown of all updates. A huge thanks to @martin-purplefish, @Tanesan, @longcw, @jayeshp19, @hari-truviz, and @samirsalman for your excellent contributions!

🚢 🚢 🚢 We’re excited to introduce a new data channel feature called RPC! You can now register a custom method on one participant then invoke it remotely from another participant in the room and await a response. This is perfect for forwarding LLM function calls from an agent, for instance to query data like geolocation that’s available only in the browser. You can also use it to easily give your agent the ability to directly manipulate the user’s UI and display notifications, visual effects, or request specific user input. The possibilities are endless, and we’re excited to see what you all build with it! RPC is available now for JS (

livekit-client@2.6.0

), Python (

livekit==0.18.0

), Node (

@livekit/rtc-node@0.11.0

), and Rust (

livekit = "0.7.0"

). Updates for the mobile SDKs are coming over the next few weeks. Check out the full docs for a complete overview and examples.

🚢 agents-js 0.4.3 release 🚢 Check out the latest version of LiveKit Agents for Node.js (0.4.3) for a drop of new features: • Introducing support for

VoicePipelineAgent

🚀 • Silero VAD, ElevenLabs TTS, Deepgram STT, OpenAI LLM 🪄 • Various fixes 🛠️ Note: We are aware of a bug concerning users of tsx under certain conditions. See this issue for more details and a fix.

We have a new channel for those of you that are working with OpenAI's Realtime API: #C0816MD6LCR

🚢

livekit-agents@0.11.2

release 🚢 Hello LiveKit community! Go and grab

livekit-agents@0.11.2

for some new additions to the OpenAI plugin along a handful of bug fixes: ◦ New features in

livekit-plugins-openai@0.10.16

▪︎ Added usage metrics for the OpenAI Realtime API, which provides more visibility into the input/output token usage when using

MultimodalAgent

with this API. ▪︎ The

chat_ctx

is now synchronized with the OpenAI Realtime API - transcripts from the model are written back to the

chat_ctx

while the realtime session is updated based on input from

chat_ctx

. • Fix to correctly follow the

min_endpointing_delay

configuration • Fix for

agent_speech_committed

not being called in certain scenarios Please refer to the changelog for a full rundown of all updates - and thanks to @jayeshp19, @martin-purplefish, and @longcw for your contributions!

🚢 Node.js

@livekit/agents-plugin-openai

0.5.0 release 🚢 Version (0.5.0) of the OpenAI plugin for Node.js Agents is now available and includes the following updates: • Support for OpenAI TTS and STT • Full compatibility with

VoicePipelineAgent

as of

agents-js

Version 0.4.4

🚢 Agents for Node.js Release 0.4.5 🚢 We’ve just shipped a bugfix release for LiveKit Agents for Node.js (0.4.5) which you should grab at your convenience. This release also includes a new version of the Node.js OpenAI plugin, which has now bumped to Version 0.6.0. Highlights: • Ensures all plugins are using the same version of Agents • OpenAI function calling now allows for raw JSON arguments • Various code quality improvements (see changelog) - a huge thanks to @astonishing-boots-14432 for your excellent contributions!

🚢 Agents for Node.js Release 0.4.6 🚢 LiveKit Agents for Node.js 0.4.6 is now available, which includes: • Agent/user speech committed events • New OpenAI Realtime voices • Fixed an issue where agent state was not updating correctly • Updated version of Node.js OpenAI plugin (0.6.1) Thanks to @john-royal for your contributions!

🚢 Agents for Node.js Release 0.5.0 🚢 Agents for Node.js 0.5.0 is now available, which includes: • Native support for CommonJS, which makes it easy to use Agents in frameworks like NestJS • Silero VAD is now compatible with Windows • Fixes to interruption handling in

VoicePipelineAgent

Weekly Roundup - Dec 14 2024 Here’s a recap of a few new things that have landed this week. Check them out and let us know what you think! == Outbound calling guide for agents == We’ve published a new guide on how to build an AI voice agent that can make outgoing calls to phone numbers using LiveKit. You’ll find it helpful for a variety of use cases such as returning customer calls, following up on purchased items, confirming appointments and more. The new guide walks through several new features that can be used in these scenarios, including explicit dispatch, job dispatch metadata, voicemail detection, end call intent handling, and appointment scheduling. In order to make it easy to get started building an agent with outbound calling capabilities, we’ve also created a new bootstrap template which you can use to get started. == Voice agent example using Gemini 2.0 Flash == We’ve added a new Python example that demonstrates how to create a voice agent using the Gemini 2.0 Flash model along with Google STT and TTS. You can use this as a starting point for building your Gemini-powered agents. == Agents for Node.js Version 0.5.1 == Update to the latest version for the following updates: • Fixed LLM tool calling not executing multiple times when a question has more than one tool call • Fixed sentence tokenizer • Allow attributes to be set on

accept()

== Agents for Python Version 0.12.2 == Update to the latest version for the following updates: • Improvements to endpointing latency • Improvements to end of turn plugin • Fixing duplicate messages committed with function calls • Fixes handling of optional func args in tool calls

Hey <!everyone>! We’ve just released a new feature for Agents: a language-aware turn detection model. This adds a semantic layer on top of the usual voice activity detection (VAD). Instead of just listening for silence to figure out when a user’s turn ends, it analyzes the text to decide if the user has completed their thought 🤯 This makes conversations feel a lot smoother and more natural. For example, if someone says “I’d like to order a…” and pauses, the model knows they’re not done yet and keeps waiting. But if the sentence is complete, it triggers the response right away. It will make your agents sound a lot more natural in conversations with pauses. Performance-wise, it’s lightweight and built to work seamlessly with the existing Agent pipeline. Running on a 4-core instance takes ~50ms of inference and about 1.5GB of RAM. You can find the plugin and setup instructions here. It’s easy to add to your current VoicePipelineAgent, it’s a single param when you’re setting up the config for the agent. We’re still refining it and would love your feedback. Let us know how it handles your use cases, what’s working well, and where it struggles. Your input is invaluable as we iterate on this!

🚢 Agents for Node.js Release 0.6.0 🚢 Agents for Node.js 0.6.0 is now available, which includes: • Support for performance and usage metrics logging • Ability to use

say()

inside of a function call • Fix multiple function calls not firing multiple times • Parity with Python SDK in

MultimodalAgent

events • Additional small fixes

🚢 Agents for Python 0.12.6 Release 🚢 Agents for Python 0.12.6 is now available, which includes: • Improved interruption handling: avoids getting stuck on false positive interruptions • Added manual interrupt method for VoicePipelineAgent • PlayHT/PlayAI TTS plugin • MultimodalAgent with Gemini Live API • Bug fixes for STT plugins to address incorrect handling of retries

Moving RunLLM bot to dedicated channel: #C088ZNU7QQ5 Lately, we've been testing a bot from RunLLM across our product channels in order to help answer questions related to LiveKit products. It’s been trained on our documentation and code base and can answer a range of common questions that developers have when building with LiveKit. At the same time, we’ve also found (and heard!) that it can be a bit noisy to have these bot interactions directly in conversations within our product channels. In particular, we’ve seen that it's been distracting for other developers that are engaging on these conversation threads as well. To strike a better balance here, we've created a new, dedicated channel (#C088ZNU7QQ5) for developers to engage the RunLLM bot for questions and guidance related to LiveKit products. With this channel, you no longer even need to mention RunLLM - simply create a new post in the channel, and RunLLM will respond in a thread on your post. With the new channel in place, we’ll be removing the RunLLM bot from our product channels. If you want to use the RunLLM bot, you’ll need to post in the #C088ZNU7QQ5 channel moving forward. Please reach out if you have any questions or additional feedback!

New LiveKit Cloud dashboards in alpha We’re excited to introduce early access to a new set of views for the LiveKit Cloud dashboard! We’ve overhauled several pages with improved usability and performance enhancements to handle larger loads and faster load times. Everyone can opt-in to this new experience to give it a try. When you visit your LiveKit Cloud dashboard, you’ll see a banner at the top of the page through which you can opt-in to enable the new views for your project. We consider this update to be a fairly stable alpha release and there are still some things under development, so you can switch back to the legacy views if needed at any point. We wanted to share these updates early so that your feedback can directly influence the shape of things to come - please use the feedback form at the top of the dashboard to let us know what you think!

🚢 Agents for Python 0.12.11 + Turn Detector 0.4.0 Releases 🚢 Agents for Python 0.12.11 is now available, which includes: • Fixed LLM

FallbackAdapter

to handle function calls correctly • Fixed handling of

before_llm_cb

behavior for skipped inferences • Fixed false-positive interruption handling with Anthropic and Google LLMs • Added

generate_reply

API for

MultimodalAgent

• Improved TTFB metrics for streaming TTS We’ve also released a major version update to our Turn Detector (0.4.0) model: • 3x smaller • 5x faster inference • 98% true positive rate (+1%) • 97% true negative rate (+11%)

📆 Realtime AI Meetup | Feb 27 in SF 📆 Join us at Founders, Inc in SF on February 27 for the Realtime AI Meetup! Hear from experts across the AI stack where they’ll share insights into building realtime AI agents. Discussions will cover voice agent pipelines, edge vs. cloud inference, multimodal models, agentic workflows, reasoning models, and more. The event will feature speakers from Cartesia, Cerebras, and Google DeepMind - and moderated by our co-founder & CEO, Russ d'Sa. If you’re local to the Bay Area, we’d love to see you there! RSVP and more details here: https://lu.ma/a9xvupy4

🚀 New feature: Text & Byte Streams We've got a pair of new features for you that make data channels easier to use: text streams and byte streams. • First, two simple new methods

sendText

and

sendFile

, which let you send anything of any size and we'll handle encoding, chunking, buffering, and reconstruction for you. • Additionally we have added

streamText

and

streamBytes

, which let you send anything incrementally (e.g. for streaming a long LLM response to your frontend) • Either way, receive the data in a stream with

registerTextStreamHandler

and

registerByteStreamHandler

. You can read chunks as they come in or just

readAll

if you don't need progressive rendering. • Both features rely on

topic

so you can organize them within your application for different uses. And both support sending to any or all participants in the room simultaneously. These features are currently available in a preview state in the latest release of the JS (2.9.0), Python (0.20.0), and Node.js (0.13.4) SDKs. Others coming soon. We've also published new documentation to cover all the realtime text & data features. Let us know what you think, and we'd love to see what you build with it!

🚢 Agents for Python 0.12.15 + Agents for Node.js 0.7.0 Releases 🚢 We’ve recently dropped some new updates to both the Python and Node.js distributions of the Agents framework, so be sure to grab the latest for your platform of choice, including: Agents for Python 0.12.15 • Improved performance of OpenAI TTS by switching to use Opus encoding (#1494) • Improved exception logging (#1490) • Fix interrupting nested speech from

before_llm_cb

(#1504) • Add cache tokens in

CompletionUsage

dataclass (#1478) Agents for Node.js 0.7.0 • Introduces support for Turn Detector model (#225) • Replace transcription forwarder with synchronizer (#301) • Gracefully fail on StreamAdapter errors (#299) • Skip TTS on empty LLM output (#293) • Clearer timeout handling for drain (#277) • Fix feeding null LLM input (#296)

🚨 Action Required 🚨 Managing package requirements prior to Agents for Python 1.0 release We’re nearing the release of Agents for Python v1.0 which will contain some breaking changes for agents that have been built with versions prior to v1.0. We’ll be publishing a detailed migration guide soon in order to help you navigate these changes as you upgrade your agents to use the new v1.0 package. Since v1.0 contains changes that will need to be made to your agents, we strongly recommend that you proactively manage your upgrades to the Agents for Python packages (

livekit-agents

and all associated plugins) to ensure that you don’t upgrade until you’re ready. For instance, if you’re using a pip requirements file, you can add a version constraint to each package to restrict to only versions earlier than v1.0:

livekit-agents>=0.12.6,<1.0.0

We’ll share some more details on v1.0 soon - if you have any questions in the meantime, please reach out as we’re here to help!

🚀 Analytics v2 for LiveKit Cloud is here!!! 🚀 We rebuilt our analytics from the ground up to support scale at any size for any use case. Agents, streaming, and now telephony metrics! Some other things to 👀 when you're checking the new dashboard: • Faster load times: Background data processing and streaming updates makes everything load a lot faster. • Streamlined navigation: Breadcrumb system, redesigned layout, and improved search for quick session and participant insights. • New insights: Connection minutes, quality metrics, latency stats, and 🎉 telephony call logs 🎉 . • Enhanced controls: Filter, slice, and share dashboard states easily with URL params and live updates. We built these updates directly from your feedback. Check out the announcement and let us know what you think. More feedback = better features 😄 https://x.com/livekit/status/1902010421264937246

🚀 Krisp support in Python! You can now apply enhanced noise cancellation directly inside your agent or other Python app. This is available to all LiveKit Cloud projects at no extra cost. This has a few distinct benefits over client-side integrations: • Support for Krisp's Background Voice Cancellation (BVC) model - filter not just noise but also background speakers that would otherwise confuse turn detection and transcription processes in voice agents. • Universal client support - This filter works on audio coming from anywhere, including native apps, web, telephony, and even our Unity SDK • Simpler integration for multi-platform apps - integrate the filter in just one spot instead of adding the package to Swift, Android, Flutter, and JS apps Integration couldn't be simpler:

agent = VoicePipelineAgent(
    # .. STT/TTS/LLM/etc ..,
    noise_cancellation=noise_cancellation.BVC()
)

The package and documentation are available on PyPI. Node.js support is in the works and will be ready soon. Give it a try and let us know what you think! https://pypi.org/project/livekit-plugins-noise-cancellation/

A few folks in the community have suggested the idea of open office hours and I think it sounds like a good idea. Next Thursday at 9am US Pacific/12pm US Eastern I will host an open office hour livestream where folks can raise their hand and ask questions and get answers. I will share a link here for the call 10mins before the call.

🤖 Python Agents 1.0 RC Hey everyone! We've just rolled out a big update for Python Agents! Thanks to all your feedback over the past year, we've made the framework cleaner and more flexible. Here's what's new: ✨ Multi-Agent Support ◦ Break down complex workflows into simpler tasks. 🔄 Unified AgentSession Class ◦ We've merged VoicePipelineAgent and MultimodalAgent into a single AgentSession. ◦ Handles both pipelined models and the Realtime API. ⚙️ Flexible IO and overides ◦ Flexible IO system to take input/direct output as you wish ◦ pre/post-process each part of the pipeline 📡 Improved Text Stream Handling ◦ Text input/output are now fully integrated as first-class features. 🎛️ API Updates ◦ `generate_reply`: Generate dynamic responses from LLMs. ◦ `say`: Respond with static text or pre-synthesized audio. ◦ `interrupt`: Interrupt your agent whenever needed. 🐞 Lots of Bug Fixes Check out the examples in the repo and the new docs site! To try it out:

pip install "livekit-agents[openai,silero,deepgram,cartesia]~=1.0rc"

Let us know what you think!

Reminder that in one hour we will have an open office hour (LiveStream). You will be able to raise your hand and ask questions. We will have several folks form the LiveKit staff on hand to answer the questions. (I will add additional details in the thread)

🚀 LiveKit Agents 1.0 is here! The wait is over — we’ve shipped a major upgrade to our open-source framework for building voice AI agents. Agents 1.0 introduces support for structured workflows, making it dramatically easier to build closed-loop agents for scheduling, support, and automation use cases. Key updates include: • Unified AgentSession interface for all agent types, real-time or pipeline • Modular pipeline nodes for easier customization • Synchronized captioning • Multilingual semantic turn detection • Streamlined function tool definitions (function_tool) • Fine-grained control over STT → LLM → TTS flows We’re also kicking off a closed beta for Cloud Agents, our fully-managed hosting platform for deploying agents at scale. 📚 Explore what’s new → https://docs.livekit.io/agents 🔄 Migration guide → https://docs.livekit.io/agents/start/v0-migration/ ⚡ Examples → https://github.com/livekit/agents/tree/main/examples/voice_agents 📝 Blog post → https://blog.livekit.io/livekits-series-b/ 🤖 Cloud agents beta signup → here 💬 Let us know what you’re building — and if you’re excited, please reshare the blog post or give it a boost on social! Every bit helps 🙌

Come Joins Us. Devs building in voice and video AI, we’ve got something special cooked up for next week — on 4/30, we’re doing a fireside chat with @juberti in SF. Justin is a legend. He created the WebRTC protocol, led dev for Google Meet and Stadia, started @FixieAI, and is now the Head of... https://x.com/livekit/status/1914335043549368761

Hi everyone! We just created a brand new #C08QMJD9J9G channel for everyone that's using LiveKit for robotics, machines, and other embedded platforms! Please drop in and say hi. We'd love to hear what you're working and how we can make LiveKit even better for your use case! 🦾🤖🚗