numerous-morning-99696
04/16/2025, 6:10 PMaudio_buffer = b""
async for audio_chunk in synthesized_audio_stream:
audio_buffer += audio_chunk
while len(audio_buffer) >= self.BYTES_PER_FRAME:
frame_data = audio_buffer[: self.BYTES_PER_FRAME] # Extract full frame
audio_buffer = audio_buffer[
self.BYTES_PER_FRAME :
] # Keep leftover data
audio_frame = rtc.AudioFrame.create(
self.SAMPLING_RATE_DOWNSTREAM, self.CHANNELS, self.SAMPLES_PER_FRAME
)
# Convert frame_data from bytes to a NumPy array of int16 PCM samples
frame_samples = np.frombuffer(frame_data, dtype=np.int16)
# Convert audio_frame's data buffer to a NumPy array of int16 PCM samples
audio_frame_samples = np.frombuffer(audio_frame.data, dtype=np.int16)
# Copy the extracted samples into the audio frame
np.copyto(audio_frame_samples, frame_samples)
await self._capture_frame(audio_frame, user_id)
# This is just here for context - but it is use specific to the user
backend_audio_source = rtc.AudioSource(
sample_rate=self.global_audio_config.SAMPLING_RATE_DOWNSTREAM,
num_channels=self.global_audio_config.CHANNELS,
)
backend_track = rtc.LocalAudioTrack.create_audio_track(
name=self.livekit_config.backend_track_name,
source=backend_audio_source,
)
# Publish the new track
local_track_publication = await local_participant.publish_track(
backend_track, audio_publish_options
)
# The client subscribes to the above track
async def _capture_frame(self, audio_frame: rtc.AudioFrame, user_id: str) -> None:
# redact
try:
await backend_audio_source.capture_frame(audio_frame)
except Exception as e:
raise e
Everything works fine and dandy, but it used to be once in a blue moon, but it seems once a day now, we'd get:
await backend_audio_source.capture_frame(audio_frame) would throw:
"/app/our_app/services/audio_to_client/send_audio_to_client_service_livekit.py", line 170, in _capture_frame
await backend_audio_source.capture_frame(audio_frame)
File "/usr/local/lib/python3.12/site-packages/livekit/rtc/audio_source.py", line 152, in capture_frame
raise Exception(cb.capture_audio_frame.error)
Exception: an RtcError occured: InvalidState - failed to capture frame.
I'm also looking at the LiveKit's code that throws this error - audio_source.py:
queue = FfiClient.instance.queue.subscribe(loop=self._loop)
try:
resp = FfiClient.instance.request(req)
cb: proto_ffi.FfiEvent = await queue.wait_for(
lambda e: e.capture_audio_frame.async_id
== resp.capture_audio_frame.async_id
)
finally:
FfiClient.instance.queue.unsubscribe(queue)
if cb.capture_audio_frame.error:
raise Exception(cb.capture_audio_frame.error)
raise Exception(cb.capture_audio_frame.error) is the error from the Stacktrace and it is just,
"Exception: an RtcError occured: InvalidState - failed to capture frame.".
Is there a way to find out/log more details as to what exactly did made LiveKit internals think
that "state" was "Invalid"?
Our users' experience is that they get dropped mid conversation.
We added logic where we tolerate 5 such failures in a row for specific users AudioSource(in our code:
backend_audio_source = rtc.AudioSource
)
We are thinking of logging more stats about the frames when the error occurs.
Also adding possibly delay after a failure, Before we try the next frame?
We are on a 2-months old version of the SDK, so We'd be upgrading that?
Searching for the error here, yielded things such as "put more compute" or "It's an ElevenLabs issue".
Neither apply for our set-up - this is just a single user on an 8 Core 32 GB RAM AKS VM.
The LiveKit github issues also have questions about this issue, but not much in terms of concrete resolution.
Any helpful suggestions? Even better, sudden solution to what seems to an issue experienced by multiple LiveKit users?numerous-morning-99696
04/16/2025, 10:47 PMfor chunk in frame.data.chunks(self.queue_size_samples as usize) {
let nb_frames = chunk.len() / self.num_channels as usize;
let (tx, rx) = oneshot::channel::<()>();
let ctx = Box::new(tx);
let ctx_ptr = Box::into_raw(ctx) as *const sys_at::SourceContext;
unsafe {
if !self.sys_handle.capture_frame(
chunk,
self.sample_rate,
self.num_channels,
nb_frames,
ctx_ptr,
sys_at::CompleteCallback(lk_audio_source_complete),
) {
return Err(RtcError {
error_type: RtcErrorType::InvalidState,
message: "failed to capture frame".to_owned(),
});
}
}
let _ = rx.await;
}numerous-morning-99696
04/17/2025, 12:59 PMaudio_source = rtc.AudioSource(
sample_rate=<our_sample_rate>,
num_channels=<our_num_channels>,
queue_size_ms=2000
I was able to reproduce the error fairly consistently by setting the queue_size_ms to the min allowed value of 10. Hope this helps to anyone that might be searching.