We are using the python sdk on our FastAPI server....
# server-api
n
We are using the python sdk on our FastAPI server. On the client side we have a Swift iOS app(using the livekit swift skd) and a React Native app(using the react native sdk). We don't use Agents, but rather have our own Agentic code, which uses Deepgram as a Text-To-Speech service. We get an audio stream from Deepgram TTS and build rtc.AudioFrames off it:
Copy code
audio_buffer = b""
async for audio_chunk in synthesized_audio_stream:


    audio_buffer += audio_chunk

    while len(audio_buffer) >= self.BYTES_PER_FRAME:
        frame_data = audio_buffer[: self.BYTES_PER_FRAME]  # Extract full frame
        audio_buffer = audio_buffer[
            self.BYTES_PER_FRAME :
        ]  # Keep leftover data

        audio_frame = rtc.AudioFrame.create(
            self.SAMPLING_RATE_DOWNSTREAM, self.CHANNELS, self.SAMPLES_PER_FRAME
        )

        # Convert frame_data from bytes to a NumPy array of int16 PCM samples
        frame_samples = np.frombuffer(frame_data, dtype=np.int16)

        # Convert audio_frame's data buffer to a NumPy array of int16 PCM samples
        audio_frame_samples = np.frombuffer(audio_frame.data, dtype=np.int16)

        # Copy the extracted samples into the audio frame
        np.copyto(audio_frame_samples, frame_samples)

        await self._capture_frame(audio_frame, user_id)

# This is just here for context - but it is use specific to the user
backend_audio_source = rtc.AudioSource(
    sample_rate=self.global_audio_config.SAMPLING_RATE_DOWNSTREAM,
    num_channels=self.global_audio_config.CHANNELS,
)
backend_track = rtc.LocalAudioTrack.create_audio_track(
    name=self.livekit_config.backend_track_name,
    source=backend_audio_source,
)

# Publish the new track
local_track_publication = await local_participant.publish_track(
    backend_track, audio_publish_options
)

# The client subscribes to the above track

async def _capture_frame(self, audio_frame: rtc.AudioFrame, user_id: str) -> None:
    # redact
    try:
        await backend_audio_source.capture_frame(audio_frame)
    except Exception as e:
        raise e
Everything works fine and dandy, but it used to be once in a blue moon, but it seems once a day now, we'd get:
Copy code
await backend_audio_source.capture_frame(audio_frame) would throw:
"/app/our_app/services/audio_to_client/send_audio_to_client_service_livekit.py", line 170, in _capture_frame
    await backend_audio_source.capture_frame(audio_frame)
  File "/usr/local/lib/python3.12/site-packages/livekit/rtc/audio_source.py", line 152, in capture_frame
    raise Exception(cb.capture_audio_frame.error)
Exception: an RtcError occured: InvalidState - failed to capture frame.
I'm also looking at the LiveKit's code that throws this error - audio_source.py:
Copy code
queue = FfiClient.instance.queue.subscribe(loop=self._loop)
try:
    resp = FfiClient.instance.request(req)
    cb: proto_ffi.FfiEvent = await queue.wait_for(
        lambda e: e.capture_audio_frame.async_id
        == resp.capture_audio_frame.async_id
    )
finally:
    FfiClient.instance.queue.unsubscribe(queue)

if cb.capture_audio_frame.error:
    raise Exception(cb.capture_audio_frame.error)
raise Exception(cb.capture_audio_frame.error) is the error from the Stacktrace and it is just, "Exception: an RtcError occured: InvalidState - failed to capture frame.". Is there a way to find out/log more details as to what exactly did made LiveKit internals think that "state" was "Invalid"? Our users' experience is that they get dropped mid conversation. We added logic where we tolerate 5 such failures in a row for specific users AudioSource(in our code: backend_audio_source = rtc.AudioSource ) We are thinking of logging more stats about the frames when the error occurs. Also adding possibly delay after a failure, Before we try the next frame? We are on a 2-months old version of the SDK, so We'd be upgrading that? Searching for the error here, yielded things such as "put more compute" or "It's an ElevenLabs issue". Neither apply for our set-up - this is just a single user on an 8 Core 32 GB RAM AKS VM. The LiveKit github issues also have questions about this issue, but not much in terms of concrete resolution. Any helpful suggestions? Even better, sudden solution to what seems to an issue experienced by multiple LiveKit users?
maybe this provides more context: I don't know if this helps but I tracked the rust code that potentially triggers this error? https://github.com/livekit/rust-sdks/blob/0773bcec4e24812b382e71166b2ab359bf1a9384/libwebrtc/src/native/audio_source.rs#L87 it seems this portion is what I throws the error I see in my python logs:
Copy code
for chunk in frame.data.chunks(self.queue_size_samples as usize) {
            let nb_frames = chunk.len() / self.num_channels as usize;
            let (tx, rx) = oneshot::channel::<()>();
            let ctx = Box::new(tx);
            let ctx_ptr = Box::into_raw(ctx) as *const sys_at::SourceContext;

            unsafe {
                if !self.sys_handle.capture_frame(
                    chunk,
                    self.sample_rate,
                    self.num_channels,
                    nb_frames,
                    ctx_ptr,
                    sys_at::CompleteCallback(lk_audio_source_complete),
                ) {
                    return Err(RtcError {
                        error_type: RtcErrorType::InvalidState,
                        message: "failed to capture frame".to_owned(),
                    });
                }
            }

            let _ = rx.await;
        }
I dug down to this Rust code: https://github.com/livekit/rust-sdks/blob/0773bcec4e24812b382e71166b2ab359bf1a9384/libwebrtc/src/native/audio_source.rs#L102 and with some help from a friendly LLM, I was able to alleviate the issue in our python code by doubling the size of the queue_size_ms:
Copy code
audio_source = rtc.AudioSource(
            sample_rate=<our_sample_rate>,
            num_channels=<our_num_channels>,
            queue_size_ms=2000
I was able to reproduce the error fairly consistently by setting the queue_size_ms to the min allowed value of 10. Hope this helps to anyone that might be searching.
🙌 1
🙏 1