https://livekit.io logo
Join Slack
Powered by
Hey, everyone I want to know if it will be possib...
# server-api
h
Hey, everyone I want to know if it will be possible to stream the audio from Whatsapp call to a livekit room and back to whatsapp. Whatsapp’s calling api works with webrtc, they send us a 
sdp_offer
and we can create a sdp answer and send them a request to accept the call. I just wanted to know if it will be possible to create 
webrtc
connection that can accept the 
sdp_offer
from WhatsApp and stream audio back n forth b/w caller and agent (Livekit agent) I don’t have much idea about webrtc, so just asking if anyone else has done something like this previously. Thanks.
r
I see they have released their API. We will look into it.
h
thanks for the response @refined-appointment-81829 Let me know if you need any assistance from my side.
r
Well if you figure it out you can let me know. Otherwise we are looking into it.
h
yeah, trying out from my side, will let you know.
🙏 1
r
@loud-dawn-97520 FYI
h
hey guys so while trying to figure this out, I am a bit stuck. this is my main function that converts PyAV’s AudioFrame to rtc.AudioFrame
Copy code
async def stream(self) -> AsyncIterable[tuple[rtc.AudioFrame, float]]:
        if self._track is None:
            raise ValueError("track is not set")
        while True:
            av_frame: av.AudioFrame = await self._track.recv()
            frame = av_frame.to_ndarray()  # this produces interleaved channels audio so [l1, r1, l2, r2, ...]
            <http://logger.info|logger.info>("av frame data: channels: %s", av_frame.layout.nb_channels)
            <http://logger.info|logger.info>("frame: %s", frame)
            <http://logger.info|logger.info>("frame data: len(frame) %s, len(frame[0]): %s", len(frame), len(frame[0]))
            # scale frame values to int16 and do a type casting
            frame = (frame * self.SCALE_FACTOR).astype(np.int16)
            frame = frame.flatten()
            duration = len(frame) / self._media_info.audio_sample_rate  # 48kHz
            samples_per_channel = int(len(frame) / av_frame.layout.nb_channels)

            rtc_frame = rtc.AudioFrame(
                data=frame.tobytes(),
                sample_rate=self._media_info.audio_sample_rate,
                num_channels=av_frame.layout.nb_channels,
                samples_per_channel=samples_per_channel,
            )
            <http://logger.info|logger.info>("rtc_frame: %s", rtc_frame)
            <http://logger.info|logger.info>("sample rate: %s", rtc_frame.sample_rate)
            <http://logger.info|logger.info>("rtc frame channels: %s", rtc_frame.num_channels)
            <http://logger.info|logger.info>("rtc frame samples per channel: %s", rtc_frame.samples_per_channel)
            yield (
                rtc_frame,
                duration,
            )
this stream is then consumed by an audio source.
Copy code
async def _push_audio_frames(self, stream: AsyncIterable[tuple[rtc.AudioFrame, float]]):
        assert self.audio_source is not None

        async for frame, timestamp in stream:
            <http://logger.info|logger.info>("livekit capturng frame: %s", frame)
            await self.audio_source.capture_frame(frame)
            await asyncio.sleep(0)

    def start_streaming(self):
        asyncio.create_task(self._push_audio_frames(self.streamer.stream()))
None of this produces any error but agent also does not gets anything. this the last log that agent worker shows
Copy code
{
  "participant": "whatsapp_917889685075",
  "source": "SOURCE_MICROPHONE",
  "pid": 32073,
  "job_id": "AJ_LCoWm4qTGzEx"
}
I am though getting audio from whatsapp and am saving that in 
.wav
file as well.
my guess is that I am missing maybe resampling, trying it out. but I could be completely off the track here.
👍 1
l
@hundreds-dawn-90573 are you converting the audio frames to PCM?
h
I found out the issue actually. I am sending rtc.AudioFrames with 
stereo
(2 channels) but agent worker is receiving only 
mono
single channel.
@loud-dawn-97520 are you talking about converting the av audio_frame to pcm?
it is getting converted right when we do 
<http://av_frame.to|av_frame.to>_ndarray()
?
guys even after converting stereo to mono its not working. I am running monitoring in agent worker and I can see the rtc.AudioFrame’s being received. But Agent’s STT still doesn’t triggers and I get to no transcription. This is code on agent worker for monitoring.
Copy code
async def entrypoint(ctx: JobContext):
    logger.debug("Entrypint called with context: %s", ctx)

    # Add room event debugging
    @ctx.room.on("track_published")
    def on_track_published(publication, participant):
        <http://logger.info|logger.info>(f"🎵 Track published: {publication.name} by {participant.identity}")

    @ctx.room.on("track_subscribed")
    def on_track_subscribed(track, publication, participant):
        <http://logger.info|logger.info>(f"🎵 Track subscribed: {publication.name} by {participant.identity}, kind: {track.kind}")
        if track.kind == rtc.TrackKind.KIND_AUDIO:
            # Add audio track inspection
            audio_track = track
            <http://logger.info|logger.info>("Audio track details:")
            <http://logger.info|logger.info>(f"  - Track SID: {audio_track.sid}")
            <http://logger.info|logger.info>(f"  - Source: {publication.source}")

            # Check if the track is actually receiving data
            asyncio.create_task(monitor_audio_track(audio_track))
     ...


async def monitor_audio_track(audio_track):
    """Monitor if audio track is receiving data"""
    frame_count = 0
    try:
        async for frame in rtc.AudioStream(audio_track):
            frame_count += 1
            if frame_count % 50 == 0:  # Log every 50 frames
                <http://logger.info|logger.info>(
                    f"Received audio frame {frame_count}: {len(frame.frame.data)} bytes, sample_rate: {frame.frame.sample_rate}"
                )
                <http://logger.info|logger.info>(
                    f"Frame format - Sample rate: {frame.frame.sample_rate}, Channels: {frame.frame.num_channels}, Samples per channel: {frame.frame.samples_per_channel}"
                )
                audio_data = np.frombuffer(frame.frame.data.tobytes(), dtype=np.int16)
                <http://logger.info|logger.info>(f"Audio data shape: {len(audio_data)} samples, first 10 values: {audio_data[:10]}")
    except Exception as e:
        logger.error(f"Error monitoring audio track: {e}")
AND this is how I am doing conversions.
Copy code
async def stream(self) -> AsyncIterable[tuple[rtc.AudioFrame, float]]:
        if self._track is None:
            raise ValueError("track is not set")
        while True:
            av_frame: av.AudioFrame = await self._track.recv()

            # <http://logger.info|logger.info>("av frame data: channels: %s", av_frame.layout.nb_channels)
            # <http://logger.info|logger.info>("frame: %s", frame)
            # <http://logger.info|logger.info>("frame data: len(frame) %s, len(frame[0]): %s", len(frame), len(frame[0]))
            # scale frame values to int16 and do a type casting
            resampled_frames = self._resampler.resample(av_frame)
            for mono_frame in resampled_frames:
                self._webrtc_frame_count += 1
                if self._webrtc_frame_count % 50 == 0:
                    <http://logger.info|logger.info>(
                        f"[WEBRTC->BRIDGE] Frame {self._webrtc_frame_count}: {len(mono_frame.to_ndarray().flatten())}, stereo: {len(av_frame.to_ndarray().flatten())}, samples, sample_rate: {mono_frame.sample_rate}, channels: {mono_frame.layout.nb_channels}"
                    )
                    <http://logger.info|logger.info>("mono-av frame data: %s", mono_frame.to_ndarray().flatten()[:10])
                mono_frame = mono_frame.to_ndarray()
                mono_frame = (mono_frame * self.SCALE_FACTOR).astype(np.int16)
                mono_frame = mono_frame.flatten()
                duration = len(mono_frame) / self._media_info.audio_sample_rate  # 48kHz
                samples_per_channel = int(len(mono_frame))

                rtc_frame = rtc.AudioFrame(
                    data=mono_frame.tobytes(),
                    sample_rate=self._media_info.audio_sample_rate,
                    num_channels=1,
                    samples_per_channel=samples_per_channel,
                )
                # <http://logger.info|logger.info>("rtc_frame: %s", rtc_frame)
                # <http://logger.info|logger.info>("sample rate: %s", rtc_frame.sample_rate)
                # <http://logger.info|logger.info>("rtc frame channels: %s", rtc_frame.num_channels)
                # <http://logger.info|logger.info>("rtc frame samples per channel: %s", rtc_frame.samples_per_channel)
                yield (
                    rtc_frame,
                    duration,
                )
kinda stuck on this one.
m
@hundreds-dawn-90573 were you able to solve this?
2 Views
Open in Slack
PreviousNext
Powered by