To diagnose agent speech with gaps similar to packet loss in LiveKit, follow these steps: • Monitor Packet Loss Metrics: Use metrics like

packets_lost

,

packet_loss_rate

,

packet_loss_percentage

, and

gap_histogram

. The

gap_histogram

is particularly useful for identifying sequence number gap patterns. • Analyze Sequence Number Gaps: Check the ingress system's gap detection logic to determine if gaps are due to actual packet loss or sequence number discontinuities. • Connection Quality Scoring: Use LiveKit's connection quality system to differentiate between actual packet loss and sequence number gaps that might appear as false positives. • Agent-Specific Audio Processing: Investigate Voice Activity Detection (VAD) and turn detection systems. Ensure that VAD sensitivity and turn detection modes (e.g., VAD, STT signals, realtime LLM detection) are configured correctly. • RTCP Feedback: Utilize WebRTC's Receiver Reports and Transport-Wide Congestion Control (TWCC) to diagnose audio quality issues. • Retransmission Patterns: Monitor metrics like

nacks

,

nack_acks

,

nack_misses

, and

nack_repeated

to understand retransmission behavior. • Jitter and Timing: Examine

jitter_current

,

jitter_max

,

rtt_current

, and

rtt_max

to identify timing-related issues. • Audio Pipeline Health: Check components like

rtc.AudioSource

(e.g.,

queue_size_ms

parameter),

AudioRecognition

, and

AgentActivity

for processing delays or buffering issues. • Simulcast Layer Switching: Be aware that switching between simulcast layers can create sequence number gaps, which might be misinterpreted as packet loss. This comprehensive approach combines network-level diagnostics with agent-specific monitoring to identify and resolve speech quality issues effectively. Sources: livekit-protocol/livekit/protocol/models.py | pkg/whip/sdk_whip_track_handler.go | pkg/sfu/connectionquality/scorer.go | Speech & Audio Processing | livekit/livekit_composite | DeepWiki