I want to monitor psrpc latency, please advice any exist metrics or middleware ?

Thanks for reaching out, Thai. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

What i mean is pubsub rpc based on redis, I got the issue high latency of livekit api with high ccu traffic in my self host lk cluster

Thanks — I’m responding to your follow-up about monitoring the Redis-based pubsub RPC used by LiveKit (psrpc) in your self-hosted cluster, and about high latency under high CCU. Quick questions so I give targeted steps: • Are you using Prometheus/Grafana, Datadog, or another monitoring stack? • Is Redis standalone, Redis Cluster, or managed (e.g., AWS ElastiCache)? Are LiveKit services co-located with Redis? • Do you want per-RPC latency (method-level), overall Redis latency, or both? What you can do now (options you can pick from): • Monitor Redis directly (essential) ◦ Run

redis-cli --latency

and

redis-cli info stats|memory|clients

for quick checks. ◦ Deploy the Redis Exporter and scrape it with Prometheus to get: command rates, connected clients, memory, pub/sub metrics, and command duration distributions. ◦ Dashboard/alerts to watch:

instantaneous_ops_per_sec

,

used_memory

,

connected_clients

,

pubsub_channels

, and command latency/slowlog. • Infer psrpc latency from LiveKit metrics ◦ LiveKit exports Prometheus metrics for service-level health. Use those alongside Redis metrics to correlate spikes in API latency with Redis load. ◦ Correlate LiveKit request rates, error rates, and service CPU/memory with Redis metrics. • Add instrumentation for psrpc (recommended for per-RPC visibility) ◦ Add a small interceptor/middleware around psrpc handlers to record request start/end times and export histograms/counters (Prometheus or OpenTelemetry). ◦ Capture labels: RPC method name, success/failure, room or node id (avoid high cardinality). ◦ Record histogram buckets for latency and counters for inflight requests to spot backpressure. • Short-term tracing / probing ◦ Use request-level tracing (OpenTelemetry) or logs that annotate timestamps at RPC send/receive to measure round-trip times. ◦ If running in k8s, sidecar/tracing (Istio/OTel) or eBPF tools can observe Redis client calls without code changes. • Scale / tune Redis (common fix for high CCU) ◦ Ensure Redis has sufficient CPU/RAM; consider Redis Cluster/sharding when command throughput is high. ◦ Use dedicated instances and fast networking; colocate LiveKit and Redis or use Unix sockets when possible to reduce latency. ◦ Review slowlog and consider increasing maxclients/adjusting persistence settings that block I/O. If you want, I can: • Provide a Prometheus + Grafana dashboard suggestion and concrete queries to track Redis + LiveKit correlation. • Draft a small Prometheus/OTel middleware snippet you can add to psrpc handlers to capture per-method latency. Which monitoring stack do you use and which of the follow-up items would you like me to produce first? Sources: Universal Egress | Universal Egress | Universal Egress | Server-side | livekit/livekit_composite | DeepWiki

I am using redis standalone, what metrics i could have a look quickly to mearsure the performance

Thanks — responding to your question about which Redis metrics to check quickly for a standalone instance to diagnose psrpc (pubsub RPC) latency. Check these immediately (commands and what to look for): •

used_memory

/

used_memory_rss

(run

redis-cli info memory

) ◦ High or growing memory and RSS indicates memory pressure or swapping; swapping causes latency spikes. •

instantaneous_ops_per_sec

and

total_commands_processed

(run

redis-cli info stats

) ◦ Watch spikes in ops/sec and sudden drops — high sustained ops/sec can saturate Redis CPU. •

connected_clients

and

blocked_clients

(run

redis-cli info clients

) ◦ Very high client count or non-zero

blocked_clients

means connection pressure or blocking operations affecting pub/sub. •

latency

checks (run

redis-cli --latency

and

redis-cli --intrinsic-latency

) ◦ These give live latency samples; elevated values show network or server-side delays. •

slowlog

(run

redis-cli slowlog get

and

slowlog len

) ◦ Shows commands exceeding configured latency thresholds; recurring entries point to problematic commands. • Persistence / background I/O stats (

redis-cli info persistence

) ◦ Look for

rdb_bgsave_in_progress

,

aof_rewrite_in_progress

,

aof_pending_bio_fsync

— background saves/fsyncs can cause pauses. • Network throughput (

redis-cli info server

and

redis-cli info stats

for

total_net_input_bytes

/

total_net_output_bytes

) ◦ High throughput may saturate network; check NIC metrics on the host too. Quick commands bundle: •

redis-cli --latency

•

redis-cli info stats

•

redis-cli info memory

•

redis-cli info clients

•

redis-cli info persistence

•

redis-cli slowlog get

If you have Prometheus, deploy redis_exporter and watch these metrics: • redis_memory_used_bytes • redis_commands_processed_total and redis_installed_commands (or instantaneous ops) • redis_connected_clients • redis_slowlog_latest_entries / custom slowlog exporter • redis_net_input_bytes_total / redis_net_output_bytes_total Next steps I can help with: • A short Prometheus queries/dashboard panels to visualize these metrics and correlate with LiveKit API latency. • A checklist for Redis tuning (maxclients, persistence settings, CPU sizing, or moving to Redis Cluster) if we confirm Redis is the bottleneck. Which of those would you like me to produce first? Sources: Going beyond a single-core | Custom agent deployments | LiveKit docs | Introducing LiveKit Inference: A unified model interface for voice AI | examples/frontdesk | livekit/livekit_composite | DeepWiki | examples/drive-thru | livekit/livekit_composite | DeepWiki

Give me redis tuning advice

Thanks — responding with standalone Redis tuning advice to reduce psrpc/pubsub latency under high CCU in your self-hosted LiveKit cluster. Quick checklist (apply progressively, test after each change): 1) Reduce disk I/O and persistence overhead • If you can tolerate lost messages on restart, disable persistence: ◦

save ""

◦

appendonly no

• If persistence is required, prefer infrequent RDB snapshots and

appendfsync everysec

for AOF. 2) Prevent swapping and memory pressure • Set

vm.swappiness=1

. • Ensure enough RAM so

used_memory_rss

<< available RAM. • Configure

maxmemory

and an eviction policy (e.g.,

maxmemory <size>

and

allkeys-lru

) to avoid OOM kills. 3) Tune TCP / connection handling • In redis.conf:

tcp-backlog 511

• Sysctl: ◦

net.core.somaxconn=511

◦

net.ipv4.tcp_max_syn_backlog=1024

• Increase file descriptor limits (

ulimit -n

) and set

maxclients

appropriately. 4) Use I/O threads and CPU tuning (Redis 6+) •

io-threads <n>

(e.g., 2–4) and

io-threads-do-reads yes

if host has spare cores. • Pin Redis to dedicated CPUs (taskset / CPU affinity) to reduce contention. 5) Reduce event-loop blocking • Ensure subscribers/processors avoid long/blocking commands. • Tune

slowlog-log-slower-than

and monitor

SLOWLOG GET

to find blocking ops. • Avoid heavy Lua scripts or long-running commands in pub/sub paths. 6) Network and co-location • Co-locate Redis and LiveKit processes on the same host or same rack to reduce network latency; consider Unix domain socket if on same host. • Ensure NICs and network path aren’t saturated; monitor host NIC metrics. 7) Persistence / fsync tuning (if using AOF) • Use

appendfsync everysec

rather than

always

. • Put AOF/RDB on fast storage (NVMe) and use

noatime

on mounts. 8) Memory allocator and OS settings • Disable Transparent Huge Pages (THP): ◦

echo never > /sys/kernel/mm/transparent_hugepage/enabled

• Consider jemalloc (default) tuning; avoid switching allocators unless tested. 9) Scale and isolation • For sustained high throughput consider Redis Cluster or sharding pub/sub workloads across multiple Redis instances. • Use separate Redis instances for distinct workloads (signaling vs presence) to isolate spikes. 10) Monitoring to validate changes • Watch:

used_memory

,

used_memory_rss

,

instantaneous_ops_per_sec

,

connected_clients

,

blocked_clients

,

total_net_input_bytes

,

total_net_output_bytes

, slowlog entries, and latency via

redis-cli --latency

. • Use redis_exporter + Prometheus and alert on rising

instantaneous_ops_per_sec

, high

blocked_clients

, and increasing `latency`/slowlog rate. Commands to run for quick checks: •

redis-cli --latency

•

redis-cli info stats

•

redis-cli info memory

•

redis-cli info clients

•

redis-cli info persistence

•

redis-cli slowlog get

If you want, I can: • Propose specific redis.conf snippets for your instance size, • Draft Prometheus queries/Grafana panels to monitor the key metrics above, • Suggest a sharding plan based on expected ops/sec and client counts. Which of those would you like next? Sources: How We Built a Globally Distributed WebRTC Mesh Network