colossal-glass-16853
03/11/2026, 1:42 PMbpfrb/0 25%
bpfrb/1 25%
bpfrb/2 25%
bpfrb/3 25%
I've tried the following without success:
Upgrading and downgrading LiveKit Server versions (tested 1.8.0, 1.8.3, 1.9.x)
Adding interface filters in livekit.yaml (interfaces: includes: [ens192], excludes: [lo])
• Adjusting batch_io settings (batch_size, max_flush_interval)
The VM only has 2 network interfaces: lo and ens192. No Docker bridge or veth interfaces present.
When I run LiveKit SIP alone (without LiveKit Server on the same machine), CPU usage is significantly lower. So the issue seems to be specific to running both services together.
Questions:
1. Is this level of CPU usage from bpfrb threads expected when running LiveKit Server and SIP on the same VM?
2. Is there a known configuration to reduce bpfrb CPU consumption?
3. Are there any plans to optimize this in future versions?tall-airline-16756
03/11/2026, 1:43 PMtall-airline-16756
03/11/2026, 1:43 PMuname -a)
1. LiveKit components and how they’re run
◦ Exact LiveKit Server and LiveKit SIP versions currently running together (you mentioned testing 1.8.0, 1.8.3, 1.9.x — which one is running now?).
◦ Are both services running as system processes, inside containers, or via some process manager? If containers, share the runtime (Docker, Podman) and whether they use host networking.
1. Network details
◦ Output of ip -o link (or ip addr) showing interfaces and MTU.
◦ Whether the SIP endpoints and browser/clients are on the same LAN or remote over internet.
1. LiveKit configuration excerpts
◦ Your livekit.yaml network-related section (interfaces, batch_io settings, and any RTP/UDP port ranges). (If you prefer, paste only the relevant fields.)
1. CPU affinity / cgroups
◦ Any cpuset/cgroup or CPU pinning in place for either process?
1. Observed behaviour
◦ You said SIP-alone consumes much less CPU — approximately how much CPU does that process consume alone per call?
◦ Does CPU jump immediately when a call connects, or ramp up over time?
What I can recommend immediately (do these while you collect the above): 1) Verify kernel features and offloads
• Ensure kernel is reasonably recent (older kernels can make eBPF processing heavier). Share uname -r if unsure.
• Check NIC offloads (e.g., GRO/LRO/TSO): run ethtool -k ens192. If large receive/offload features are disabled, that can increase packet processing load. Toggling those has an impact; if they’re off, try enabling them temporarily to test.
2) Confirm BPF program activity and counts
• Get a snapshot of per-thread CPU usage with top -H -p <pid-of-sip> or ps -L -p <pid> while the call is active. That confirms which process owns the bpfrb threads.
• Collect perf top or perf record on one of the bpfrb threads for a short period (if you can) to see where time is spent. If you can share results I can help interpret them.
3) Network interface filtering vs binding
• Your interfaces: includes: [ens192], excludes: [lo] was a good attempt. Also verify that LiveKit isn’t listening on multiple interfaces or creating additional veths at runtime (some SIP setups or NAT helpers create virtual interfaces).
• If feasible, run SIP and LiveKit Server with stricter binding so that SIP handles only its ports and Server only its own RTP/U/V/UDP ranges. That can reduce cross-processing.
4) batch_io tuning suggestions to experiment with
• You already tried batch_size and max_flush_interval. Try increasing batch_size substantially (e.g., 2x–4x) and increasing max_flush_interval slightly to reduce flush frequency, then test CPU impact. These changes trade latency for CPU — if real-time constraints allow it, you can reduce interrupts/flush overhead.
• Test one change at a time and measure.
5) Separate hosts or resource controls
• Temporary workaround: run SIP and Server on separate VMs or set CPU pinning/cgroup limits so one service can’t contend. You noted SIP alone is fine, so separating them is an effective workaround if immediate optimization isn’t possible.
6) Collect logs and metrics
• Enable or collect LiveKit metrics (if enabled) and CPU/memory snapshots during the call. Metrics can show whether the spike correlates with packet rates or other events.
If you can share the details requested above (kernel, how services are run, interface output, and the small livekit.yaml network section), I’ll give concrete configuration changes and a step-by-step test plan to reduce bpfrb CPU usage. If you can also share a short top -H snapshot of the threads while the call is active, that will help me point to the exact cause faster.colossal-glass-16853
03/11/2026, 1:47 PMdocker CLI via Podman emulation)
Network interfaces:
• lo — loopback, MTU 65536
• ens192 — primary NIC, MTU 1500, state UP
No Docker bridge, veth, or additional virtual interfaces present.tall-airline-16756
03/11/2026, 1:47 PMbatch_size (try 2x–4x current) and increase max_flush_interval slightly. Measure CPU vs latency.
◦ If interfaces.includes is set to ens192, also set explicit bind_addrs or ip fields for SIP and server sockets if available, to ensure each component only processes its required traffic.
6) Test isolating components (fast workaround)
• If possible, run SIP and Server in separate Podman network namespaces or separate VMs. You noted SIP alone is fine — separating them is an effective short-term mitigation while we tune.
• Alternatively, use cgroups/cpu pinning so SIP and Server don’t compete for the same CPU cores.
7) Collect LiveKit logs and metrics
• Enable or gather LiveKit logs/metrics during the test. Share any RTP packet-rate, dropped-packet, or worker-related metrics if available.
• Why: helps determine whether high packet rates, retransmits, or something else is driving the CPU.
What to paste back here (so I can give specific config changes)
• Output of: uname -r (you already gave kernel, but full uname -a is useful)
• Output of: ethtool -k ens192
• A short top -H -p <pid> snapshot showing the bpfrb threads and their CPU at the time of an active call
• The interfaces + batch_io section from your livekit.yaml
• podman inspect networking mode for both containers
If you prefer, start with ethtool and top snapshots — those often point to NIC offload or thread ownership issues quickly. Once you paste results I’ll recommend exact livekit.yaml changes and a concise test plan to reduce CPU or isolate the root cause.colossal-glass-16853
03/11/2026, 1:49 PMgeneric-receive-offload: on ✅
• large-receive-offload: on ✅
• generic-segmentation-offload: on ✅
• tcp-segmentation-offload: on ✅tall-airline-16756
03/11/2026, 1:50 PMhost network mode or something else. If both use host networking, that explains why both see packets directly on the host NIC (and why they can interact).
3) The relevant livekit.yaml network section
• Paste just the interfaces, batch_io, and any explicit bind_addrs / bind_ips / RTP port ranges you set.
• If you prefer, redact any secrets — I only need the networking-related fields.
Quick guidance you can try in parallel (non-destructive, low-risk tests)
• Isolate network namespaces: If both containers are using host mode, try running the SIP container in its own network namespace (non-host) temporarily so it doesn’t attach to the host NIC directly. This can reveal whether shared host packet processing is the cause.
• CPU pinning: Temporarily pin SIP and Server to separate CPU cores via Podman/cgroups to avoid contention and confirm this is a scheduling/processing interaction.
• batch_io experiments: Increase batch_size (2x–4x) and increase max_flush_interval modestly to reduce flush frequency. Test one change at a time and measure CPU and latency.
• perf snapshot (optional if you can): sudo perf record -p <thread-pid> -- sleep 10; sudo perf report — this shows where CPU cycles are spent if the above steps don’t surface the cause.
Why I’m asking for those items
• The top snapshot will tell me exactly which binary/thread owns the bpfrb work. If bpfrb threads belong to the LiveKit Server process, the tuning path differs from the SIP container owning them.
• Podman network mode determines whether packets are being processed in the host network stack (and by BPF attached there) for both services simultaneously.
• livekit.yaml will let me propose precise settings (bind addresses, interface filtering, batch_io values) rather than generic advice.
If you paste the top output, Podman network modes, and the network snippet from livekit.yaml, I’ll give a short, concrete plan with exact config edits and tests to reduce bpfrb CPU usage.colossal-glass-16853
03/11/2026, 1:51 PMnetwork_mode: host. No bridge or veth interfaces are present on the host.
NIC offloads are all enabled as shown previously.
Given that:
• Host networking is confirmed
• NIC offloads are all on
• Only 2 interfaces exist (lo and ens192)
• Interface filtering was already tested (includes: ens192, excludes: lo) with no improvement
• Version downgrade from 1.9.1 to 1.8.0 also had no effect
The bpfrb threads still consume ~100% of one full CPU core during a single SIP call.
Can you now provide specific configuration changes to reduce bpfrb CPU usage? Or is this a known inefficiency when running LiveKit Server and SIP together on the same host with host networking?tall-airline-16756
03/11/2026, 1:51 PMpodman inspect --format '{{.State.Pid}}' <container>).
• Run (replace <pid> with the container main PID that handles the bpfrb threads):
◦ top -H -p <pid> -b -n 1 | egrep "bpfrb|[0-9]+\.[0-9]+|R|S"
• Alternatively:
◦ ps -L -p <pid> -o pid,tid,pcpu,comm | egrep "bpfrb|sip|livekit|<pid>"
What I need from you: the lines that show the bpfrb threads with their TID and CPU%.
2) livekit.yaml network section (so I can propose precise field changes)
• Paste the interfaces, batch_io, and any bind_addrs / bind_ips / RTP port ranges currently in your livekit.yaml.
What I need from you: those exact fields (redact any secrets if present).
Why these two items matter
• The top/ps output confirms whether bpfrb threads belong to the SIP container, the LiveKit Server container, or both. The tuning steps differ depending on ownership.
• The livekit.yaml snippet lets me recommend explicit changes (exact values for batch_size, max_flush_interval, and any bind/interface changes) rather than generic advice.
If you want to try a quick experiment now while you gather the outputs:
• Temporarily pin the two containers to separate CPU cores via Podman/cgroup to see if isolating CPUs reduces observable impact (this is only a test to validate contention vs eBPF cost).
• Increase batch_size by 2×–4× and raise max_flush_interval slightly, then test one change at a time and measure CPU/latency. I’ll give precise numbers after I see your livekit.yaml.
Send the two outputs (top/ps lines and the livekit.yaml network section) and I’ll reply with exact configuration edits and a concise test plan tailored to your setup.colossal-glass-16853
03/11/2026, 1:52 PMtall-airline-16756
03/11/2026, 1:53 PMbatch_io.batch_size to reduce how often packet batches are flushed. Try 2x–4x your current value.
• Increase batch_io.max_flush_interval slightly to reduce flush frequency (this trades a small amount of increased latency for lower CPU).
• Change one parameter at a time and measure CPU and end-to-end latency to ensure call quality remains acceptable.
4) Reduce incoming packet rate / per-call load
• Ensure the SIP bridge is sending only the necessary streams to the Server:
◦ Disable simulcast/multiple spatial layers for SIP tracks (SIP endpoints don’t need simulcast). This reduces the number of parallel streams the Server must process.
◦ Apply bitrate caps for SIP sessions when acceptable. Lowering encode bitrates reduces packets per second and CPU usage.
• If LiveKit Server is doing any transcoding or additional processing for SIP-originated tracks, avoid or offload that work if possible.
5) CPU isolation and cgroups as a diagnostic/workaround
• Pin LiveKit Server and LiveKit SIP to separate CPU cores using cgroups / podman CPU pinning to reduce interference and confirm whether isolating CPUs mitigates the user-visible impact.
• If isolating cores reduces the problem, that confirms contention/interrupt processing is a factor and separation or horizontal scaling is recommended.
6) Confirm and collect diagnostic traces (if you want deeper root cause)
• Collect a lightweight perf/top snapshot while a call is active to confirm where Server time is spent (this confirms eBPF/bpf program hot paths vs syscall/packet-handling).
• If you share the interfaces and batch_io fields from your livekit.yaml I can recommend exact values for batch_size and max_flush_interval for your workload.
Notes about bpfrb threads and product-level tuning
• There is no documented LiveKit Server option to change bpfrb thread count or polling frequency directly. The bpfrb activity you’re seeing is coming from low-level packet/BPF handling used by the underlying networking stack that LiveKit relies on.
• Because these threads are part of low-level packet handling tied to how the process receives packets on host networking, the most effective fixes are: reduce packet rate, bind sockets more narrowly, or run Server and SIP on separate hosts/namespaces so the Server doesn’t receive all SIP-originating packets at NIC line-rate.
Recommended immediate experiment (short checklist)
1. Temporarily run the SIP container without host networking (or move it to another VM). Measure CPU on the Server host during a call. If CPU drops, proceed to separate them in production.
2. If separation isn’t viable, increase batch_io.batch_size 2× and batch_io.max_flush_interval modestly. Restart Server and measure CPU/latency.
3. Disable simulcast for SIP tracks and apply a modest bitrate cap on the SIP bridge. Test CPU impact.
4. If you want, paste your interfaces and batch_io config and I’ll suggest exact numeric values to try.
If you want, I can also:
• Suggest exact batch_size and max_flush_interval values based on your current settings.
• Help craft a short test script/checklist to reproduce and verify CPU improvements.
Tell me which of the above you want to try first (move to separate host, change batch_io, disable simulcast, or share your livekit.yaml) and I’ll give the specific commands/config edits.
Sources: Going beyond a single-core | Introduction | LiveKit Documentation | livekit/livekit_composite | DeepWiki | Announcing LiveKit Cloud | Bringing Zoom&#39;s end-to-end optimizations to WebRTCcolossal-glass-16853
03/11/2026, 1:58 PMtall-airline-16756
03/11/2026, 1:58 PMrtc and batch_io settings I asked for, and confirmed the Server owns the bpfrb threads. I’ll be direct and actionable: what you can change in your LiveKit Server config, why it matters, and a short test plan so you can measure impact.
Key facts up front
• There is no LiveKit Server setting to directly change or limit bpfrb thread count or polling frequency. Those threads are low-level packet/BPF handling tied to how the process receives packets on host networking.
• But you can reduce the packet-processing load the Server receives (which lowers bpfrb CPU usage) via the config values and deployment choices below.
Recommended configuration changes (apply one at a time and measure) 1) Reduce packet churn with batch_io
• Change:
◦ batch_io.batch_size: increase from 512 → 1024 (or 2048 if stable)
◦ batch_io.max_flush_interval: increase from 10ms → 20ms (then try 50ms if latency tolerance allows)
• Why: Larger batches + slightly longer flush intervals reduce the number of packet processing events and syscalls the Server must handle per second. This trades a small increase in packet processing latency for lower CPU usage.
• How to test: Restart the Server, make a SIP call, measure the bpfrb threads CPU %. If CPU falls but latency rises unacceptably, try smaller steps.
2) Narrow what the Server listens to (bind addresses / RTP port range)
• Change (conceptual — apply based on your deployment):
◦ Ensure Server is bound only to the IP(s) it needs. With host networking the process currently sees all NIC traffic. If possible, configure Server to bind explicitly to the host IP rather than wildcard, or place the Server in a network namespace to reduce unsolicited traffic.
◦ Your current port_range_start: 50000, port_range_end: 60000 is good. Ensure SIP bridge only uses ports within that range and that Server doesn’t have to scan or accept other host traffic.
• Why: Reducing the set of packets reaching the Server’s sockets reduces eBPF and packet dispatch work.
3) Disable simulcast / reduce streams for SIP
• On the SIP bridge or in how you bridge SIP to LiveKit, ensure SIP tracks are sent as single-quality (no simulcast) and cap bitrate if feasible.
• Why: Simulcast multiplies streams/packets per participant. SIP endpoints don’t benefit from simulcast, so disabling it substantially reduces packet rate going through the Server.
4) Reduce per-call packet rate (bitrate caps)
• Apply reasonable bitrate limits for SIP participants (e.g., 300–800 kbps for audio+video as appropriate).
• Why: Lower PTS/payload sizes and fewer packets/second reduce packet-processing CPU.
5) Consider network namespace separation (quick mitigation)
• Run Server in host networking but run SIP container without host networking (or vice versa) so only one process receives host NIC packets directly.
• Why: This prevents the Server from observing all SIP-originated packets at NIC line-rate and is often the fastest mitigation.
6) CPU pinning as a diagnostic and temporary relief
• Pin the Server and SIP containers to separate CPU cores using Podman/cgroup settings to reduce contention.
• Why: If isolating CPUs reduces observable user impact, that confirms contention/interrupt processing as a contributor. It’s a useful short-term mitigation while you tune.
Short prioritized test plan
1. Change only batch_io:
◦ Set batch_size to 1024 and max_flush_interval to 20ms.
◦ Restart Server, run a SIP call, measure bpfrb thread CPU and end-to-end latency.
1. If CPU still high, disable simulcast for SIP tracks and cap bitrate on the SIP side; retest.
2. If CPU still high or latency unacceptable, try moving SIP container off host networking (or to another host) to verify separation fixes it.
3. As a final step if separation improves things, consider permanently running Server and SIP on separate hosts or networks.
What to measure
• Per-thread CPU (top -H -p <server-pid>) while a call is active.
• End-to-end call latency / packet loss metrics.
• RTP packets/sec for the SIP->Server stream (if available).
• Observe whether changes reduce bpfrb CPU and whether call quality remains acceptable.
If you want, I can suggest exact numeric values beyond the initial batch_io changes (for example: batch_size=2048 / max_flush_interval=50ms) — but I recommend stepping up conservatively to avoid unexpected latency impact.
If you’d like, run the first test (batch_size: 1024, max_flush_interval: 20ms) and paste the before/after top thread CPU lines. I’ll interpret the results and recommend next exact steps.colossal-glass-16853
03/11/2026, 2:06 PMtall-airline-16756
03/11/2026, 2:07 PM