Subject: High CPU usage from bpfrb threads with SI...
# ask-ai
c
Subject: High CPU usage from bpfrb threads with SIP + LiveKit Server on same VM Hi, I'm running LiveKit Server and LiveKit SIP on the same VM (4 CPU cores, 8GB RAM). When a single SIP call comes in, the bpfrb threads consume nearly 100% of one full CPU core (each of the 4 threads takes ~25%):
Copy code
bpfrb/0  25%
bpfrb/1  25%
bpfrb/2  25%
bpfrb/3  25%
I've tried the following without success: Upgrading and downgrading LiveKit Server versions (tested 1.8.0, 1.8.3, 1.9.x) Adding interface filters in livekit.yaml (
interfaces: includes: [ens192], excludes: [lo]
) • Adjusting
batch_io
settings (
batch_size
,
max_flush_interval
) The VM only has 2 network interfaces:
lo
and
ens192
. No Docker bridge or veth interfaces present. When I run LiveKit SIP alone (without LiveKit Server on the same machine), CPU usage is significantly lower. So the issue seems to be specific to running both services together. Questions: 1. Is this level of CPU usage from bpfrb threads expected when running LiveKit Server and SIP on the same VM? 2. Is there a known configuration to reduce bpfrb CPU consumption? 3. Are there any plans to optimize this in future versions?
t
Thanks for reaching out, Eray. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a clear and helpful report. I’m responding to the three questions you asked about the high CPU usage from the bpfrb threads when running LiveKit Server and LiveKit SIP together. Short answers first • No: that much CPU from bpfrb threads is not expected as a normal result of a single SIP call. • There are configuration levers that can reduce CPU, but I need a few more details to give targeted advice. • I can’t confirm specific planned work, but I can help you collect diagnostics and tune the system now and suggest follow-ups if an upstream optimization is needed. Before I give specific tuning suggestions, please help me with these quick clarifying details (these will narrow the correct advice): 1. Kernel and distro ◦ Which Linux distribution and kernel version are you running? (output of
uname -a
) 1. LiveKit components and how they’re run ◦ Exact LiveKit Server and LiveKit SIP versions currently running together (you mentioned testing 1.8.0, 1.8.3, 1.9.x — which one is running now?). ◦ Are both services running as system processes, inside containers, or via some process manager? If containers, share the runtime (Docker, Podman) and whether they use host networking. 1. Network details ◦ Output of
ip -o link
(or
ip addr
) showing interfaces and MTU. ◦ Whether the SIP endpoints and browser/clients are on the same LAN or remote over internet. 1. LiveKit configuration excerpts ◦ Your
livekit.yaml
network-related section (interfaces, batch_io settings, and any RTP/UDP port ranges). (If you prefer, paste only the relevant fields.) 1. CPU affinity / cgroups ◦ Any cpuset/cgroup or CPU pinning in place for either process? 1. Observed behaviour ◦ You said SIP-alone consumes much less CPU — approximately how much CPU does that process consume alone per call? ◦ Does CPU jump immediately when a call connects, or ramp up over time? What I can recommend immediately (do these while you collect the above): 1) Verify kernel features and offloads • Ensure kernel is reasonably recent (older kernels can make eBPF processing heavier). Share
uname -r
if unsure. • Check NIC offloads (e.g., GRO/LRO/TSO): run
ethtool -k ens192
. If large receive/offload features are disabled, that can increase packet processing load. Toggling those has an impact; if they’re off, try enabling them temporarily to test. 2) Confirm BPF program activity and counts • Get a snapshot of per-thread CPU usage with
top -H -p <pid-of-sip>
or
ps -L -p <pid>
while the call is active. That confirms which process owns the bpfrb threads. • Collect
perf top
or
perf record
on one of the bpfrb threads for a short period (if you can) to see where time is spent. If you can share results I can help interpret them. 3) Network interface filtering vs binding • Your
interfaces: includes: [ens192], excludes: [lo]
was a good attempt. Also verify that LiveKit isn’t listening on multiple interfaces or creating additional veths at runtime (some SIP setups or NAT helpers create virtual interfaces). • If feasible, run SIP and LiveKit Server with stricter binding so that SIP handles only its ports and Server only its own RTP/U/V/UDP ranges. That can reduce cross-processing. 4) batch_io tuning suggestions to experiment with • You already tried
batch_size
and
max_flush_interval
. Try increasing
batch_size
substantially (e.g., 2x–4x) and increasing
max_flush_interval
slightly to reduce flush frequency, then test CPU impact. These changes trade latency for CPU — if real-time constraints allow it, you can reduce interrupts/flush overhead. • Test one change at a time and measure. 5) Separate hosts or resource controls • Temporary workaround: run SIP and Server on separate VMs or set CPU pinning/cgroup limits so one service can’t contend. You noted SIP alone is fine, so separating them is an effective workaround if immediate optimization isn’t possible. 6) Collect logs and metrics • Enable or collect LiveKit metrics (if enabled) and CPU/memory snapshots during the call. Metrics can show whether the spike correlates with packet rates or other events. If you can share the details requested above (kernel, how services are run, interface output, and the small
livekit.yaml
network section), I’ll give concrete configuration changes and a step-by-step test plan to reduce bpfrb CPU usage. If you can also share a short
top -H
snapshot of the threads while the call is active, that will help me point to the exact cause faster.
c
Sure! Here's the follow-up with the details you requested: Kernel: Linux 5.14.0-611.27.1.el9_7.x86_64 (RHEL/Rocky Linux 9), SMP PREEMPT_DYNAMIC LiveKit Server version: v1.9.1 (running in Podman container) LiveKit SIP version: running alongside LiveKit Server on the same VM LiveKit Worker version: v0.4.4 (running on a separate VM) How services are run: Podman containers (using
docker
CLI via Podman emulation) Network interfaces: •
lo
— loopback, MTU 65536 •
ens192
— primary NIC, MTU 1500, state UP No Docker bridge, veth, or additional virtual interfaces present.
t
Thanks — this is very helpful. I’m responding to your three original questions and will include targeted next steps and tests now that I have the kernel, container/runtime, and interface details. Short answers • Is this level of CPU usage expected? No — seeing nearly 100% of a core split across four bpfrb threads for a single SIP call is not typical. • Is there a known configuration to reduce bpfrb CPU consumption? Yes — there are a few knobs and diagnostics that usually help (NIC offloads, BPF activity, batch_io tuning, and interface binding). I’ll list concrete tests below. • Plans to optimize in future releases? I can’t confirm roadmap items here, but we can collect diagnostics to determine whether the cause is configuration-related or a code-level inefficiency that should be investigated further. Please try the following checks and tests (in order). Each step explains what to run, what to look for, and why it matters. 1) Confirm which process owns the bpfrb threads • Run while a call is active: ◦ ps -L -p <pid-of-livekit-sip-or-server> (or) top -H -p <pid> • What to look for: confirm the bpfrb threads are owned by the SIP container process (or the Server). This verifies which binary’s BPF program is active. 2) Check NIC offloads (quick test) • Run on the host: ◦ ethtool -k ens192 • What to look for: whether large-receive/offload features (GRO/LRO/TSO/UFO) are enabled. • Why: disabled offloads increase per-packet CPU work and can amplify eBPF processing cost. If offloads are off, try enabling them temporarily and re-run the SIP call test. 3) Capture a per-thread CPU snapshot • While the call is active, capture threads: ◦ top -H -p <pid> (or) ps -L -p <pid> • Share that output so I can confirm thread names and exact CPU shares. • If you can run perf and are comfortable sharing results: ◦ sudo perf top -p <pid> (or) sudo perf record -g -p <thread-pid> -- sleep 10; sudo perf report • Why: this shows where time is spent and whether the cost is in BPF program execution, syscall/interrupt handling, or elsewhere. 4) Verify Podman networking mode • Confirm Podman container networking mode for both containers (host vs bridge): ◦ podman inspect <container> | grep -i NetworkMode -A2 • Why: host networking vs bridge changes packet flow. Host mode means packet handling happens on host NIC and BPF programs apply directly; bridge can add veth and different overhead. You said no veths present — confirm both containers are using host networking. If they are, try running SIP container with network=host but Server in a separate network namespace to see if isolating namespaces helps. 5) Review and test livekit.yaml network & batch_io settings • Please paste the relevant network section you used (interfaces and batch_io fields). Example fields I’m interested in: ◦ interfaces: includes/excludes ◦ batch_io: batch_size, max_flush_interval ◦ any explicit bind IPs or RTP port ranges • Tests to run: ◦ Increase
batch_size
(try 2x–4x current) and increase
max_flush_interval
slightly. Measure CPU vs latency. ◦ If
interfaces.includes
is set to
ens192
, also set explicit
bind_addrs
or
ip
fields for SIP and server sockets if available, to ensure each component only processes its required traffic. 6) Test isolating components (fast workaround) • If possible, run SIP and Server in separate Podman network namespaces or separate VMs. You noted SIP alone is fine — separating them is an effective short-term mitigation while we tune. • Alternatively, use cgroups/cpu pinning so SIP and Server don’t compete for the same CPU cores. 7) Collect LiveKit logs and metrics • Enable or gather LiveKit logs/metrics during the test. Share any RTP packet-rate, dropped-packet, or worker-related metrics if available. • Why: helps determine whether high packet rates, retransmits, or something else is driving the CPU. What to paste back here (so I can give specific config changes) • Output of: uname -r (you already gave kernel, but full
uname -a
is useful) • Output of: ethtool -k ens192 • A short
top -H -p <pid>
snapshot showing the bpfrb threads and their CPU at the time of an active call • The
interfaces
+
batch_io
section from your
livekit.yaml
• podman inspect networking mode for both containers If you prefer, start with ethtool and top snapshots — those often point to NIC offload or thread ownership issues quickly. Once you paste results I’ll recommend exact livekit.yaml changes and a concise test plan to reduce CPU or isolate the root cause.
c
•
generic-receive-offload: on
✅ •
large-receive-offload: on
✅ •
generic-segmentation-offload: on
✅ •
tcp-segmentation-offload: on
✅
t
Thanks — great to see NIC offloads are enabled. That helps rule out a common cause. Next steps I’ll need so I can give targeted tuning and the shortest test plan: 1) Confirm thread ownership and exact CPU snapshot • While a call is active, run: ◦ top -H -p <pid-of-the-container-process-on-host> ◦ Or inside the container (if you can): top -H -p 1 (or use the container process PID) • Paste the lines showing the bpfrb threads and their thread IDs/CPU% so I can confirm which container/process owns them. 2) Podman networking mode for each container • Run on the host: ◦ podman inspect <livekit-server-container> --format '{{.HostConfig.NetworkMode}}' ◦ podman inspect <livekit-sip-container> --format '{{.HostConfig.NetworkMode}}' • Tell me whether each container is using
host
network mode or something else. If both use host networking, that explains why both see packets directly on the host NIC (and why they can interact). 3) The relevant livekit.yaml network section • Paste just the
interfaces
,
batch_io
, and any explicit
bind_addrs
/
bind_ips
/ RTP port ranges you set. • If you prefer, redact any secrets — I only need the networking-related fields. Quick guidance you can try in parallel (non-destructive, low-risk tests) • Isolate network namespaces: If both containers are using host mode, try running the SIP container in its own network namespace (non-host) temporarily so it doesn’t attach to the host NIC directly. This can reveal whether shared host packet processing is the cause. • CPU pinning: Temporarily pin SIP and Server to separate CPU cores via Podman/cgroups to avoid contention and confirm this is a scheduling/processing interaction. • batch_io experiments: Increase
batch_size
(2x–4x) and increase
max_flush_interval
modestly to reduce flush frequency. Test one change at a time and measure CPU and latency. • perf snapshot (optional if you can): sudo perf record -p <thread-pid> -- sleep 10; sudo perf report — this shows where CPU cycles are spent if the above steps don’t surface the cause. Why I’m asking for those items • The top snapshot will tell me exactly which binary/thread owns the bpfrb work. If bpfrb threads belong to the LiveKit Server process, the tuning path differs from the SIP container owning them. • Podman network mode determines whether packets are being processed in the host network stack (and by BPF attached there) for both services simultaneously. • livekit.yaml will let me propose precise settings (bind addresses, interface filtering, batch_io values) rather than generic advice. If you paste the top output, Podman network modes, and the network snippet from livekit.yaml, I’ll give a short, concrete plan with exact config edits and tests to reduce bpfrb CPU usage.
c
Both containers (livekit-server and livekit-sip) are running with
network_mode: host
. No bridge or veth interfaces are present on the host. NIC offloads are all enabled as shown previously. Given that: • Host networking is confirmed • NIC offloads are all on • Only 2 interfaces exist (lo and ens192) • Interface filtering was already tested (includes: ens192, excludes: lo) with no improvement • Version downgrade from 1.9.1 to 1.8.0 also had no effect The bpfrb threads still consume ~100% of one full CPU core during a single SIP call. Can you now provide specific configuration changes to reduce bpfrb CPU usage? Or is this a known inefficiency when running LiveKit Server and SIP together on the same host with host networking?
t
Thanks — this is very useful context. I understand you’ve confirmed host networking for both containers, NIC offloads are enabled, interface filtering and version changes didn’t help, and bpfrb threads still take ~100% of one core for a single SIP call. I can help with specific config changes and tests, but I need two short diagnostics so I can target the right fixes. Please run the two commands below while a SIP call is active and paste their outputs. These will tell me which process/thread runs the bpfrb work and let me recommend exact livekit.yaml edits or container/network changes. 1) Per-thread CPU snapshot (shows which process owns the bpfrb threads) • On the host, find the container main PID for each container (sudo podman ps -a and note the PID column or use
podman inspect --format '{{.State.Pid}}' <container>
). • Run (replace <pid> with the container main PID that handles the bpfrb threads): ◦ top -H -p <pid> -b -n 1 | egrep "bpfrb|[0-9]+\.[0-9]+|R|S" • Alternatively: ◦ ps -L -p <pid> -o pid,tid,pcpu,comm | egrep "bpfrb|sip|livekit|<pid>" What I need from you: the lines that show the bpfrb threads with their TID and CPU%. 2) livekit.yaml network section (so I can propose precise field changes) • Paste the
interfaces
,
batch_io
, and any
bind_addrs
/
bind_ips
/ RTP port ranges currently in your
livekit.yaml
. What I need from you: those exact fields (redact any secrets if present). Why these two items matter • The top/ps output confirms whether bpfrb threads belong to the SIP container, the LiveKit Server container, or both. The tuning steps differ depending on ownership. • The livekit.yaml snippet lets me recommend explicit changes (exact values for
batch_size
,
max_flush_interval
, and any bind/interface changes) rather than generic advice. If you want to try a quick experiment now while you gather the outputs: • Temporarily pin the two containers to separate CPU cores via Podman/cgroup to see if isolating CPUs reduces observable impact (this is only a test to validate contention vs eBPF cost). • Increase
batch_size
by 2×–4× and raise
max_flush_interval
slightly, then test one change at a time and measure CPU/latency. I’ll give precise numbers after I see your
livekit.yaml
. Send the two outputs (top/ps lines and the livekit.yaml network section) and I’ll reply with exact configuration edits and a concise test plan tailored to your setup.
c
We identified the root cause: the bpfrb threads belong to the LiveKit Server container, not the SIP container. When running LiveKit SIP alone on a VM, CPU usage is normal during calls. When LiveKit Server is added to the same VM, bpfrb threads spike to ~100% of one CPU core for a single call. This confirms the issue is in how LiveKit Server processes media traffic coming from the SIP bridge, not in the SIP service itself. Given this, what specific LiveKit Server configuration changes can reduce bpfrb CPU usage? Is there a way to limit the number of bpfrb threads or reduce their polling frequency?
t
Thanks — this is very helpful and clear. You’ve confirmed the bpfrb threads belong to the LiveKit Server container and the SIP service itself is not the cause. I’ll respond directly to your questions and give concrete actions you can take now. Summary answers • Is there a LiveKit Server configuration to directly limit bpfrb threads or their polling frequency? ◦ No. There is no exposed LiveKit Server configuration specifically named to limit bpfrb thread count or change their polling frequency. • Can we reduce the observed CPU usage through server-side configuration or deployment changes? ◦ Yes. While bpfrb threads themselves aren’t directly configurable, there are practical configuration and deployment changes that reduce the packet-processing load LiveKit Server must handle and therefore lower CPU usage. Concrete, prioritized steps to reduce CPU usage now 1) Separate workloads (best immediate mitigation) • Move LiveKit Server off the same host as the SIP bridge (run SIP on a separate VM/container host), or run the LiveKit Server container in a different machine. You already observed SIP-alone is fine — separating them prevents the Server’s bpfrb threads from seeing and processing SIP-originating packets at line-rate on the host NIC. • If moving to a different host isn’t immediately possible, run one service in a dedicated network namespace rather than host networking so that the Server does not receive all host-level packets directly. 2) Restrict which packets LiveKit Server processes • Bind Server sockets explicitly to the IPs/ports it needs (so it only receives relevant traffic). If your current config uses host networking and the server accepts packets on all addresses, explicitly binding reduces unnecessary packet reception. • Confirm and, if available, set `bind_addrs`/`bind_ips` or explicitly set the RTP/RTCP port range in your livekit.yaml so Server only listens on the intended addresses and ports. 3) Batch IO tuning (reduce per-packet processing) • Increase
batch_io.batch_size
to reduce how often packet batches are flushed. Try 2x–4x your current value. • Increase
batch_io.max_flush_interval
slightly to reduce flush frequency (this trades a small amount of increased latency for lower CPU). • Change one parameter at a time and measure CPU and end-to-end latency to ensure call quality remains acceptable. 4) Reduce incoming packet rate / per-call load • Ensure the SIP bridge is sending only the necessary streams to the Server: ◦ Disable simulcast/multiple spatial layers for SIP tracks (SIP endpoints don’t need simulcast). This reduces the number of parallel streams the Server must process. ◦ Apply bitrate caps for SIP sessions when acceptable. Lowering encode bitrates reduces packets per second and CPU usage. • If LiveKit Server is doing any transcoding or additional processing for SIP-originated tracks, avoid or offload that work if possible. 5) CPU isolation and cgroups as a diagnostic/workaround • Pin LiveKit Server and LiveKit SIP to separate CPU cores using cgroups / podman CPU pinning to reduce interference and confirm whether isolating CPUs mitigates the user-visible impact. • If isolating cores reduces the problem, that confirms contention/interrupt processing is a factor and separation or horizontal scaling is recommended. 6) Confirm and collect diagnostic traces (if you want deeper root cause) • Collect a lightweight perf/top snapshot while a call is active to confirm where Server time is spent (this confirms eBPF/bpf program hot paths vs syscall/packet-handling). • If you share the
interfaces
and
batch_io
fields from your livekit.yaml I can recommend exact values for batch_size and max_flush_interval for your workload. Notes about bpfrb threads and product-level tuning • There is no documented LiveKit Server option to change bpfrb thread count or polling frequency directly. The bpfrb activity you’re seeing is coming from low-level packet/BPF handling used by the underlying networking stack that LiveKit relies on. • Because these threads are part of low-level packet handling tied to how the process receives packets on host networking, the most effective fixes are: reduce packet rate, bind sockets more narrowly, or run Server and SIP on separate hosts/namespaces so the Server doesn’t receive all SIP-originating packets at NIC line-rate. Recommended immediate experiment (short checklist) 1. Temporarily run the SIP container without host networking (or move it to another VM). Measure CPU on the Server host during a call. If CPU drops, proceed to separate them in production. 2. If separation isn’t viable, increase
batch_io.batch_size
2× and
batch_io.max_flush_interval
modestly. Restart Server and measure CPU/latency. 3. Disable simulcast for SIP tracks and apply a modest bitrate cap on the SIP bridge. Test CPU impact. 4. If you want, paste your
interfaces
and
batch_io
config and I’ll suggest exact numeric values to try. If you want, I can also: • Suggest exact
batch_size
and
max_flush_interval
values based on your current settings. • Help craft a short test script/checklist to reproduce and verify CPU improvements. Tell me which of the above you want to try first (move to separate host, change
batch_io
, disable simulcast, or share your livekit.yaml) and I’ll give the specific commands/config edits. Sources: Going beyond a single-core | Introduction | LiveKit Documentation | livekit/livekit_composite | DeepWiki | Announcing LiveKit Cloud | Bringing Zoom&amp;#39;s end-to-end optimizations to WebRTC
c
port: 7880 log_level: info keys: myapikey: [REDACTED] redis: address: sentinel password: [REDACTED] sentinel_addresses: - [REDACTED_IP]:26379 - [REDACTED_IP]:26379 - [REDACTED_IP]:26379 - [REDACTED_IP]:26379 - [REDACTED_IP]:26379 - [REDACTED_IP]:26379 sentinel_master_name: mymaster sentinel_password: [REDACTED] sentinel_username: [REDACTED] username: [REDACTED] rtc: tcp_port: 7881 port_range_start: 50000 port_range_end: 60000 use_external_ip: false batch_io: batch_size: 512 max_flush_interval: 10ms turn: enabled: false room: max_participants: 200 signal_relay: retry_timeout: 30s min_retry_interval: 500ms max_retry_interval: 5s stream_buffer_size: 1000
t
Thanks — this is very helpful. You’ve given the exact
rtc
and
batch_io
settings I asked for, and confirmed the Server owns the bpfrb threads. I’ll be direct and actionable: what you can change in your LiveKit Server config, why it matters, and a short test plan so you can measure impact. Key facts up front • There is no LiveKit Server setting to directly change or limit bpfrb thread count or polling frequency. Those threads are low-level packet/BPF handling tied to how the process receives packets on host networking. • But you can reduce the packet-processing load the Server receives (which lowers bpfrb CPU usage) via the config values and deployment choices below. Recommended configuration changes (apply one at a time and measure) 1) Reduce packet churn with batch_io • Change: ◦ batch_io.batch_size: increase from 512 → 1024 (or 2048 if stable) ◦ batch_io.max_flush_interval: increase from 10ms → 20ms (then try 50ms if latency tolerance allows) • Why: Larger batches + slightly longer flush intervals reduce the number of packet processing events and syscalls the Server must handle per second. This trades a small increase in packet processing latency for lower CPU usage. • How to test: Restart the Server, make a SIP call, measure the bpfrb threads CPU %. If CPU falls but latency rises unacceptably, try smaller steps. 2) Narrow what the Server listens to (bind addresses / RTP port range) • Change (conceptual — apply based on your deployment): ◦ Ensure Server is bound only to the IP(s) it needs. With host networking the process currently sees all NIC traffic. If possible, configure Server to bind explicitly to the host IP rather than wildcard, or place the Server in a network namespace to reduce unsolicited traffic. ◦ Your current
port_range_start: 50000
,
port_range_end: 60000
is good. Ensure SIP bridge only uses ports within that range and that Server doesn’t have to scan or accept other host traffic. • Why: Reducing the set of packets reaching the Server’s sockets reduces eBPF and packet dispatch work. 3) Disable simulcast / reduce streams for SIP • On the SIP bridge or in how you bridge SIP to LiveKit, ensure SIP tracks are sent as single-quality (no simulcast) and cap bitrate if feasible. • Why: Simulcast multiplies streams/packets per participant. SIP endpoints don’t benefit from simulcast, so disabling it substantially reduces packet rate going through the Server. 4) Reduce per-call packet rate (bitrate caps) • Apply reasonable bitrate limits for SIP participants (e.g., 300–800 kbps for audio+video as appropriate). • Why: Lower PTS/payload sizes and fewer packets/second reduce packet-processing CPU. 5) Consider network namespace separation (quick mitigation) • Run Server in host networking but run SIP container without host networking (or vice versa) so only one process receives host NIC packets directly. • Why: This prevents the Server from observing all SIP-originated packets at NIC line-rate and is often the fastest mitigation. 6) CPU pinning as a diagnostic and temporary relief • Pin the Server and SIP containers to separate CPU cores using Podman/cgroup settings to reduce contention. • Why: If isolating CPUs reduces observable user impact, that confirms contention/interrupt processing as a contributor. It’s a useful short-term mitigation while you tune. Short prioritized test plan 1. Change only batch_io: ◦ Set batch_size to 1024 and max_flush_interval to 20ms. ◦ Restart Server, run a SIP call, measure bpfrb thread CPU and end-to-end latency. 1. If CPU still high, disable simulcast for SIP tracks and cap bitrate on the SIP side; retest. 2. If CPU still high or latency unacceptable, try moving SIP container off host networking (or to another host) to verify separation fixes it. 3. As a final step if separation improves things, consider permanently running Server and SIP on separate hosts or networks. What to measure • Per-thread CPU (top -H -p <server-pid>) while a call is active. • End-to-end call latency / packet loss metrics. • RTP packets/sec for the SIP->Server stream (if available). • Observe whether changes reduce bpfrb CPU and whether call quality remains acceptable. If you want, I can suggest exact numeric values beyond the initial batch_io changes (for example: batch_size=2048 / max_flush_interval=50ms) — but I recommend stepping up conservatively to avoid unexpected latency impact. If you’d like, run the first test (batch_size: 1024, max_flush_interval: 20ms) and paste the before/after top thread CPU lines. I’ll interpret the results and recommend next exact steps.
c
We also noticed CrowdStrike Falcon Sensor is running on the same host. Could the interaction between Falcon's eBPF hooks and LiveKit Server's bpfrb threads be contributing to the high CPU usage?
t
Thanks — good catch asking about CrowdStrike Falcon. You’ve already collected very useful evidence (bpfrb threads belong to LiveKit Server, host networking, NIC offloads on), and adding a host-level security agent that uses eBPF hooks is a plausible contributor. I’ll explain why that can matter, how to test it safely, and practical mitigations you can try. Why Falcon (or other kernel-level agents) can affect bpfrb CPU • CrowdStrike Falcon installs kernel-level instrumentation that can attach eBPF programs or other hooks into the packet path. When another process (LiveKit Server) is receiving/processing lots of packets on host networking, those kernel hooks run for each packet or socket event. • Additional eBPF work increases per-packet CPU work and can amplify CPU usage in the same threads/processes that handle packets (so bpfrb-related threads can show higher CPU when such hooks are active). • In short: yes — Falcon’s eBPF hooks are a plausible contributor to the increased CPU usage you’re seeing. Safe, targeted tests to confirm whether Falcon is contributing • Test 1 — Controlled disable (best if you have a maintenance window or test host): ◦ Temporarily disable or pause the Falcon sensor on the host and reproduce the SIP call test. ◦ Measure per-thread CPU on the LiveKit Server during the call (top -H -p <server-pid>) and compare to the prior baseline. ◦ If CPU drops significantly with Falcon disabled, the sensor is a major contributor. • Test 2 — Work with Falcon performance mode / tuning (if disable isn’t allowed): ◦ If disabling is not permitted, ask your security team to switch the agent to a lower-impact or performance mode for a short test window, then re-run the call test and compare CPU. • Test 3 — Use an isolated host for validation: ◦ Run LiveKit Server + SIP on an identical host without Falcon (or with Falcon disabled) to reproduce. This gives you a clean comparison if you can’t touch production sensor settings. Mitigations and options to try (work with your security team) • Exclusion rules for the agent: ◦ Create exclusions for LiveKit binaries, container PIDs, or the specific network ports/addresses LiveKit uses. Excluding the LiveKit Server process or its RTP ports from deep packet inspection can reduce eBPF invocation on hot packet paths. ◦ Exclude the container/runtime (podman/docker) network stack or the specific pod/container if your environment supports safe exclusions. • Agent configuration changes: ◦ Use a lower-sensitivity/performance profile for the agent while you tune LiveKit, if your security policy allows. ◦ If Falcon supports disabling particular eBPF features (network inspection vs. telemetry), disable only the network/packet inspection features for the LiveKit host during tests. • Network namespace / host-network changes (complementary mitigation): ◦ Moving LiveKit Server or SIP off host networking reduces the number of host-level packet events that hit the agent, lowering the intersection between Falcon hooks and LiveKit packet processing. • CPU isolation and scaling: ◦ Pin LiveKit Server to dedicated CPUs and ensure the sensor process isn’t pinned to the same cores, to reduce CPU interference (diagnostic / temporary mitigation). ◦ Run Server and SIP on separate hosts if feasible — avoids the whole interaction. • Monitoring and iterative testing: ◦ Measure before/after for each change (per-thread CPU, packet rates, and call quality metrics). This will show which mitigation gives the best tradeoff. What I recommend as the next steps (practical, minimal-risk) 1. If you have a test host: disable Falcon there and reproduce the SIP call. Compare per-thread CPU. This is the fastest way to confirm whether Falcon contributes materially. 2. If you cannot disable Falcon: ask your security team to temporarily exclude the LiveKit Server process/container (or the RTP port range) and rerun the test. 3. If exclusions aren’t allowed or don’t help: combine LiveKit tuning (increase batch_size / max_flush_interval) and run Server out of host network namespace or on a separate host — both reduce the packet events that trigger kernel hooks. 4. Share a before/after CPU snapshot (top -H -p <server-pid>) for the test you run and I’ll help interpret the impact and recommend the next configuration to converge on. If you want, I can provide the exact commands to: • capture the per-thread CPU snapshot, • pause/disable Falcon safely (generic guidance — you’ll need to follow your org’s Falcon procedures), or • configure an agent exclusion for the LiveKit container/ports (you’ll need to coordinate with your security team). Would you like the exact top/perf commands and an exclusion checklist to share with your security team?