Thanks — this is helpful detail. I’m responding to your question about possible causes of the growing RSS in the ProcessPoolExecutor worker (forkserver worker) and what I need to diagnose it. Short answer Any of the things you listed can cause growth: allocator arena fragmentation (glibc), large base64/JSON retained references, native/C-extension allocations from SDKs (gRPC/protobuf/LiveKit/OCI), or fork/forkserver behavior duplicating memory if heavy imports happen before fork. I need a few diagnostics to pinpoint which one it is. Please provide or run these diagnostics and share results 1) Environment snapshot • Python exact version (e.g. 3.12.0) and install method. • Linux distro + kernel, libc (glibc) version. • Container base image and runtime (containerd/docker), and whether jemalloc/tcmalloc is used. • LiveKit Python SDK version and OCI Vision client/protobuf/grpc versions. • Confirm you use

mp.get_context("forkserver")

on Linux. 2) Workload characteristics • Typical size (min/avg/max) of a full assembled JSON (including base64). • Arrival rate (msgs/sec) and concurrent in-flight messages. • Whether base64 images are full-size or thumbnails. 3) Quick runtime checks (run inside failing pod while RSS is rising) • Short time-series of RSS for the pool_worker (ps aux or top snapshots). • One pmap -x PID or cat /proc/PID/smaps snapshot (preferably at two times: early and when higher). • /proc/PID/status (VmRSS, VmSize) at same times. • Environment variable showing MALLOC_ARENA_MAX (or lack thereof). • If possible, tracemalloc snapshots before/after processing many messages (or run a small script to take snapshots and diff top allocations). 4) Quick experiments you can try now • Set MALLOC_ARENA_MAX=1 in the container and rerun to see if RSS growth reduces. • Run the worker with

mp.get_context("spawn")

(or use no forkserver) to see if behavior differs. • Force worker process to exit after N tasks (disable reuse) to see if memory is reclaimed on exit. • Add periodic

gc.collect()

and explicit

del

of large variables (e.g.,

del full_payload

,

del buffers

) right after use, then log RSS. What I’ll do once you share diagnostics • Correlate pmap/smaps growth with Python-level allocations (tracemalloc) to tell whether growth is Python objects or native memory/allocator fragmentation. • If growth is native (anonymous mappings / C heap), recommend MALLOC_ARENA_MAX, jemalloc, or short-lived worker processes; if Python objects hold it, point to exact code locations keeping references. • If forkserver is implicated, show which imports loaded before fork are large and recommend moving heavy imports into the child process or switching context. If you prefer, start by pasting: • Python + glibc + SDK versions, and • one pmap -x PID (or smaps) snapshot plus a short RSS time-series. I’ll analyze those and give targeted next steps. , with interactive elements

Thanks — I’m following up on your tracking worker memory issue (ProcessPoolExecutor worker RSS growing until OOM). I need a few targeted diagnostics to pinpoint the cause and give a concrete fix. Could you please provide these items or run these quick checks in the failing pod? 1) Environment snapshot • Python exact version (e.g. 3.12.x). • Linux distro + kernel version and glibc version. • Container base image and runtime (containerd/docker). • Whether you use jemalloc/tcmalloc or any custom allocator. • LiveKit Python SDK version and OCI Vision / protobuf / gRPC versions. • Confirm you’re using

mp.get_context("forkserver")

in production. 2) Workload characteristics • Typical assembled JSON size (min/avg/max, including base64). • Message arrival rate and typical concurrent in-flight messages. • Whether images are full-resolution frames or thumbnails. 3) Runtime snapshots (take at two times: early and after RSS has grown) • ps output for pool_worker (PID + RSS) over time (a short time series). •

pmap -x <PID>

or

/proc/<PID>/smaps

at both times. •

/proc/<PID>/status

(VmRSS, VmSize). • Current value of MALLOC_ARENA_MAX (env) in the container. 4) Quick experiments you can try now (each will help isolate cause) • Set

MALLOC_ARENA_MAX=1

and rerun to see if RSS growth reduces. • Run the worker with

mp.get_context("spawn")

instead of

forkserver

and compare. • Make the pool worker exit/restart after N tasks (disable reuse) to see if memory is reclaimed on exit. • Add

del full_payload

,

del buffers

, and

gc.collect()

immediately after processing each message and log RSS. If you prefer to start small, paste your Python + glibc + SDK versions and one

pmap -x <PID>

(or smaps) snapshot plus a short RSS time-series — I’ll analyze and recommend the next concrete steps. , with interactive elements