This message was deleted.
# ask-ai
s
This message was deleted.
t
Thanks for reaching out, Jack. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this log is very helpful. I’m responding to the worker failure during InferenceExecutor initialization (the
DuplexClosed
/
IncompleteReadError
followed by “process exited with non-zero exit code -9”). Short answer: the worker subprocess is being killed (exit code -9 / SIGKILL). The DuplexClosed/IncompletedRead is a symptom of that sudden termination. The most likely cause is an out-of-memory (OOM) kill during model loading or plugin initialization. What in the logs points to that • The worker hits an incomplete read on the Unix-domain IPC channel, then raises
DuplexClosed
— this means the other end closed unexpectedly while reading a message. • The process exit shows code -9, which is SIGKILL and usually comes from the OS (commonly the OOM killer). • You preload many plugins (ONNX-based models like Silero can use hundreds of MBs), which increases memory use during startup. Actions to diagnose (please run these and share results) 1. Check host/container OOM logs: ◦ On the host: run
dmesg | grep -i -E "kill|oom"
or
journalctl -k | grep -i oom
. Look for messages like “Out of memory: Kill process [pid] ([python])”. ◦ In Kubernetes:
kubectl describe pod <pod>
and check for OOMKilled in the container status. ◦ In Docker:
docker inspect <container>
and check
State.OOMKilled
. Share any relevant lines if you want me to interpret them. 1. Confirm container memory limits: ◦ Docker: were you running the agent with a
-m
/
--memory
limit? ◦ Kubernetes: do you have
resources.limits.memory
set for the pod? Tell me the memory limit and the host total RAM. 1. Get agent memory usage before the crash (if possible): ◦ If it restarts too fast, try running the container interactively and monitor with
top
/
htop
while the agent starts, or run
docker stats <container>
to watch peak usage. Quick fixes you can try now • Increase memory available to the agent container (e.g., 2Gi or more) and restart. This is the most likely fix. • Temporarily reduce preloaded plugins/models: remove plugins you don’t need so fewer models load on start (that lowers memory at startup). • If running on a small VM (burstable T3/T4g), try a non-burstable instance type (more stable CPU/memory) or a larger instance. Notes on non-blocking warnings • The ONNX cpuid_info warning (“Unknown CPU vendor”) is generally benign and not the crash cause — it just indicates ONNX runtime is running in a virtualized environment. If you’d like, next steps I can help with • Interpret any OOM log lines you paste here. • Recommend a target memory size based on which plugins you need (tell me which ones you actually use). • Provide steps to disable specific plugins for the agent if you want to test a lighter startup. If you can share the results of the diagnostic checks (dmesg / pod status / container inspect) or the list of plugins you need active, I’ll guide you through the best fix. Sources: Troubleshooting Latency and Timeout Errors with Turn Detection on AWS