What I tried:
1. First attempt: Nemotron-3-Super (123B MoE, A12B active) via Ollama. Slow and tasks would not
complete.
2. Second attempt: switched the route to Nemotron-Nano-Omni Q8_0 via direct llama.cpp on :30000. Ran
noticeably faster but still failed to complete the same task.
What does "complete the task" mean here: I am not expecting a working production payments switch from one
run. I would consider it complete if the agent produces architecture documentation, partial scaffolding,
a security model, and a plan, the way Opus or GPT-5.5 would on the same prompt. What is actually
happening is that the agent loops without producing any meaningful artifact before either throwing
AgentStuckInLoopError or quietly stalling.
What is already known to be brittle on this stack:
- vLLM-FP8 crashed the host on 2026-04-30, suspected FlashInfer MoE FP4 backend on sm_121. NVIDIA's
Spark guide prescribes VLLM_USE_FLASHINFER_MOE_FP4=0, VLLM_NVFP4_GEMM_BACKEND=marlin,
FLASHINFER_FUSED_MOE_DISABLE_CUTLASS=1 to avoid that crash class.
- Ollama-served Nemotron-3-Super was reportedly ~44x slower than llama.cpp-direct on the smaller Nano
Omni, which is why nss-elite was cut over to direct llama.cpp on 2026-05-09.
- Anthropic API credit on the personal key was exhausted 2026-04-30, so cloud fallbacks for elite-tier
reasoning are unavailable. Everything stays local.
Questions for you:
(a) What is the right diagnostic to capture? I can grab the routed model name at failure from LiteLLM
logs, the OpenHands runtime log around the failure, and journalctl -u ollama -n 200. Is there a specific
OpenHands log line that confirms whether tool calls were parsed or not?
(b) As an A/B test to isolate parser issues from runtime/container issues, would you recommend forcing
the route to a known-good OpenAI-format tool-call model? Qwen3-Coder 30B via Ollama has been verified to
emit native OpenAI tool_calls in this stack. If the loop error disappears under Qwen3-Coder, that
confirms the root cause is parser/format, not OpenHands runtime.
(c) For Nemotron-3-Super specifically: is there a recommended llama.cpp launch profile or tool-call
parser plugin path for the Nemotron-3 family on OpenHands 0.44? Or is the safer near-term route to serve
Super via vLLM with the three NVIDIA env vars above plus --tool-call-parser qwen3_coder?
On the broader goal. The immediate aim is to get agent loops completing reliably on a local-only stack.
Longer aim is to approach hosted-frontier quality (Opus 4.7, GPT 5.5 class), knowing that it is an ongoing
pursuit and not a one-model swap, but a model for a specific use case.
Appreciate any guidance.