I've been struggling to make delegation work consi...
# proj-agent
w
I've been struggling to make delegation work consistently in agent-server. I'd say the success rate is < 50%, but simple test cases seem to work fine 🧡
this error seems like the most common one
Copy code
Agent error (delegate): Error validating args {"command": "delegate", "tasks": "{\"system\": \"Explore this machine's system setup thoroughly. Report back ALL findings in detail:\\n1. OS info: `cat /etc/os-release`, `uname -a`\\n2. Hardware: `lscpu`, `free -h`, `df -h`, `lsblk`\\n3. Disk state: what's on the disks, how much space used/free\\n4. Running processes: `ps aux` - what's running?\\n5. Docker: `docker ps`, `docker images` if available\\n6. Systemd services: `systemctl list-units --type=service --state=running`\\n7. Cron jobs: `crontab -l`, check /etc/cron*\\n8. Environment variables (non-secret): `env | grep -v TOKEN | grep -v KEY | grep -v SECRET | grep -v PASSWORD | grep -v AUTH`\\n9. Users: `cat /etc/passwd | grep -v nologin`, `whoami`, `id`\\n10. What's in /root, /home, and other important dirs\\n\\nReport ALL raw output so I can analyze it.\", \"networking\": \"Explore this machine's networking and DNS setup thoroughly. Report back ALL findings in detail:\\n1. `ip addr` or `ifconfig` - network interfaces\\n2. `cat /etc/resolv.conf` - DNS config\\n3. `cat /etc/hosts`\\n4. Open ports: `ss -tlnp` or `netstat -tlnp`\\n5. Firewall: `iptables -L -n` or `ufw status`\\n6. Look at nginx config: `cat /etc/nginx/nginx.conf`, `ls /etc/nginx/sites-enabled/`, `ls /etc/nginx/conf.d/`, and cat ALL nginx config files you find\\n7. Look at any Caddy, Apache, or other reverse proxy configs\\n8. Check for SSL/TLS certs: `ls /etc/letsencrypt/` or similar\\n9. Check domain/hostname: `hostname`, `hostname -f`\\n10. Any DNS management tools or scripts\\n11. How are subdomains configured? Look at wildcard certs, DNS records management, nginx server blocks\\n\\nReport ALL raw output so I can analyze it.\", \"apps\": \"Explore ~/git/openhands-remote thoroughly and ALL apps being served on this machine. Report back ALL findings in detail:\\n1. `ls -la ~/git/` - what repos are checked out\\n2. `ls -la ~/git/openhands-remote/` - full listing\\n3. Read the README, any docs, docker-compose files, Makefiles in ~/git/openhands-remote\\n4. Explore EVERY subdirectory in ~/git/openhands-remote - list files, read key config files, understand what each app does\\n5. Look at the 'meta' app specifically - read all its source files, understand how it navigates between apps\\n6. How are new subdomains/apps added? Look for scripts, templates, or documentation\\n7. Look at any docker-compose.yml, .env files, package.json files\\n8. Check for any deployment scripts or CI/CD config\\n9. Read all source code files in the meta app\\n10. Check ~/git/ for ANY other repos and briefly describe what they contain\\n\\nReport ALL raw output and file contents so I can analyze everything.\", \"openhands\": \"Explore ~/.openhands thoroughly and the OpenHands setup on this machine. Report back ALL findings in detail:\\n1. `ls -laR ~/.openhands/` - full recursive listing\\n2. Read ALL files in ~/.openhands/ - config files, skills, etc.\\n3. Read every skill file in ~/.openhands/skills/*/SKILL.md\\n4. Look at any OpenHands config: ~/.openhands/config.toml or similar\\n5. Check for OpenHands processes running\\n6. Look at any OpenHands docker containers\\n7. Check /workspace/ - what's here\\n8. How is OpenHands deployed/configured on this machine?\\n9. Read any AGENTS.md files you find\\n10. Check for any openhands-related systemd services or docker containers\\n\\nReport ALL raw output and file contents so I can analyze everything.\", \"auth\": \"Explore authentication and security setup on this machine. Report back ALL findings in detail:\\n1. SSH config: `cat ~/.ssh/config`, `ls -la ~/.ssh/`\\n2. Git config: `git config --global --list`\\n3. Check for any OAuth/auth services running\\n4. Look at nginx auth configs (basic auth, OAuth proxy, etc.)\\n5. Check for any .htpasswd files\\n6. Look at any auth-related environment variables (just names, not values)\\n7. Check for any auth middleware in the apps\\n8. Look at SSL/TLS cert management (certbot, acme, etc.)\\n9. Check for any API keys or token management (just the mechanism, not actual secrets)\\n10. Look for any identity/SSO services\\n\\nReport ALL raw output so I can analyze it.\"}\n<parameter name=\"security_risk\">LOW", "summary": "Delegate 5 parallel exploration tasks to sub-agents"} for tool 'delegate': 1 validation error for DelegateAction tasks Input should be a valid dictionary [type=dict_type, input_value='{"system": "Explore this...ame="security_risk">LOW', input_type=str] For further information visit <https://errors.pydantic.dev/2.12/v/dict_type>
w
Which model are you using here? Looks like it might just be filling out the delegate tool call incorrectly.
m
I believe the problem lies in how we are parsing the tool call. The
DelegateTool
API includes a dictionary argument, which is causing a JSON parsing conflict. I’ve already fixed a similar issue, check out this PR for reference.
πŸ‘ 1
w
This is Opus 4.6
I'm using the main branch as of ~24h ago, so your fix is in there Vasco. Do you think there's a separate problem?
m
Probably the fix I did was not covering all the cases. But not 100% sure. I will investigate.
πŸ‘ 1
@nutritious-balloon-65098 Please open an issue in
software-agent-sdk
regarding the error discussed in this Slack thread. You can assign it to me.
n
I'm on it! Vasco can track my progress here.
## Summary I've successfully created a GitHub issue in the
software-agent-sdk
repository documenting the delegate tool parsing error discussed in the Slack thread. ### Completed Actions - βœ… Created issue [#2670](https://github.com/OpenHands/software-agent-sdk/issues/2670) in OpenHands/software-agent-sdk - βœ… Assigned the issue to you (VascoSch92) - βœ… Documented the error with the full error message from the Slack thread - βœ… Referenced the related PR #2217 that attempted to fix a similar issue - βœ… Included context about the model (Opus 4.6) and success rate (<50%) - βœ… Added observations and possible investigation areas ### Issue Details Title: Delegate tool: tasks dict still parsed as string intermittently (Opus 4.6) The issue captures that the
tasks
dictionary parameter is being passed as a JSON string instead of a dictionary, causing Pydantic validation to fail. It also notes the interesting detail from the error message that XML-like content (
<parameter name="security_risk">LOW
) appears concatenated with the JSON, which may be interfering with parsing. No code changes were made - this was purely an issue creation task.
m
I looked into the issue, and it seems that @wide-doctor-37830 was right: the model is using the tool incorrectly. You can find more details in the issue here. It would be possible to fix this, but the question is: should we? We don't want to pollute the code with these kinds of workarounds, especially since it seems the error is originating from the model itself. Please let me know what you think.
w
this was Opus, so I imagine other models are only going to be worse
maybe we need to align the tool call to be closer to how Claude Code does it?
I wonder if Anthropic models are fine-tuned for that
w
They're fine-tuned on tool-calling (returning structured JSON), probably on the signatures of their default tools as well. Interestingly the error that started this thread shows Opus switching from JSON to XML, which is also probably related to Anthropic fine-tuning. They love XML for structuring prompts.
It's so close to being well-formatted, I bet we could get rid of 99% of these cases by surfacing the parse errors to the LLM and asking for a redo.
πŸ‘ 1
m
It’s interesting to observe the performance gap between models fine-tuned for standard tool calling versus those fine-tuned for subagent orchestration. For instance, Opus is significantly more adept at invoking subagents correctly, knowing exactly when to call them and how to prompt them, compared to Sonnet.
maybe we need to align the tool call to be closer to how Claude Code does it?
The
TaskToolSet
is already aligned πŸ™‚