While this isn’t new, I was just reminded of it: i...
# proj-agent
c
While this isn’t new, I was just reminded of it: it seems CC has a
Skill
tool for, I think, loading an AgentSkill. I… am not sure why it would need that?
s
I looked around when I was (still am) having problems with OH consistently trigger my skill. I saw it was pretty common to have a skill tool. At least Gemini cli and windsurf. My guess was by adding both skill descriptions plus a skill tool to context was enough of a hint to LLM that it should probably load skills on its own instead of relying on trigger words populating hints to do same.
👍 1
c
That makes sense
TBH I usually prompt for it, but not with trigger words, I just tell it to make sure to use the Cloudflare skill
s
Same but mostly for reliability. When tool is around, I also had luck with recursive skill loading (one skill might mention git which then triggers loading GitHub skill for example). No luck with recursion in OH.
c
cc @limited-student-10747 You had a PR to improve the LLM awareness of skills it should load IIRC, I wonder if this may explain it actually
l
Hmm, interesting. Yeah, I had an idle vibe-coded PR, but I wonder if I should do it more seriously
c
I think maybe we only tell the LLM “Available skills” title, and for each the location…
s
I double checked the system prompt being sent to confirm it had location because I was wondering if it was missing and related to my reliability problem. Its there all right but I think I see another small part of my issue. I see now that descriptions are truncated at 200 bytes and I stopped counting after the 20th truncated skill description (almost all from extensions repo). Combine that with the fact that most examples tend to put "Use this skill when..." at the end of description and thus the most likely part to get truncated. Having a skill tool will not help in this case.
aha 1
c
I didn’t know they’re truncated, that seems like a bug
Could it have been the logging?
s
I have a proxy I use to dump raw http requests so its not logging. Looks like its intentional? 200 limit in skill.py:to_prompt() in SDK.
c
@nutritious-balloon-65098 Find and investigate the Available Skills section in the prompt, and how we populate it with an Agentskill with progressive disclosure. Is the FULL description field inserted in the prompt sent to the LLM? Look in examples/ directory to find an example you could run to test a skill with a bit bigger description. Clone yourself extensions/ repo from the same org to find the three largest descriptions, and run the example with debug enabled to log exactly the prompt. Use a .pr/ directory in the root dir for all conversation logs/prompts. Confirm or not, whether all three got sent in full to the LLM. Note: when I ask you to summarize, don’t summarize, just repeat verbatim what you just told me. Really, I mean it, in this session don’t summarize even when I ask you, just repeat what you just said. (you could use the openai key with gpt-5-nano, though the response won’t matter, just the prompt)
n
I'm on it! can track my progress here.
c
Oh, thank you, sorry. I was just sending OH on a wild goose chase, then
😄 1
😂
n
Done. - Branch pushed:
openhands-workspace-2Uhx7DjJeGPO2Hx2uTqFs0
- PR opened: #2635 - PR URL: https://github.com/OpenHands/software-agent-sdk/pull/2635 The PR title is: -
chore: capture skill prompt truncation investigation
The committed work is on: -
d94962c9
c
Actually it found it:
Copy code
<description>Babysit a GitHub pull request by continuously polling CI checks/workflow runs, new review comments, and mergeability state until the PR is ready to merge (or merged/closed). Diagnose failures, retry l... [352 characters truncated. View /workspace/project/extensions/skills/babysit-pr/SKILL.md for complete information]</description>
l
I think skill descriptions are limited to a particular length by the standard
but if we aren’t reading in all skills then that seems problematic
I think maybe we could: • stop truncating them regardless, or give them, idk, 1k characters. after all, being more lenient is probably good? • check at submission to
extensions/
repo, and disallow descriptions larger than 200 characters • send an LLM call to a small LLM where possible, to summarize them? but it risks losing data just the same, and it introduces randomness in behavior
l
description is 1024 characters, are we truncating at less than that? https://agentskills.io/specification
n
Pushed again. - Restored
.pr/
by force-pushing the PR branch with lease - Branch:
openhands-workspace-2Uhx7DjJeGPO2Hx2uTqFs0
- New branch tip:
e99049a6
- PR remains: https://github.com/OpenHands/software-agent-sdk/pull/2635 The remote automated commit only deleted
.pr/
, so I replaced it with the branch state that includes: - the 1000-character truncation change - updated tests - regenerated
.pr/skill_prompt_probe/
artifacts showing all three largest descriptions are now sent in full
Yes — I stopped it. - Run
23827870512
(
PR Review by OpenHands
) is now
completed
with conclusion
cancelled
- The other related run
23827873769
is
completed
with conclusion
skipped
Direct links: - Cancelled run: https://github.com/OpenHands/software-agent-sdk/actions/runs/23827870512 - Skipped run: https://github.com/OpenHands/software-agent-sdk/actions/runs/23827873769 If you want, I can also check whether that workflow is likely to retrigger again on this PR and tell you how to avoid it.
Yes — I updated the PR description now. It now reflects the current state: - the actual code change is 200 → 1000 characters in
to_prompt()
- the three largest tested skills are now documented as being sent in full - the focused test file is called out -
AGENTS.md
is mentioned as updated I also added a dedicated Reviewer artifacts section so reviewers know where to look: -
.pr/skill_prompt_probe.py
-
.pr/skill_prompt_probe/report.md
-
.pr/skill_prompt_probe/description_comparison.json
-
.pr/skill_prompt_probe/available_skills_prompt.xml
-
.pr/skill_prompt_probe/system_prompt.txt
-
.pr/skill_prompt_probe/request_payload.json
-
.pr/skill_prompt_probe/completions/*.json
-
.pr/skill_prompt_probe/top_skills.json
I preserved the existing agent-server image block in the PR body too. PR: - https://github.com/OpenHands/software-agent-sdk/pull/2635 One note: I left the “Is the github CI passing?” checkbox unchecked, since I didn’t want to claim full green status while workflow state is still settling / one review workflow was intentionally cancelled.
c
Ohh! Yes, we were truncating at 200. @nutritious-balloon-65098 let's make it 1024 actually.
n
OpenHands encountered an error: **Failed to send message to agent server: HTTP 503 error: no available server ** [See the conversation](https://app.all-hands.dev/conversations/36837d51-56bb-4596-9fe4-eaee717c51e7) for more information.
c
@limited-student-10747 Tiny PR for 1024 default https://github.com/OpenHands/software-agent-sdk/pull/2635
n
Yes — I checked the inline review comments with
gh
, found that thread, verified the spec, and updated the PR. What I verified -
<https://agentskills.io/specification>
says the AgentSkills
description
field is: - “Max 1024 characters” - “Must be 1-1024 characters” What I changed - Added the rationale in code: -
openhands-sdk/openhands/sdk/context/skills/skill.py
- Added the rationale in repo memory: -
AGENTS.md
- Updated the PR description to say 1024 matches the AgentSkills spec Commit pushed -
429077be
—
docs(sdk): note AgentSkills 1024-char standard
Review thread - I replied on the inline thread explaining the fix and the spec verification - I resolved that thread I only addressed the “why 1024” thread you pointed at. If you want, I can also go through the remaining unresolved review thread(s) and handle them too.
s
Thanks for the PR! Looks like it should help me. I double checked gemini-cli's system prompt and it has both description and location as well. My new educated guess on why they have the "activate_skill" tool is because it simplifies things because their read file tool would normally block reading files in some of these locations outside the current workspace. At least I noticed something that looked like an unpredictable path for some of the skills gemini-cli ships with.
aha 1
l
Did a review!
gratitude thank you 1
c
This hypothesis makes a lot of sense to me too... Indeed OpenHands lets the agent run the tools where it wants to, blocks are by security analyzer, confirmation mode, mechanisms that are not tools. If a
file read
tool is restricted to the workspace, then, well, skills can be global, in
Path.home()
somewhere 🤔, or in caches Though, a concept of allowed directories would be enough...