Hi all, Looking at skill invocation. Matt Pocock'...
# proj-agent-canvas
m
Hi all, Looking at skill invocation. Matt Pocock's skills make extensive use of
disable-model-invocation: true
in the skill front-matter. They're designed for use with Claude, and I think the behaviour in Claude is that the model cannot invoke that skill, but the user still can, perhaps both deterministically via trigger matching on the name, but by model-judgement invocation too. In the
software-agent-sdk
case, it will be excluded from
<available-skills>
but the
InvokeSkillExecutor
won't see it either, so it essentially becomes invocable only if a
trigger
matches on an KeywordTrigger/TaskTrigger/PathTrigger. I'm not sure that's desirable behaviour because those are rare fields for skill frontmatter from my experience. This is doubly confusing behaviour because the UI shows skills as available when
disable-model-invocation: true
... so it looks like they'd be easily invoked. If somebody could confirm this/the below approach, would appreciate. glm-5.2 take on the issue: > Currently the only deterministic activation is
match_trigger()
, which requires frontmatter triggers. To get Claude-Code parity you'd need: > 1. A user-dispatch path in
LocalConversation.send_message
(or `send_message`'s caller / a dedicated API). When a message begins with
/<name>
(whole-token) and
<name>
matches a loaded skill that is user-invocable, inject the skill content deterministically — bypassing
match_trigger
and bypassing the
disable_model_invocation
check in
InvokeSkillExecutor
. Mechanically it would look like `get_user_message_suffix()`: append the rendered content as
extended_content
on the
MessageEvent
, record the name in `activated_knowledge_skills`/`invoked_skills` for dedup. For TaskTrigger skills with
inputs:
, this is also where `$ARGUMENTS`/variable substitution and the missing-inputs prompt would slot in. > 2. Semantics: treat
disable-model-invocation
as "user-invocable-only", not "trigger-only". Concretely: > ◦ `_partition_skills()`stays as-is (still excluded from`available_skills`) — the point is the model can'tproactivelyinvoke it. > ◦ `InvokeSkillExecutor`gets a distinction: invocations originating from a user-dispatch message are allowed even for disabled skills (or simpler: user dispatch never goes through the tool at all, so the executor check only ever sees model-initiated calls and can stay strict). > ◦ The "Available skills" list in the error message can keep excluding them — model-initiated calls on disabled skills remain an error. > 3. Skill model addition (optional but useful for parity): a
user_invocable
flag (default
true
) so the complement behavior exists, plus a way to report user-invocability in the
/api/skills
payload so the frontend can act on it. > 4. Frontend (
OpenHands/
) wiring:
use-slash-command.ts
already builds a derived
/<name>
entry for every AgentSkills skill, including disabled ones — it just currently splices text and hopes for a trigger. Once the backend dispatch exists, that entry becomes real. You'd also want: > ◦ honor a future`user-invocable: false`(hide from the menu), > ◦ the "disable model invocation" pill could gain a tooltip ("user-invocable only — use /name"). > 5. Edge cases to nail down: precedence when a skill name collides with built-in slash commands (
/new
,
/model
) or plugin entry commands; whether the dispatch should match
/<name>
anywhere in the message or only as the leading token (Claude Code matches leading-token dispatch); and interaction with `stuck_detection`/condenser — the injected content should record
invoked_skills
the same way
invoke_skill
does so condensation and dedup behave consistently.
c
Sorry, I don't understand this: • the model cannot invoke that skill • but the user can, both via trigger matching and by model-judgment invocation Either the model can, or it cannot, right? What is “model judgment invocation“ if different from “the model can invoke“? Sorry I'm confused.
For clarity, the triggers are precisely what the user can type, to inject the skill
So if we want the user to e.g. inject with “/test-live” a skill Live Testing, then “/test-live” should be declared as a trigger in frontmatter, and that's it?
m
Currently, either model can AND the user can, or both cannot, yea, it's all-or-nothing. If you load a skill with
disable_model_invocation: true
in the frontmatter, then it gets removed from
<available_skills>
(
openhands-sdk/openhands/sdk/context/agent_context.py
) :
Copy code
if s.is_agentskills_format or s.trigger is not None:
                # Path rules force disable_model_invocation (Skill validator), so
                # they fall out here — injected only on file-touch, never listed.
                if not s.disable_model_invocation:
                    available_skills.append(s)
That's desirable, I think. That's the block that prevents the model from proactively invoking it "autonomously". Currently, we also puts a hard block on
invoke_skill
(
openhands-sdk/openhands/sdk/tool/builtins/invoke_skill.py
) :
Copy code
match = next((s for s in skills if s.name == name), None)
        if match is None:
            available = (
                ", ".join(
                    sorted(s.name for s in skills if not s.disable_model_invocation)
                )
                or "<none>"
            )
            return self._error(
                name, f"Unknown skill '{name}'. Available skills: {available}."
            )
        if match.disable_model_invocation:
            return self._error(
                name,
                (
                    f"Skill '{name}' cannot be invoked directly. "
                    "It can only be activated by trigger matching."
                ),
            )

        rendered = render_content_with_commands(match.content, working_dir=working_dir)
From my understanding, this means any skill with
disable_model_invocation: true
cannot be invoked at all. Either by the User in a prompt, or the model. That's not intention behind that flag though. The intention behind the flag is:
`disable-model-invocation: true`: Only you can invoke the skill. Use this for workflows with side effects or that you want to control timing, like
/commit
,
/deploy
, or
/send-slack-message
. You don't want Claude deciding to deploy because your code looks ready.
It is inconsistent though, because if the skill has been loaded to the project, it's entirely possible the model just searches for the skill file and then adds it to the context---but that's not how skill-loading should work and it fails if they're User skills, not project or repo skills.
c
I think
invoke_skill
is only about model, not user
I mean if the user sends
/commit
in a user message, then this is not the code path that is in use - or if it is in use, and prevents them, then it’s a bug The execution path for user prompt with
/commit
should be looking at “triggers”: if /commit is one of the triggers of a skill, it should be matched and loaded automatically in context. Have you seen it not work?
As you see here, the path of that source file is in
sdk/tools/builtins
, those are tools loaded in LLM’s context, I mean, well, tools. Those apply only to the agent, not the user
m
Yes, you're right, but the issue is that the user now has no invocation path at all.
send_message()
only runs
get_user_message_suffix()
->
match_trigge()
as you say. That requires frontmatter triggers, which skills often don't have. For trigger-less skills with a
disable-model-invocation: true
flag, there is no way they can ever be called. In reality, skills with the `_`disable-model-invocation: true`_` flag need to be model-invisible but user-invocable and they're not. They're just impossible to use.
Yes, it has happened to me. All of Matt Pocock's skills are unusuable for me (~20.3M downloads) so it must be happening to other people.
🤔 1
c
Ah, I see. But under what name(s) should the user be able to use them, if they have no triggers? Those were the … names
The exact skill name?
Could you please help me a bit, I understand now where you’re coming from, and I totally agree we want them to be user-usable. Could we please take an example, a skill from Matt Pocock and let’s look at it, and tell how would you like to type, to use it?
m
Generally his skill workflow starts with "grill me" so we can design a build plan: github.com/mattpocock/skills/blob/…/SKILL.md github.com/mattpocock/skills/…/to-spec Turn that build plan into issues/tickets: github.com/mattpocock/skills/…/to-tickets Implement it by resolving issues/tickets: github.com/mattpocock/skills/blob/…/SKILL.md As you can see, all of those skills have the
disable-model-invocation: true
flag. The User should be able to do
/grill-me-with-docs <design feature>
,
/to-spec
,
/to-tickets
,
/implement let's implement issue #54
. From my experience there are three agents trajectories: 1. I can't find the skills (they've been loaded as User skills), I'm stopping here 2. I can't find the skills (they've been loaded as User skills), but I know what /implement means and I have Git access already and I see
Issue #54
, so I can muddle through 3. I can't find the skills (because the disabled flag) but they were added as project skills and I've found the skill by grepping around and loaded it manually into the convo
h
I use his skills quite a bit but I've not had any issues like that. Just to be sure it wasn't something new I updated and did a quick
/handoff
for a conversation and the agent had no issue. How are you installing/using them in agent canvas and what LLM?
m
Interesting. I just use
npx
and install them to the skills folder. I am using GLM-5.2 and GLM-53/GLM-5.3-flash. They show up/register in the OpenHands as below:
Trajectory where agent finds it manually:
I just ended up deleting
disable-model-invocation: true
from his skills so I can use them, but re-added it for the above test.
As soon as I delete
disable-model-invocation: true
, the injection/loading works fine
h
I also added it with npx via skills.sh and tried with GLM 5.2 just now. What happens if you just run
/handoff
without asking the agent about it? ex
m
It does exactly the same thing. It just manually searches for it and then finds it and uses it. Can I ask if you have ran
/setup-matt-pocock-skills
as this often installs the skills into the project directory, and then they can be easily found (but the agent still has to search for them in my situation). OpenHands will only show them in the
~/.agents
directory though, because of its precedence.
With the flag set to false:
With the flag set to true:
Also, this is being run on a repo that has no knowledge/history that these are skills or guidance in
AGENTS.md
around them
h
No I don't do the full
/setup...
from his skill, just install them into my agents folder. Let me try running it in other projects and an empty one.
Interesting. If I run
/handoff
in the same project it works fine on a conversation where things have actually happened but on a fresh conversation in the same project the agent doesn't use it but says it is because there is nothing to handoff. Trying to invoke it on a fresh conversation is the only place I see it fail but at that point there is nothing for the skill to do.
m
What about a skill with
disable-model-invocation: true
that doesn't require previous repo work to do something sensible like
/grill-with-docs
. That should fail on a fresh convo/repo with no project skills added too