https://promptfoo.dev/ logo
Evaluating Agent Skills (curl validation and refer...
# ❓|questions
a
Hi guys. I have some skill I'd like to evaluate with Promptfoo. The structure looks like this
Copy code
.
├── references
│   └── workflows
│       ├── sub-task-a.md
│       ├── sub-task-b.md
│       └── sub-task-c.md
└── SKILL.md
The skill has roughly 5 steps. First, it needs to decide which reference file to use based on the user prompt to execute a certain sub-task. The last step would call some
curl
command to validate the generated results is valid or not. My questions: 1. How should I configure promptfoo to verify that the right reference file is used for a given prompt? 2. How should I assert that
curl
command? Currently, I use custom provider with tracing enabled to verify the
curl
command and its payload. I'd like to learn from the community how would you guys do these evaluations. Thank you in advanced.
2 Views