https://promptfoo.dev/ logo
Filtering tests/scenarios
# ❓|questions
s
Heya. Tldr: how do you write, debug and run parts of your tests & scenarios? I have a pretty big
promptfooconfig.yaml
file with different scenarios etc. to catch regressions in CI and evaluate potential prompts & models. I use a custom Python provider & some custom python assertions, along with built in ones. My current process of authoring new tests is: - add new test - comment out most of the file - if something goes wrong, add print statements, rinse and repeat Obviously this is not the best. I was never able to get
--filter-pattern
or
--filter-metadata
to work, and the entire
promptfooconfig
file is always run. I just found a way to potentially attach a python debugger, but it's a bit rough: https://github.com/promptfoo/promptfoo/commit/41cc82b2489efce4b167ebb25cc8cc6bcaf667b9
i
hmm, well for starters
--filter-pattern
should work, so if it doesn't, we should fix that!
the other approach I see people taking is to break out their test cases into separate yamls and then include them in the main file. That makes it a bit easier to test different parts
g
We have many files, and the file name is used to filter them. — config evals/smoke*.yaml will run the smoke test evals —config evals/full*.yaml will run the big evaluations —config evals/edge*.yaml will run the edge cases
This is helpful also to run the smoke evaluations in CI/CD also
s
@ian That's what I tried at some point, but I'm not sure what the pattern matches against (description? assert rules? id? tests? scenarios?), some better docs/examples would be great there! And if the pattern doesn't match anything or matches everything, it'd be great if it errored out, instead of running the whole test suite (which is an expensive way to debug your filter pattern 😄
I guess the docs suggest that it's regex against
tests[*].description
. So there's 2 issues at play: * for my tests that are defined as top-level
tests
, I have assigned
id
to them, not a
description
, so the filter has no effect. * most of my other tests are buried in
scenarios
, so the filters have no effect there: https://github.com/promptfoo/promptfoo/issues/1061