straygar
06/18/2025, 12:07 PMpromptfooconfig.yaml file with different scenarios etc. to catch regressions in CI and evaluate potential prompts & models. I use a custom Python provider & some custom python assertions, along with built in ones.
My current process of authoring new tests is:
- add new test
- comment out most of the file
- if something goes wrong, add print statements, rinse and repeat
Obviously this is not the best. I was never able to get --filter-pattern or --filter-metadata to work, and the entire promptfooconfig file is always run. I just found a way to potentially attach a python debugger, but it's a bit rough: https://github.com/promptfoo/promptfoo/commit/41cc82b2489efce4b167ebb25cc8cc6bcaf667b9ian
06/20/2025, 7:09 PM--filter-pattern should work, so if it doesn't, we should fix that!ian
06/20/2025, 7:10 PMguillermob_25239
06/21/2025, 8:30 AMguillermob_25239
06/21/2025, 8:30 AMstraygar
06/21/2025, 1:46 PMstraygar
06/21/2025, 1:52 PMtests[*].description. So there's 2 issues at play:
* for my tests that are defined as top-level tests, I have assigned id to them, not a description, so the filter has no effect.
* most of my other tests are buried in scenarios, so the filters have no effect there: https://github.com/promptfoo/promptfoo/issues/1061