https://promptfoo.dev/ logo
Default test structure advice
# ❓|questions
j
Hi, I have one test with multiple asserts that I want to run over two models and many variables. I want this test to first check things on the unparsed json, and then parse it so the remainder of my asserts are on value from the json. I would like to use llm rubrics on the parsed data. I struggle with using options: transform, as I have some default options also, and I don’t want the json to be parsed immediately. Any advice on this? I imagine I should split it into two tests - but is there a way of having two default tests so I don’t have to repeat the test many times in my yaml? Or read in from a file if not?
i
Hey @jimmy_soup this is a good question.. there currently isn't support for multiple default tests. It sounds like what you have is this:
Copy code
yaml
tests:
  - description: 'unparsed json tests'
    assert:
      - type: is-json
      # ...
  - description: 'parsed json tests'
    options:
      transform: JSON.parse(output)
    assert:
      - type: llm-rubric
        value: # ...
but you want to avoid repeating everything for each set of `var`s. Maybe it's not ideal, but you could use
defaultTest
and put the
transform
on each assertion.
Copy code
yaml
defaultTest:
  assert:
    # JSON assertions
    - type: is-json
    # ...
    # Parsed assertions
    - type: llm-rubric
      transform: JSON.parse(output)
      value: # ...
You'll have to repeat the
transform
for each parsed assertion, but at least you won't be repeating it in every single testcase. Hope I understood correctly, and that this helps.
j
yes that’s perfect, i was trying to move the transform and the options tag, didn’t realise i could just move the transform tag
i’ll try this right now
worked perfect thanks!
i
excellent
3 Views