https://promptfoo.dev/ logo
How can I pass additional parameters into my custo...
# ❓|questions
d
I want to feed some additional criteria to my custom assertion. But I could not find out how to do that. Lets take the documentation example https://www.promptfoo.dev/docs/configuration/expected-outputs/python/#multiline-functions If I want to place the inline assertion into a function and I want to add the constant "Expected Output" as a parameter to the assertion, how would I do that?
i
Hey @Dieter, is the issue that you want some constant that is specific to the test case? One way to do this is to set a var on the test case. You should be able to access that var in the assertion
d
Hi @lurker, yes that would be the idea. I have the following lines in my config file:
Copy code
tests:
  - vars:
      expectation: "yes yes no no yes"
  - assert:
      - type: python
        value: file://python.py
and I'n my python file I have the following, just for testing that it works:
Copy code
from typing import Union, Dict, Any

def get_assert(output, context) -> Union[bool, float, Dict[str, Any]]:
    expectation = context['vars']['expectation']
    print("expectation: ", expectation)
    return False
and thats a part of the output (I can't post more then 4000 chars): │ │ [FAIL] Error: Python script execution failed: Error: Error running Python script: │ │ │ KeyError: 'expectation' │ │ │ Stack Trace: Error: KeyError: 'expectation' │ │ │ at PythonShell.parseError │ │ │ (X:\git\promptfoo\tut1\node_modules\python-sh... │ └──────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘ I have so many questions: - Why do I get one Pass and one Fail? I have one test case with one assertion which, for debugging purpose, always Fail. - Why do I get this KeyError? - Where is the output of my print statement?
i
The issue is with the yaml. You have an extra
-
instead of an indented
assert
, which breaks it into 2 separate items within the
tests
array. That is why you have a pass and fail and why your python can't access
expectation
The python stdout is visible with
LOG_LEVEL=debug
or
--verbose
d
Hi @User thanks for the info. I will look into it!
Yep, works now. Thanks!
4 Views