Dieter
04/30/2024, 7:33 AMian
04/30/2024, 8:34 AMDieter
04/30/2024, 8:46 AMtests:
- vars:
expectation: "yes yes no no yes"
- assert:
- type: python
value: file://python.py
and I'n my python file I have the following, just for testing that it works:
from typing import Union, Dict, Any
def get_assert(output, context) -> Union[bool, float, Dict[str, Any]]:
expectation = context['vars']['expectation']
print("expectation: ", expectation)
return False
and thats a part of the output (I can't post more then 4000 chars):
│ │ [FAIL] Error: Python script execution failed: Error: Error running Python script: │
│ │ KeyError: 'expectation' │
│ │ Stack Trace: Error: KeyError: 'expectation' │
│ │ at PythonShell.parseError │
│ │ (X:\git\promptfoo\tut1\node_modules\python-sh... │
└──────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────┘
I have so many questions:
- Why do I get one Pass and one Fail? I have one test case with one assertion which, for debugging purpose, always Fail.
- Why do I get this KeyError?
- Where is the output of my print statement?ian
04/30/2024, 9:06 AM- instead of an indented assert, which breaks it into 2 separate items within the tests array. That is why you have a pass and fail and why your python can't access expectation
The python stdout is visible with LOG_LEVEL=debug or --verboseDieter
04/30/2024, 9:10 AMDieter
04/30/2024, 11:15 AM