asterwork
11/14/2023, 6:15 PMyaml
assert:
- type: python
value: |
def main():
return True
print(main())
the same thing put into a python file returns an error that the command line is too long.
yaml
assert:
- type: python
value: file://python-assertion.py
i know that external python files are run in the shell. it seems like the entire conversation history is being passed as a command line argument to the python script whenever True is printed, which is causing the command line input to exceed the system's limit. here's a somewhat formatted sample...
cmd
[FAIL] Error: Command failed: python python-assertion.py "the model's sample response" "{\"vars\":{\"var1\":\"value\",\"_conversation\":[{\"prompt\":\"The prompt.\",\"input\":\"The prompt.\",\"output\":\"the model's sample response.\"},{\"prompt\":\"The prompt.\",\"input\":\"The prompt.\",\"output\":\"the model's sample response.\"}{\"prompt\":\"The prompt.\",\"input\":\"The prompt.\",\"output\":\"the model's sample response.\"}
(...prints multiple times cycling through each var...)
]}}"
The command line is too long.
why and how do i fix this?ian
11/14/2023, 6:26 PM_conversation builtin by setting the environment variable `PROMPTFOO_DISABLE_CONVERSATION_VAR=1`:
for example:
PROMPTFOO_DISABLE_CONVERSATION_VAR=1 promptfoo evalasterwork
11/14/2023, 6:29 PMasterwork
11/14/2023, 6:30 PM1 promptfoo eval the full value of the env var?asterwork
11/14/2023, 6:30 PM1 and then promptfoo eval is the commandasterwork
11/14/2023, 6:32 PM1 worked!asterwork
11/14/2023, 6:33 PMasterwork
11/14/2023, 7:06 PMpromptfooconfig.yaml file so that we don't have to individually set the env variables on our machinesasterwork
11/14/2023, 7:08 PMyaml
env:
PROMPTFOO_DISABLE_CONVERSATION_VAR: 1asterwork
11/14/2023, 7:10 PMian
11/14/2023, 7:12 PMian
11/14/2023, 7:13 PMasterwork
11/14/2023, 7:13 PMian
11/14/2023, 10:54 PM0.27.1, you should be good to go without the environment variable once you upgrade.
Specifically, the _conversation variable should not be present in the context because you don't use it. That is what's causing the python arg list to be too long.