https://promptfoo.dev/ logo
python external file ERROR
# ❓|questions
a
i'm trying to use an external python file for assertions. this inline python works:
Copy code
yaml
        assert:
          - type: python
            value: |
              def main():
                  return True
              print(main())
the same thing put into a python file returns an error that the command line is too long.
Copy code
yaml
        assert:
          - type: python
            value: file://python-assertion.py
i know that external python files are run in the shell. it seems like the entire conversation history is being passed as a command line argument to the python script whenever
True
is printed, which is causing the command line input to exceed the system's limit. here's a somewhat formatted sample...
Copy code
cmd
[FAIL] Error: Command failed: python python-assertion.py "the model's sample response" "{\"vars\":{\"var1\":\"value\",\"_conversation\":[{\"prompt\":\"The prompt.\",\"input\":\"The prompt.\",\"output\":\"the model's sample response.\"},{\"prompt\":\"The prompt.\",\"input\":\"The prompt.\",\"output\":\"the model's sample response.\"}{\"prompt\":\"The prompt.\",\"input\":\"The prompt.\",\"output\":\"the model's sample response.\"}

(...prints multiple times cycling through each var...)

]}}"

The command line is too long.
why and how do i fix this?
i
hmm, do you have a particularly large prompt or set of vars? if not, one idea is to disable the
_conversation
builtin by setting the environment variable `PROMPTFOO_DISABLE_CONVERSATION_VAR=1`: for example:
Copy code
PROMPTFOO_DISABLE_CONVERSATION_VAR=1 promptfoo eval
a
i'll try that
is
1 promptfoo eval
the full value of the env var?
or is it just
1
and then
promptfoo eval
is the command
oh setting it to
1
worked!
thank you kindly
is there a way we can set this in the
promptfooconfig.yaml
file so that we don't have to individually set the env variables on our machines
is it just
Copy code
yaml
env:
  PROMPTFOO_DISABLE_CONVERSATION_VAR: 1
hmm that didn't work
i
unfortunately not right now, but you've uncovered a bug with the python assert that I will fix
so, the envar is a temp fix, and I will make sure a proper fix is released today
a
thank you!!
i
thanks for reporting the issue . I'll follow up when the fix is released
just published
0.27.1
, you should be good to go without the environment variable once you upgrade. Specifically, the
_conversation
variable should not be present in the context because you don't use it. That is what's causing the python arg list to be too long.
3 Views