https://promptfoo.dev/ logo
Use python script instead of YAML
# ❓|questions
z
I was wondering if I can use my
app.py
files for Streamlit apps, instead of writing separate YAML files just for the sake of testing/evals. Is there a work around for that?
i
One approach would be to use a script provider: https://promptfoo.dev/docs/providers/custom-script You could set a provider:
Copy code
yaml
providers:
  - exec: python script.py
with a python script that produces output directly
z
Getting the following error
Copy code
[FAIL] Error: Command failed: python promptfoo_evaluat.py "" "{\"id\":\"exec:python promptfoo_evaluat.py\",\"config\":{\"basePath\":\"\"}}" "{\"vars\":{\"query\":\"I'm looking to buy a home in Wales\"}}"/bin/sh: python: command not found
It looks like it's unable to read the file. @ian
the YAML and .py file are present in the same directory.
i
It looks like you don't have a binary named
python
See instructions on overriding the python binary in that doc. Your binary is probably named python3 or something
z
I was not in my virtual environment. Activated it, working now. Thanks a lot.
One more thing. I need to call llm multiple chains in the .py file and show the result of the last chain only. However, promptfoo keeps printing verbosity of all of the chains. How can I work around it?
i
hmm, is there any way to adjust the verbosity of the chains in the python code itself?
2 Views