https://promptfoo.dev/ logo
Custom Provider - Python?
# ❓|questions
d
Hello! I was wondering if utilizing a custom provider must use javascript in order to make the API call to the custom provider? My project is mostly using python and if I could make that API call with Python instead that would be of great help, if possible?
i
the best way to do this is to use a script provider: https://promptfoo.dev/docs/providers/custom-script let me know if you have any questions!
d
Thank you! 😄
c
Good evening, I followed the documentation to have Python as a provider but I need to make use of Python's own libraries to be able to generate my output, but when I run the npx eval it says that that module does not exist. Do I have to specify the libraries I use or how could I solve this? Thanks a lot @ian @douar
i
Just to make sure I understand, you are importing your own python module from the module with call_api?
c
Yes, or you could also include the logic within that script. But when I'm going to do the eval with npx it says that the module doesn't exist
i
you may need to set the
PYTHONPATH
environment variable to include the path to your module
c
Look, that's how I did it, but I think I'm doing something wrong 😦
And my yaml is
i
hmm can you show me
prueba2.py
?
c
Clear
i
A couple ideas: 1. You set
PYTHONPATH
to a binary path. It should instead contain the path to wherever your python modules are. 2. Do you have multiple python binaries on your system? To direct promptfoo to a specific python binary, use the
PROMPTFOO_PYTHON
envar. For example,
PROMPTFOO_PYTHON='/usr/local/bin/python'
3. You've definitely installed openai via
pip install openai
? Is there a virtualenv that you need to activate?
c
If openai is installed, when I run the Python script only with "Python prueba2.py" it runs well. I'm going to try the solutions you gave me, thank you very much
Run this commands
i
PYTHONPATH
must be a directory that contains your modules.
PROMPTFOO_PYTHON
must be the path to the python binary
c
Perfect, thank you 😄
w
Idk why but I can't get custom params inside call_api function. The only config param available is basePath, check screenshots
I'm wordering whether options are even passed to the python script, not very familiar with promptfoo codebase, but it looks like only basePath is passed. Isn't it a bug?
because examples only uses .get() to eventually get the key value, but looks like no one really used it.
i
@wuodarr thanks for the catch. This PR fixes the issue with passing extra config vars
w
great
2 Views