https://promptfoo.dev/ logo
Getting fetch failed when running red teaming
# ❓|questions
l
I'm trying to run a red team on one of the models, but I'm getting the following error: [HarmfulCompletionProvider] Error: Request failed after 2 retries: fetch failed Also getting these errors when trying to generate test cases: Failed to generate tests for [X]
i
hey @lucky_monkey_44174, can you try running the redteam with the
--verbose
option so we can get more debug output? That will help us troubleshoot
Copy code
promptfoo redteam run --verbose
l
Seems like I can't connect to the api of promptfoo because of networking issues: Request to https://api.promptfoo.app/task failed (attempt #1), retrying: TypeError: fetch failed (Cause: Error: Client network socket disconnected before secure TLS connection was established) Any ideas how I might be able to fix this? I'm currently on a company VPN but I need the company VPN in order to ping the models I want to evaluate.
Also, it seems like I can't properly ping my model when I'm using promptfoo. I can ping it with postman but promptfoo gives me some firewall issues. I've seen in another thread that it was possible to use
providers
to put a python script instead of a request. I created the .py script but it doesn't seem to be able to access it. What whould my config file look like if I want to ping a function inside python instead of passing by promptfoo to ping my API?
I currently have this in the config file: description: red team test targets: - id: 'file://call_apis.py' label: 'Test script 1' config: prompts: - '{{prompt}}' redteam: purpose: |- [.....] And call_apis.py contains the call_api function like in the doc
i
I'm guessing your company VPN or firewall is blocking this endpoint
https://api.promptfoo.app/task
cc @michaelmichaelmichael
l
Any potential workarounds for fetching the test sets? For testing the model, is my understanding that we can replace the HTTP request in the
target
with a python script correct? Referencing here: https://discord.com/channels/1153767083381903389/1309447535416315967 Thank you for your help! Super appreciated.
i
is there a proxy or other configuration that you need to use in order to access the internet programmatically? in other words, does
curl https://api.promptfoo.app/task
work or do you have to do something to authenticate?
your
call_apis.py
target looks ok to me. what exactly is not working? are you able to see more when you pass
--verbose
to the run?
l
For the promptfoo API, I've just tested it and it works when i'm ont on VPN, but I get this error on VPN: curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.promptfoo.app:443 Would running the test generation seperately from the eval be the only option?
EDIT: Seems to be working now for the call_apis! Thanks. Is it possible to call specific configs when running promptfoo redteam run ?
And for the first point: I can't run my models without the VPN, so I'm not sure how to proceed to do test generation before running the redteam eval.
i
glad you got it working, any idea what was going wrong on VPN? 🤔
To run a specific config, you can do
Copy code
promptfoo redteam run -c config.yaml
It's also possible to run test generation separately from the eval itself. For example:
Copy code
promptfoo redteam generate -c config.yaml
This will create a
redteam.yaml
. Then you can run
Copy code
promptfoo redteam eval
l
No clue! Still haven't got the VPN working, but using the python function to call the model is now working. I've created the test cases without the VPN and I'm now connected to the VPN to ping my model. Thanks for your help you've been super helpful.
3 Views