https://promptfoo.dev/ logo
custom guardrails testing
# ❓|questions
u
I have a custom guardrails to test, it is an API endpoint that receives a prompt and returns a json, so I wrote a custom provider to send the prompt and get a json like
{ "flagged": true/false, "category": "something" }
, the config looks like
Copy code
targets:
- id: 'file://custom_guard.py'
    config:
      endpoint: '{{env.ENDPOINT}}'
      key: '{{env.TOKEN}}'

redteam:
  plugins: ...
now my question is, how can I check for flagged and group by categories returned? thnaks
m
Do you have a dataset with labels for whether or not something should have been flagged and what the category should be? Here are our deterministic metrics - https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/ - you can use something like contains or parse the returned object https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/ If you provide more of your config I can suggest more
u
the prompts datasets are not labeled with the same categories that I get from the guardrail response, but I don't need to match them, what I have is a dataset with benign and harmful prompts used to score the guardrails. And another dataset with all hamful/malicious prompts that I need to use overtime to see the improvements, so the guardaril returns a verdict that is flagged: true or false and the category that the prompt triggered (it can be more than one for each prompt), my question is if I use that json format
{ "flagged": true/false, "category": "something" }
do I need to use transformResponse to parse it for the evaluation? there are configuration example that show a full config and not just pieces of them that is sometimes complicated to glue all together?
m
It might be easiest if we hop on a quick call to walk through your setup. Throw some time on my calendar here: https://cal.com/michael-dangelo/15min, and we can figure it out together.
u
sure, I will schedule one for next week, thanks
2 Views