https://promptfoo.dev/ logo
all my assertions are passing despite giving wrong...
# ❓|questions
a
tests: - vars: topic: bananas - vars: topic: avocado toast assert: - type: icontains value: 'mango' - type: javascript value: 1 / (output.length + 1) - vars: topic: new york city assert: - type: llm-rubric value: ensure that the output is funny provider: custom-api-provider https://cdn.discordapp.com/attachments/1265701055195381781/1265701055354769629/image.png?ex=66a27777&is=66a125f7&hm=cccd81608a37ea97e62657fa621c91856006cbde0a91ac4cfad1e7a6a3237b92&
i
Yaml indentation error. "assert" should not be under "vars"
a
thank you! that fixed the error
i
you entered
custom-api-provider
as the provider. Looks like you do not actually have a custom provider? You can just remove that line
a
I have a custom provider that I have provided in the "providers " config not sure how to do that in separate assertions
worked, thanks!
i
a
is it not possible to use the label that I have specified as part of providers? Ideally, I should provide it once in the top level and use it at the assertions level right now I have to give the full absolute path in all assertions
for example - in this example https://github.com/promptfoo/promptfoo/blob/main/examples/gpt-3.5-temperature-comparison/promptfooconfig.yaml is there not a way to use lowtempand hightemp providers for different assertions
i
ah nope, that's a good idea but right now they are independent of each other
a
ok It would help ,, in my case most of the metrics are model graded so giving the entire absolute path for each assertion is an overkill.
Also I have written a custom script that loops through the folder containing the config files and runs them and write the result to stdout. However, I see all colors are lost from the output /table. Is there a workarounf for this?
10 Views