https://promptfoo.dev/ logo
Can promptfoo web view show each assertion result ...
# ❓|questions
a
Was playing around with the web view and I couldn't figure this part out
i
Do you mean breaking up test cases that have multiple assertions so each assertion result can be viewed at once?
a
yeah
right now if 1 test case has multiple assertions it seems like the assertion values will just be averaged out and combined into one column
i
Yeah there isn't a supported way to do this right now. I'm not sure I would add columns, but maybe break up the table cell to show multiple scores for the result?
a
Ok got it
i
Or super hacky way would be to create N test cases for 1 test case with N assertions. Won't be the prettiest UI but just a few lines of code
a
Yeah I don't think that would be pretty
esp. with a lot of test cases
i
Yeah
a
I think I will just evaluate the output result separately
Unrelated - is there a way to generate log from a custom python assertions
i
Their outputs should appear if you set LOG_LEVEL=debug
But there will be a lot of other stuff in that output too
a
Ok got it
I will try it
i
Open to ideas to improve that
a
is this similar to the
--verbose
CLI arg?
I guess it sounds the same but, not sure
i
Yeah they are the same under the hood
a
Error running Python script: Error: asyncio.exceptions.CancelledError
I am trying to troubleshoot this right now lol
i
😱 ok lmk if you're not getting the debug output you need
a
Yeah I am not getting it
thats why I am confused
i
Hmm
a
from the code seems like I can pass async functions as well right?
i
You should be able to
a
as in from the wrapper.py code
i
Yeah
I need to step out for a bit but will check in
a
async def get_assert(output, context) -> Union[bool, float, Dict[str, Any]]:
Ok thank you!
i
If you strip out everything that the assert function does and return a placeholder
Do you still get the error?
a
trying it out
ok it didn't
I will debug the function on my end
Ok it works, I found the bug
thank you!
5 Views