https://promptfoo.dev/ logo
Metrics defined via named_scores in python file as...
# ❓|questions
d
I am hesitant to open a bug issue in the github repo simply because this issue (https://github.com/promptfoo/promptfoo/issues/1626) mentions there has a been a fix last year. However, I am trying the same return from the OP of the issue and not getting any metrics shown in the UI:
Copy code
python
def get_assert(output: dict[str, any], context) -> bool | float | GradingResult:
    return {
      'pass': True,
      'score': 0.11,
      'reason': 'Looks good to me',
      'named_scores': {
         'answer_similarity': 0.12,
         'answer_correctness': 0.13,
         'answer_relevancy': 0.14,
      }
    }
I was expecting to see the three
answer_*
named metrics appearing up top https://cdn.discordapp.com/attachments/1380231112252723332/1380231112629944341/Screenshot_2025-06-05_at_14.04.57.png?ex=68431fe4&is=6841ce64&hm=1e59574abdb29e00278b719905d74efcea8620d330947203cbe31d0c4fc9301b&
u
I can double check on this, I just pushed a huge update to this view and it might have introduced a regression.
d
Hey steve, I failed to mention that I was using 0.111.0. Let me move to latest and re-test
u
that'd be great, I'm not able to reproduce. I added
Copy code
defaultTest:
  assert:
    - type: equals
      value: Yarr
      metric: Tone
To the example in
examples/simple-test
and it showed up in the uI https://cdn.discordapp.com/attachments/1380231112252723332/1380241366256193677/image.png?ex=68432971&is=6841d7f1&hm=d67f3fe461eaf8a19e3d9db93eb13887f28e506ed3b32006ceeb9fe90387d0b7&
d
So, when I define metrics like that they also work... but try a test with an assertion from a python file that returns
named_scores
According to the docs they should be shown as metrics in the ui
I updated to latest and the problem remains
Can I open an issue? Is this really a bug?
u
Please do!
d
Hey @[Promptfoo] Steve I creted this issue: https://github.com/promptfoo/promptfoo/issues/4386 and also a repo to replicate the issue: https://github.com/donatoaz/promptfoo-named-scores-issue I used a bedrock provider, cause it's the only one I have access to, but feel free to customize (is there such thing as a Null provider? just to test?)