https://promptfoo.dev/ logo
Bug with Different prompts per model when provider...
# ❓|questions
h
Hello, following the doc here https://promptfoo.dev/docs/configuration/parameters#different-prompts-per-model I am trying to execute test cases for some prompts/custom scripts as providers. The issue i get is that, when two providers have the same id (in my case the same custom script), even if i assign them a different label, they do not appear in the promptfoo view report. For instance in the image attached, I would normally expect 5 x Test cases (I have 11 test cases) = 55 test cases, but I only get 44 test cases and here is the report (attached). The label does not appear, and it seems lake the latest provider id of the promptfooconfig.yaml overrides the first ones because the got the same id (even with different labels) My work-around was to duplicate the python:custom.py scripts and modifiy their name, but that's not what i want because i will end up with a lot of duplicated files that only differ by some hardcoded argument (here the post_prompt argument) https://cdn.discordapp.com/attachments/1222597110378266765/1222597111317794866/image.png?ex=6616cb4a&is=6604564a&hm=9958b2ba070be19687e40071a2ae3097ec9e30389da0e4c70b1d3b4e519380de& https://cdn.discordapp.com/attachments/1222597110378266765/1222597111615455452/image.png?ex=6616cb4a&is=6604564a&hm=216f7ed311b01e4f89786608b0dded4351c4ec3369911293cde09fe93af1d403&
As a workaround, I duplicated the python scripts executed in order for them to show separately in the assertions... I dont know if it is a bug or am i doing something wrong https://cdn.discordapp.com/attachments/1222597110378266765/1223233845109985330/image.png?ex=66191c4b&is=6606a74b&hm=30240e67096b8085daf439b605bdae1abed61c1f3109556eded0e505247a41b7&
i
In the latest release I fixed an issue that was causing labels for python providers to not appear correctly. But I think I haven't reproed the core issue that you're reporting yet, which is that the number of testcase permutations is incorrect
I'm trying to test with a simplified setup like this
Copy code
yaml
prompts:
  prompt1.txt: prompt1
  prompt2.txt: prompt2
  prompt3.txt: prompt3

providers:
  - id: 'python:provider.py'
    label: 'First provider'
    prompts:
      - prompt1
      - prompt2

  - id: 'python:provider2.py'
    label: 'Second provider'
    prompts:
      - prompt1
      - prompt3

tests:
  - vars:
      topic: bananas
and am seeing 4 columns as expected
h
It seems the latest release has fixed the issue because now the "label" help differentiate between the columns ! The problem i was seeing if i take your case was only using "python:provider.py" but giving it a different argument in config
Copy code
prompts:
  prompt1.txt: prompt1
  prompt2.txt: prompt2
  prompt3.txt: prompt3

providers:
  - id: 'python:provider.py'
    label: 'First provider'
    prompts:
      - prompt1
      - prompt2
    config:
      arg: some_value

  - id: 'python:provider.py'
    label: 'Second provider'
    prompts:
      - prompt1
      - prompt3
    config:
      arg: another_value


tests:
  - vars:
      topic: bananas
I love this project ! thanks a lot to you for making it a easier to score prompts and provider on our custom use cases without relying on the skewed public leaderboards !!