https://promptfoo.dev/ logo
prompt file glob not working
# ❓|questions
j
using - file://path/to/file/*.txt gives error cannot find .txt treating it as a text prompt and then uses path/to/file/.txt as the prompt
i
Can you show me the config? This works ok for me with multiple prompt files, for example
Copy code
yaml
description: 'Glob test'

prompts:
  - file://./prompts/*.txt

providers:
  - openai:gpt-4-0125-preview

tests:
  - vars:
      topic: bananas
j
i thought i had it set up almost the same, i'll have to take naother look
as far as i can see i have it set up the same, does using a python file for providers make a difference?
i know the path is right because changing the * to a file name works
python file vs other provider didn’t help
i
one more thought. are you running
promptfoo eval
from within the same directory? thinking maybe there is some bug with relative path + globs
j
i’m running it from outside of the transcripts folder, same folder as the yaml i did try putting a text file in the source folder but it still didn’t find it
i
which version are you using?
promptfoo --version
j
0.49.0
updated to 0.50.0 and it still isn’t working for me
i
yeah i haven't been able to repro this issue :/ Just trying with the glob config I provided here https://discord.com/channels/1153767083381903389/1220819017376469093/1221099547078430730 and running
promptfoo eval
is it possible to share your config file?
or I guess you already did..
j
i wonder if because i am using powershell to run it that makes a difference?
i
that is strange
hmmm
that could be it
I don't have a windows machine to test, unfortunately
but may be able to write a unit test and run on github actions
j
i used a bash shell and it still did not work for me
i
I see
j
it probably is something wrong on my side, i’ll test some stuff out
i
here is my exact setup: promptfooconfig.yaml:
Copy code
description: 'Glob test'

prompts:
  - file://./prompts/*.txt

providers:
  - openai:gpt-4-0125-preview

tests:
  - vars:
      topic: bananas
prompts/1.txt:
Copy code
Write a tweet about {{topic}}
prompts/2.txt:
Copy code
Write an instagram post about {{topic}}
if it's not too much trouble, can you set this up and then run
promptfoo eval
in the config directory?
that will help us tell if it's an issue with glob logic or some system compatibility issue
j
will do within the next 60ish minutes! thanks for the help
i
Thanks so much for providing this debugging detail. Definitely windows specific with the glob behavior. Fix is in progress here: https://github.com/promptfoo/promptfoo/pull/612
@jimmy_soup This fix was shipped in 0.50.1. Please LMK if you continue to see problems. I wrote unit tests that ran on Windows via Github Actions, but I don't have a Windows machine myself 🙂
j
thanks! i’ll test it soon and let you know
worked great thanks!
is there a way to use glob for variables? i use tests: vars: variable: file1 vars: variable: file2 and so on
i
globs for variables aren't supported right now 😦 it's an interesting idea...
j
no problem! thanks for the help
i
@jimmy_soup support for this will be in the next release: https://github.com/promptfoo/promptfoo/pull/641
j
does that work for this situation^
where i have a bunch of text files that i would want as the same names variables over as separate?
i think it does just checking!
i
I think so, it basically expands them into separate tests for each file that matches the glob
j
perfect thanks
works great!!