https://promptfoo.dev/ logo
Lots of unexpected items in output
# ❓|questions
h
Hello, I'm trying to get my model to create a summary of various articles which is then graded by openAI. I'm noticing a lot of random extra content in the output (even though prompt specifies nothing else than summary) such as random references, hashtags, letter formats, further conversations, repetitions, blank lines etc. I'm having to use a lot of stop words. The model does not behave like this when tested in LM studio. Is there a provider option / config that I missed perhaps? I'd love some direction and help. Thanks.
u
Can you share your config with us? It's dependent on your model and prompt
h
Sorry about the late reply! We are using our own huggingface model spun up using llama.cpp.
Prompt is: Reference article: {{ article }} Produce a two-sentence summary under 200 characters, presenting the Reference article’s main theme. No extra context, no personal angle—just the article’s pure substance. Make it precise and faultless. Do not include anything else other than the summary content. Output should be in format of "Summary: {content}"
i
do you know if this model expects a certain prompt format, such as chatml?
h
I don't believe so
i
hmm, hard to say without more information. My best guess is it relates to the expected format of LLM inputs. Sometimes when a chat model is not prompted correctly, or when a completion model is allowed to run for a while, it'll start to spit jibberish
h
Do you think a better approach would be to use the API endpoint to test without additional prompting?
A confusing point is that the output quality varies quite a bit between LM Studio vs Production vs Promptfoo - we're trying to figure out why this is
i
try running the eval with
promptfoo eval --verbose
. You should be able to see the exact request that's being sent to LM Studio
actually not totally sure what debug output is included in llama.cpp. [This PR](https://github.com/promptfoo/promptfoo/pull/2569) adds more debug info
> Do you think a better approach would be to use the API endpoint to test without additional prompting? what exactly do you mean by this?
h
Ah sorry, I meant the ML Engineers use LM Studio during development and prompt engineering. The output from this vs. promptfoo (which we use llama.cpp to spin up models with) vs our production environment (which also uses llama) are all different.
We were looking into directly hitting our API endpoint for the model service instead of using a local instance of a llama server to test with promptfoo. I admittedly haven't looked into this too much but would we still need prompts in this case (given service already has prompts embedded within it)
Again, I am more of a test automation engineer / FE dev so apologies if I don't get the vernaculars correctly.