https://promptfoo.dev/ logo
Bedrock messages api ParseMessages
# ❓|questions
j
Hi! I need to know if the parse messages method - or anything else - changes the way the prompt is sent to the model. I am struggling when using bedrock to recreate the results I see when accessing bedrock through promptfoo. I need to know what the difference is
hmm i don’t think it’s the parse messages function, but is there anything else that could be different than directly accessing bedrock through boto3?
it might just be haiku sending inconsistent results, but i was consistently getting one style back from promptfoo and a different from boto3 - any ideas?
i
I'm not aware of any explanation for the discrepancy, but I would run it with debug on so you can see the exact request that gets sent to bedrock:
Copy code
LOG_LEVEL=debug promptfoo eval
j
running it with —verbose, we don’t see the model name in the api call, only the anthropic level
the output seems consistent with the sonnet model NOT the haiku model - is there a bug where it’s not taking model name properly?
i
Are you able to share your provider config? When I'm running [this bedrock/anthropic example](https://github.com/promptfoo/promptfoo/blob/main/examples/amazon-bedrock/promptfooconfig.claude.yaml#L4-L19) with 0 temperature, I do get different results for haiku vs sonnet