Hi! I need to know if the parse messages method - or anything else - changes the way the prompt is sent to the model. I am struggling when using bedrock to recreate the results I see when accessing bedrock through promptfoo. I need to know what the difference is
jimmy_soup
05/02/2024, 7:53 PM
hmm i don’t think it’s the parse messages function, but is there anything else that could be different than directly accessing bedrock through boto3?
jimmy_soup
05/02/2024, 8:43 PM
it might just be haiku sending inconsistent results, but i was consistently getting one style back from promptfoo and a different from boto3 - any ideas?
i
ian
05/03/2024, 5:39 AM
I'm not aware of any explanation for the discrepancy, but I would run it with debug on so you can see the exact request that gets sent to bedrock:
Copy code
LOG_LEVEL=debug promptfoo eval
j
jimmy_soup
05/03/2024, 4:15 PM
running it with —verbose, we don’t see the model name in the api call, only the anthropic level
jimmy_soup
05/03/2024, 4:16 PM
the output seems consistent with the sonnet model NOT the haiku model - is there a bug where it’s not taking model name properly?