https://promptfoo.dev/ logo
Bedrock model access error when running asserts
# ❓|questions
j
When running asserts only on model output I am getting an access denied error, you don’t have access to the model with the specified model ID. When running fully it works. I am also getting this error running the node library
by using the id keyword in the node library it works, still not sure why it doesn’t in the cli
i
Have you overridden the provider for assertion? You might have to do that if you want Bedrock to do the grading
Feel free to share an example config if I misunderstood the question
j
i was overriding in the node package and for the cli for the node library instead of provider : bedrock:… i had to do provider: { id: bedrock:…} for the cli, using the assertions parameter and —grading didn’t work
i
ah yeah, the cli doesn't have a way to override the provider specifically for assertions as a CLI arg. You'd have to configure that in the YAML
j
got it
i actually have another quick question
using EvaluateOptions.cache i can’t seem to disable the cache in the node library
in bedrock.ts, lines 184 and 185 would need to be moved to 188
is there any specific reason they couldn’t be moved there?
tested it locally and it works, can’t make a PR right now
i
If I understand your problem correctly, I would propose to fix this another way for all providers, which is to make sure that
getCache
doesn't have any side effects (e.g. creating an empty cache folder) if the cache is disabled. Is that the problem you are trying to solve?
j
i think so!