Hi all, I'm encountering persistent 401 Unauthorized errors when trying to use OpenRouter providers in my self evaluation and model comparison configs, despite having a working API key and successful direct API calls. Error Details: [ERROR] API error: 401 Unauthorized {"error":{"message":"No auth credentials found","code":401}} What's Working: OpenRouter API key works perfectly with direct curl calls Successfully configured and ran deterministic A/B testing for 3 LLMs using OpenRouter Environment variable OPENROUTER_API_KEY is properly set Current Configuration (that works for A/B testing): providers: - id: openrouter:anthropic/claude-3.5-sonnet config: temperature: 0.0 max_tokens: 2000 apiKey: ${OPENROUTER_API_KEY} What's Failing: Self-grading config with identical provider setup Model comparison config with identical provider setup All attempts result in 401 errors Attempted Fixes: Variable syntax variations: ${OPENROUTER_API_KEY}, "{{ env.OPENROUTER_API_KEY }}" Provider ID variations: different model names and versions Configuration approaches: Direct OpenRouter, OpenAI with custom base URL, Anthropic with custom base URL Environment handling: shell variables, --var flag, --env-file flag Removed llm-rubric assertions in attempt to fix authentication issues System Info: Promptfoo version: 0.118.17 OS: macOS Any insights on what might be causing this inconsistent behavior would be greatly appreciated!

Hey @haveles ! I think I found your issue. You're using

${OPENROUTER_API_KEY}

in your config, but Promptfoo uses Nunjucks templating, not shell syntax. That means it's literally sending the string

"${OPENROUTER_API_KEY}"

as your API key instead of the actual value. Just remove the

apiKey

line entirely from your config. OpenRouter automatically picks up the

OPENROUTER_API_KEY

environment variable, so you don't need to specify it:

yaml
providers:
  - id: openrouter:anthropic/claude-3.5-sonnet
    config:
      temperature: 0.0
      max_tokens: 2000
      # That's it - no apiKey needed!

If you really want to specify it explicitly, use Nunjucks syntax instead:

yaml
apiKey: "{{ env.OPENROUTER_API_KEY }}"

This is probably why your A/B testing config worked - it likely wasn't specifying the apiKey at all and just letting it auto-detect from the environment. Give that a try and let me know if it works!

@michaelmichaelmichael Thank you so much! That is the correct fix/answer. Both the removal of the explicit apiKey definition and the implementation of the Nunjucks syntax were functional options.