Not clear how to set up using LangChain w/ BedrockConverse, and Claude as the model promptfoo #❓

Not clear how to set up using LangChain w/ Bedrock...

ptrin

12/11/2025, 2:50 PM

I'm just looking through documentation trying to get acquainted with how promptfoo can be used. Our setup is that we use LangChain to manage LLM inference, using BedrockConverse and we're experimenting with different models (which is part of why I'm looking into promptfoo, for model comparison). Our main use case is structured output, can anyone offer pointers on how to evaluate structured output responses from multiple models on the same provider?

michaelmichaelmichael

12/11/2025, 4:35 PM

Hey! There are a couple ways to approach this depending on whether you actually need LangChain in the loop or just want to compare Bedrock models. ## Option 1: Use promptfoo's native Bedrock provider directly This is the simpler path if you just want to compare models. We have a

bedrock:converse

provider that works with all Bedrock models out of the box:

Copy code

yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  # Claude 4.5 Sonnet
  - id: bedrock:converse:us.anthropic.claude-sonnet-4-5-20250929-v1:0
  # Claude 3.5 Sonnet
  - id: bedrock:converse:us.anthropic.claude-3-5-sonnet-20241022-v2:0
  # Amazon Nova Pro
  - id: bedrock:converse:amazon.nova-pro-v1:0

prompts:
  - |
    Extract the following information as JSON:
    {{input}}

    Return only valid JSON with fields: name, date, amount

tests:
  - vars:
      input: "Invoice from Acme Corp dated Jan 15 2025 for $500"
    assert:
      - type: is-json
        value:
          type: object
          required: [name, date, amount]
          properties:
            name: { type: string }
            date: { type: string }
            amount: { type: number }

The

is-json

assertion checks both that the output is valid JSON and that it matches your schema.

michaelmichaelmichael

12/11/2025, 4:35 PM

## Option 2: Wrap your LangChain setup in a Python provider If you need LangChain-specific stuff like

.with_structured_output()

, you can create a Python script provider:

Copy code

python
# langchain_bedrock.py
from langchain_aws import ChatBedrockConverse
from pydantic import BaseModel

class ExtractedData(BaseModel):
    name: str
    date: str
    amount: float

def call_api(prompt, options, context):
    model_id = options.get("config", {}).get("model_id", "us.anthropic.claude-sonnet-4-5-20250929-v1:0")

    llm = ChatBedrockConverse(model=model_id, region_name="us-east-1")
    structured_llm = llm.with_structured_output(ExtractedData)

    result = structured_llm.invoke(prompt)
    return {"output": result.model_dump_json()}

Then reference it in your config, passing different model IDs:

Copy code

yaml
providers:
  - id: file://langchain_bedrock.py
    config:
      model_id: us.anthropic.claude-sonnet-4-5-20250929-v1:0
  - id: file://langchain_bedrock.py
    config:
      model_id: us.anthropic.claude-3-5-sonnet-20241022-v2:0
  - id: file://langchain_bedrock.py
    config:
      model_id: amazon.nova-pro-v1:0

## Structured output assertions For validating structured output, you've got a few options: -

is-json

- validates JSON + optional schema validation -

javascript

- custom validation logic -

python

- if you want to use Pydantic or similar Example with schema:

Copy code

yaml
assert:
  - type: is-json
    value:
      required: [field1, field2]
      properties:
        field1: { type: string }
        field2: { type: number, minimum: 0 }

michaelmichaelmichael

12/11/2025, 4:35 PM

## Links - Bedrock docs: https://www.promptfoo.dev/docs/providers/aws-bedrock/ - Python provider docs: https://www.promptfoo.dev/docs/providers/python/ - JSON assertions: https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-json - Bedrock example: https://github.com/promptfoo/promptfoo/tree/main/examples/amazon-bedrock - LangChain Python example: https://github.com/promptfoo/promptfoo/tree/main/examples/langchain-python Let me know if you run into any issues getting it set up!

ptrin

12/17/2025, 2:32 PM

I think I'll need to look further into this custom provider thing, because it would be most convenient for me to call the LLM inference using an HTTP API (to pass different configurations for prompt/model/option variants)

Previous Next