I've been using Amazon Bedrock's batch inference service, is it possible to do this with BAML?

Yes! It's absolutely possible to use BAML with Amazon Bedrock's batch inference service. This is achieved by leveraging BAML's modular API https://docs.boundaryml.com/guide/baml-advanced/modular-api and building a custom wrapper around the parts you need. The core process involves: 1. Generate batch requests using BAML's modular API - Use

b.request.YourBAMLFunction()

to generate properly formatted HTTP requests with your BAML prompts 2. Create the input file - Format the requests into Bedrock's required JSONL format with

recordId

and

modelInput

fields 3. Upload to S3 - Upload the JSONL file to S3 as the input for the batch job 4. Submit the batch job - Tell Bedrock to process the batch job using the S3 input file 5. Get output file - Bedrock processes the job and writes results to S3 6. Parse results - Use

b.parse.YourBAMLFunction()

to parse the LLM responses with proper type casting Limitations I have found - Automatic retries - Bedrock batch jobs don't support automatic retries, so there's a chance of failures - Real-time error handling - Errors are only visible after the batch job completes My team has successfully used this approach with BAML to process millions of records through Amazon Bedrock batch inference (text, and text + image). We've observed: - 0.01% failure rate - Extremely low failure rate - Failures are likely model-related - Not due to BAML formatting or request generation - Reliable at scale - The modular API approach scales well for large batch processing I put together a quick sample script in python so you can get the idea of what it takes https://cdn.discordapp.com/attachments/1465441649072144582/1465456916494028934/bedrock_batch_baml.py?ex=6989a75b&is=698855db&hm=59e98c73048346497abaae224cd6343060ee2e63c2908730cb441c27a4430547&

Thanks!