I'm attempting to use prompt caching for aws-bedrock https://docs.boundaryml.com/guide/baml-advanced/prompt-caching-message-role-metadata but even the simplest example isn't working. So with

provider aws-bedrock

then I get no cache control ephemeral in the curl, if I change to anthropic (which won't work because we use Bedrock at our company) then I get the cache control in the curl but 401 expected beause I'm not hitting Anthropic directly. Is there a limitation with Bedrock + Anthropic + prompt caching? Is there a way I could manually inject that into the calls (not desireable but will if that's the only way) https://cdn.discordapp.com/attachments/1253172394345107466/1451247385652105428/image.png?ex=69841a33&is=6982c8b3&hm=3cc36d00254ce2f0bad681723168ab07de651175f27047c349183b0a7a59a64b& https://cdn.discordapp.com/attachments/1253172394345107466/1451247386046234637/image.png?ex=69841a33&is=6982c8b3&hm=777920a03a517a5d92d45daa8b3191640a8efdcac3e064ccad74b03a584b9c75& https://cdn.discordapp.com/attachments/1253172394345107466/1451247386549555241/image.png?ex=69841a33&is=6982c8b3&hm=e2938a2a98f5adc5f7c443abfb556101b8c964390de522dc22f8699f8dfea33b&

bedrock is sadly a bit special 🙁 (its the only special client in that way)

ok fantastic, yeah this is critical for me, and it appears that the converse api is in play here which is different than the InvokeModel even: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html. so it wants the format:

"messages": [
   {
        "role": "user",
        "content": [
            {
                "image": {
                    "bytes": "asfb14tscve..."
                }
            },
            {
                "text": "What's in this image?"
            },
            {
                "cachePoint": {
                    "type": "default"
                }
            }
      ]
  }
]

Are you tracking that? I think the converse is what applies here and not the InvokeModel API which would be the typical antropic cache header

Have there been any updates on this? This would be really helpful since while using bedrock, currently the only way to leverage caching is to use the Modular API. However when we use the Modular API, we lose the ability to track tokens with the Collector. It's been a huge pain point when it comes to using BAML with bedrock.

hey folks! sorry for the extreme delay here in responding! Currently we don't yet have a way to inject arbitrary json elements in ⁨`content`⁩ fields. we do have an update that will allow us to add more arbitrary content into there! we're working on getting that out there. this is a problem for a few things: images / media with custom properties, tool calls ids, etc. So we are attempting to design this and make it much easier to support.

Hey @samwisegardener , curious if you'd had any work around here for caching bedrock calls

sadly no update on natively supporting this yet. the modular api should work tho!

ok thanks! And we can use the collector still?

All thanks to @Kai (BAML) !!

any idea when the next BAML release is expected to come out?

it should be today! cc: @aaronv (BAML)

It's out now!

Awesome thanks @aaronv (BAML)