I'm trying to combine this amazon documentation: https://docs.aws.amazon.com/nova/latest/userguide/modalities-video-examples.html With the Baml video documentation: https://docs.boundaryml.com/ref/baml/types#video / https://docs.boundaryml.com/ref/baml_client/video Specifically looking at model compatibility I see something about S3 in Nova

AWS Bedrock    ✓    Fully multimodal. Accepts video as Base64 bytes in request or S3 URI. JSON must include format (e.g. mp4) and source.

But this does not seem to work:

class S3Location {
  uri string
  bucketOwner string?
}


class BedrockVideoSource {
  s3Location S3Location
}

class BedrockVideo {
  format string
  source BedrockVideoSource
}

class VideoExtractionResult {
  description string
  extracted_text string[]
  audio_transcript string?
}

function ExtractVideoS3(video: BedrockVideo) -> VideoExtractionResult {
  client NovaLite

  prompt #"
    {{ _.role("user") }}

    Generate a description of this video and extract any raw text included in the video.
    If the text is not in English, auto-detect the language and translate it to English.

    If there is audio, run speech-to-text on the audio and include the text verbatim.

    If you don't have access to the video, return an empty string for the description and extracted_text.

    {{ video }}

    {{ ctx.output_format }}
  "#
}

test CreativeVideoTest {
  functions [ExtractVideoS3]

  args {
    video {
      format "mp4"
      source {
        s3Location {
          uri "s3://[bucket]/[key]"
          bucketOwner "accountnum"
        }
      }
    }
  }
}

I could not figure out how to create the correct Baml function and calling syntax here though it seems alluded to that it's supported. Any help would be great!

Thanks for flagging this. We will add this in today! Sadly there is no good way for you to inject in new top level content types for a provider. But this is fast to merge so will fix asap 🙂

sweet! Happy to test it with several thousand videos a day once ready 🙂

Just want to check something: do you need cross-account buckets?

@baml-sam apologies I missed this. I would say in general we'll be using bedrock in the same account (definitely top priority), but I could certainly imagine cross account use cases down the road

https://github.com/BoundaryML/baml/pull/2681 will go out sometime this week!

epic thanks @baml-sam (and team) will check it out!