Unspecified error code: 2 Failed to parse into a response accepted by baml_runtime::internal::llm_client::primitive::vertex::types::VertexResponse: {"candidates":[{"content":{"role":"model","parts":[{"inlineData":{"mimeType":"image/png","data":"iVBORw0KGgoAA ... I="}}]},"finishReason":"STOP"}],"usageMetadata":{"promptTokenCount":608,"candidatesTokenCount":1290,"totalTokenCount":1898,"trafficType":"ON_DEMAND","promptTokensDetails":[{"modality":"IMAGE","tokenCount":516},{"modality":"TEXT","tokenCount":92}],"candidatesTokensDetails":[{"modality":"IMAGE","tokenCount":1290}]},"modelVersion":"gemini-2.5-flash-image","createTime":"2025-11-09T13:10:45.558212Z","responseId":"1ZIQaYSJIuWzmPUPwLHPyQU"}


 Caused by:
    missing field `text`

Request options: {"generationConfig":{"maxOutputTokens":32768,"temperature":1,"responseModalities":["TEXT","IMAGE"]}}

What you’re seeing now confirms the Vertex provider in the current BAML runtime can’t handle inlineData at all. Even though we force the model to add a text summary, the response still contains a second part with just inlineData. The runtime tries to deserialize each part into a struct that requires a text field, hits the image part, and throws missing field 'text' before your function ever sees the payload.

Bypass BAML’s LLM client and call Vertex directly (a small proxy you can hit from BAML).

client<llm> AdsGeminiImageEdit {
  provider vertex-ai
//   retry_policy Exponential
  options {
    model "gemini-2.5-flash-image"
    location "europe-west1"
    credentials env.GOOGLE_PROJECT_SECRET
    generationConfig {
      maxOutputTokens 32768
      temperature 1
      responseModalities ["TEXT", "IMAGE"]
    }
    media_url_handler {
      image "send_url_add_mime_type"
    }
  }
}

class GeminiInlineData {
  mimeType string?
  data string?
}

class GeminiPart {
  inlineData GeminiInlineData?
  text string?
}

class GeminiContent {
  role string?
  parts GeminiPart[]
}

class GeminiCandidate {
  content GeminiContent?
  finishReason string?
  index int?
}

class GeminiTokensDetail {
  modality string?
  tokenCount int?
}

class GeminiUsageMetadata {
  promptTokenCount int?
  candidatesTokenCount int?
  totalTokenCount int?
  trafficType string?
  promptTokensDetails GeminiTokensDetail[]?
  candidatesTokensDetails GeminiTokensDetail[]?
}

class GeminiImageEditResponse {
  candidates GeminiCandidate[]?
  usageMetadata GeminiUsageMetadata?
  modelVersion string?
  createTime string?
  responseId string?
}

function EditAdImage(
  prompt: string,
  referenceImages: image[],
) -> GeminiImageEditResponse {
  client AdsGeminiImageEdit