https://www.boundaryml.com logo
Powered by
`vertex-ai` provider with service account key auth...
# help
q
Hey team, heads up that 
vertex-ai
stopped working in v0.215.0 when using a service account key, which I'm using as a workaround for the lack of Workload Identity Federation support. I filed a bug here: https://github.com/BoundaryML/baml/issues/2891
a
@Quiqueg are you around?
q
Yes, I'm around. Yes, it does work with 0.214.0
a
hmmm interesting, let me see if i can repro
do you have GOOGLE_PROJECT_ID set in your env vars?
q
yes I do
Ah sorry, no I have 
GOOGLE_CLOUD_PROJECT
set instead
a
when you enable BAML_LOG=debug what does it say?
can you try adding project_id to the options?
happy to jump on a zoom call if that's easier
q
Sorry had to switch branches to get my test setup back
Same error with project_id in options, GOOGLE_CLOUD_PROJECT, and GOOGLE_PROJECT
a
can you add BAML_INTERNAL_LOG=debug
as an env var
q
Copy code
2025-12-30T23:45:26.725 [BAML WARN] Loading environment variables from .env file
[2025-12-30T23:45:26Z DEBUG baml_runtime::tracingv2::publisher::publisher] Skipping publisher because BOUNDARY_API_KEY is not set
Summary: 0/1 tests                                                                                                                                                                                                                                                      [2025-12-30T23:45:26Z INFO  baml_runtime] [Runtime] run_test_with_expr_events start function=ClassifyMessage test=Test1....
Summary: 0/1 tests 
⠁ Running ClassifyMessage::Test1                                                                                                                                                                                                                                        [2025-12-30T23:45:26Z INFO  baml_runtime] [Runtime] run_test_with_expr_events taking LLM path function=ClassifyMessage test=Test1
[2025-12-30T23:45:26Z INFO  baml_runtime] get_test_params: ClassifyMessage Test1
[2025-12-30T23:45:26Z DEBUG internal_llm_client::clients::vertex] Neither options.credentials nor options.credentials_content are set, falling back to env vars
[2025-12-30T23:45:26Z DEBUG internal_llm_client::clients::vertex] Using GOOGLE_APPLICATION_CREDENTIALS from env
[2025-12-30T23:45:26Z DEBUG internal_baml_jinja] Rendering template: 
    ...    
[2025-12-30T23:45:26Z DEBUG baml_runtime::internal::llm_client::primitive::vertex::std_auth] Attempting to auth using JsonFile strategy
[2025-12-30T23:45:26Z DEBUG gcp_auth::custom_service_account] found credentials project=Some("redacted") email="redacted-service-account@redacted.iam.gserviceaccount.com"
[2025-12-30T23:45:26Z DEBUG gcp_auth::custom_service_account] fetch_token; scopes=["https://www.googleapis.com/auth/cloud-platform"]
And then the test failure report follows
a
are you able to get the creds without 
baml test
? like in python or typescript
sorry is it able to run in python or typescript in general
odd, it seems to find the creds
q
I get the exact same error with TypeScript
And the same internal debug logs
a
ok i think i know what the difference is between your setup and mine
one sec
your GOOGLE_APPLICATION_CREDENTIALS -- is it just a path to a file?
q
Yes
a
mind pasting what the filepath looks like? I wonder if we arent resolving relative paths correctly perhaps
i just ran it again with GOOGLE_APPLICATION_CREDENTIALS set and it worked as well, which is strange
Copy code
(baml-tools) ➜  typescript git:(canary) ✗ npx baml-cli test -i TestVertex::TestProvider --from ../baml_src
2025-12-30T15:59:35.910 [BAML WARN] Loading environment variables from .env file
[2025-12-30T23:59:36Z DEBUG baml_runtime::tracingv2::publisher::publisher] Skipping publisher because BOUNDARY_API_KEY is not set
Summary: 0/1 tests 
⠁ Running TestVertex::TestProvider                                                                                                                       [2025-12-30T23:59:36Z INFO  baml_runtime] [Runtime] run_test_with_expr_events start function=TestVertex test=TestProvider....
[2025-12-30T23:59:36Z INFO  baml_runtime] [Runtime] run_test_with_expr_events taking LLM path function=TestVertex test=TestProvider
[2025-12-30T23:59:36Z INFO  baml_runtime] get_test_params: TestVertex TestProvider
[2025-12-30T23:59:36Z DEBUG internal_llm_client::clients::vertex] Neither options.credentials nor options.credentials_content are set, falling back to env vars
[2025-12-30T23:59:36Z DEBUG internal_llm_client::clients::vertex] Using GOOGLE_APPLICATION_CREDENTIALS from env
q
it's a relative path 
../../service-account-key.json
let me try moving into the current dir
a
can you try an absolute path, just to be sure
e.g. mine is 
export GOOGLE_APPLICATION_CREDENTIALS=/Users/aaronvillalpando/Downloads/gloo-ai-a37a32fa3313.json
q
same error with absolute path
FWIW I'm working in a Linux Docker container
a
i wonder if that is the issue
q
let me try pulling the project out into my host machine (macOS)
Works on the host machine
a
on 0.215.+ right
q
Yes
a
ok, thats probably why i cant seem to repro -- now not sure why it complains on the docker container or what changed between 0.214.0 and 0.215.0 yet, ill check a diff
q
Here are some more details on my devcontainer: 
FROM node:22-bookworm-slim
Copy code
$ arch
> aarch64
in production, where we also see this issue, we're also using 
FROM node:22-bookworm-slim
but the architecture is amd64 instead of arm64.
a
ok im trying to repro in a docker container
so you see the issue in both arm64 and amd64
q
yes
a
v
wait there are two different types of .json files google provides
can you share the json keys in your file?
q
Sure, let me revoke the key first
v
oh sorry
i meant just the keys in the json structure
like the field names
q
Oh yes, one second
a
im unable to auth on a linux docker container btw
using both GOOGLE_APPLICATION_CREDENTIALS_CONTENT or google_application_credentials (path to a json file)
q
Copy code
{
  "type": "service_account",
  "project_id": "REDACTED",
  "private_key_id": "REDACTED",
  "private_key": "REDACTED",
  "client_email": "REDACTED@REDACTED.iam.gserviceaccount.com",
  "client_id": "103210900557391259515",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/REDACTED%40REDACTED.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}
I'm able to reproduce with this Dockerfile:
Copy code
FROM node:22-bookworm

WORKDIR /app

USER root

COPY package.json package-lock.json ./

RUN npm ci

COPY . .

ENV BAML_INTERNAL_LOG=debug
ENV GOOGLE_PROJECT_ID=redacted
ENV GOOGLE_CLOUD_PROJECT=redacted

CMD ["npx", "tsx", "index.ts"]
and then these commands:
Copy code
docker build -t baml-test-app .
docker run --rm -v $(pwd)/service-account-key.json:/app/service-account-key.json -e GOOGLE_APPLICATION_CREDENTIALS=/app/service-account-key.json baml-test-app:latest
Note that I opted for 
node:22-bookworm
instead of 
node:22-bookworm-slim
because of missing CA in the slim image
and the 
index.ts
file is this:
Copy code
import { b } from "./baml_client/baml_client/index.js";

async function main() {
  const result = await b.ClassifyMessage({
    userName: "Alice",
    message: "This is a test message",
  });

  console.log(result);
}

main();
a
yeah i guet
Copy code
Failed to create request builder: Error {
             context: "Failed to build request",
             source: Error {
                 context: "Failed to parse credentials as JSON string: \"{\\n...\\n}\"",
                 source: Str(
                     "invalid private key in credentials",
                 ),
             },
         }
     BamlValidationError: BamlValidationError: LLM Failure: Failed to create request builder: Error {
         context: "Failed to build request",
         source: Error {
             context: "Failed to parse credentials as JSON string: \"{\\n...\\n}\"",
             source: Str(
                 "invalid private key in credentials",
             ),
         },
     } (Unspecified error code: 2) - Failed to create request builder: Error {
         context: "Failed to build request",
         source: Error {
             context: "Failed to parse credentials as JSON string: \"{\\n...\\n}\"",
             source: Str(
                 "invalid private key in credentials",
             ),
i think we mightve changed some parsing logic there but not sure, let me try to see which version works exactly
oh wait, this worked:
Copy code
docker run --rm --platform linux/amd64 -v /Users/aaronvillalpando/Projects/linuxgcp/gloo-ai-a37a32fa3313.json:/app/credentials.json:ro -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json -e BAML_INTERNAL_LOG=debug  timeout: 3m
      linuxgcp-amd64 2>&1 | head -80)
linux arm64
did not
can you see if running your app locally with the amd64 emulation works?
q
Yes, it works with 
--platform linux/amd64
a
we did change how we built the binary for 0.215.0, that is the likely cause
q
I'll have to revisit the exact errors we got in production (amd64), we were falling back to non-Vertex provider so it didn't show me the vertex error
but at least we've narrowed in on an arm64 error
a
specifically this change https://github.com/BoundaryML/baml/pull/2771/files for future ref. fun stuff ha
ok for now I would unfortunately recommend staying on 0.215+ with amd64 until we can fix the issue if youre able to easily swap
q
I'll have to downgrade to 0.214.0 because we're using a devcontainer for all of our local development, and emulation would be a perf killer. Thank you for taking a look!
a
yeah definitely understand
we do have a local repro which is good
ok we may have a fix, will see soon https://github.com/BoundaryML/baml/pull/2892
q
Thank you! Will test it out today
Confirmed, thank you for taking a look and fixing this! Much appreciated
a
thanks, definitely onee of the gnarliest bugs of the year, ty for the help
v
2 Views
PreviousNext
Powered by