https://www.boundaryml.com logo
Powered by
GPT-oss on vertex model garden
# help
b
Vertex is so frustrating 😂 has anyone gotten gpt-oss to work? If I try with 
openai-generic
i need to include custom headers to hardcode the 
gcloud auth print-access-token
response into the client, which obviously doens't work. But if I try to use the 
vertex
provider like this
Copy code
client<llm> VertexGptOss120B2 {
  provider "vertex-ai"
  options {
    model "openai/gpt-oss-120b-maas"
    base_url "https://aiplatform.googleapis.com/v1/projects/{GCP_PROJECT}/locations/global/endpoints/openapi/chat/completions"
  }
}
then baml includes the model in the URL, which is wrong, it needs to be included in the body, like so
Copy code
curl \
  -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" https://$\{ENDPOINT\}/v1/projects/$\{PROJECT_ID\}/locations/$\{REGION\}/endpoints/openapi/chat/completions \
  -d '{"model":"openai/gpt-oss-120b-maas", "stream":true, "messages":[{"role": "user", "content": "Summer travel plan to Paris"}]}'
can I include additional fields in the body?
a
Any fields you add to “options” will get added to the body
And if you add “headers” { key “value” } inside of options we also send those in the headers
b
right but the issue is i need the option to be 
model
but when you use vertex, that's a reserved word and it gets added to the url, because that's how it works for all their other deployments
like when i do
Copy code
client<llm> VertexGptOss120B2 {
  provider "vertex-ai"
  options {
    model "openai/gpt-oss-120b-maas"
    base_url "https://aiplatform.googleapis.com/v1/projects/{GCP_PROJECT}/locations/global/endpoints/openapi/chat/completions"
  }
}
baml generates a url like this 
https://$\{ENDPOINT\}/v1/projects/$\{PROJECT_ID\}/locations/$\{REGION\}/endpoints/openapi/chat/completions/openai/gpt-oss-120b-maas
because that's what the url setup for claude-vertex models are
v
Oh I have a workaround for this. I’ll dig it up when I’m home later at night!
I made it work for the anthropic models
But haven’t tried gpt oss yet
b
the anthropic on vertex models already works, since their url includes the model name, but the oss is structured differently [because vertex couldn't have a worse DX]
a
@Vaibhav (BAML 🐑) lets sync on the workaround so I can patch it
Ben this is still an issue right? basically -- 'how do i run gpt-oss on vertex'
v
q
Joined the discord just to find a solution for this issue. Has anyone been able to solve it?
a
we will work on this tomorrow 🙂
q
Hi folks, just a friendly ping—was there an update here that I missed? I scanned the recent changelog + commits but didn't see anything relevant.
v
hi @quiqueg we haven't managed to land this yet! Sorry for the delay here and thanks for following up. I'll do it for my friday project!
ok I have a workaround in the meanwhile! 🙂
Copy code
client<llm> VertexGptOss120B2 {
  provider "openai-generic"
  options {
    // deliberately empty
    model ""
    // actually fill these out
    base_url "https://{GCP_REGION}-aiplatform.googleapis.com/v1beta1/{ENDPOINT_RESOURCE_NAME}"
    api_key env.VERTEX_GPT_OSS_KEY
  }
}
then when you call it:
Copy code
python
import google.auth
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)

# At runtime, overwrite the env var each time
result = b.MyFunctionName(..., baml_options={"env":{"VERTEX_GPT_OSS_KEY": creds.token}})
(technically this should work also via vertex ai, but to unblock you 🙂
@benepstein @quiqueg
q
Thanks! Will try that out
v
ok well i've been debugging trying to get a hosted model on gcp for the lsat 3 hours and I can't seem to figure it out... google just says nah https://cdn.discordapp.com/attachments/1419078982808371362/1428862901028130826/google.mp4?ex=68f40bc6&is=68f2ba46&hm=17e57bc782adede56639d7436344040bda8b2c65106f1759b5aa9d770d894e9a&
if one of you would like this let me know and would love to just hop on a quick call so i can test this out and make sure it works, but will sadly have to pin in it for now 😢
q
@Vaibhav (BAML 🐑) did you try with the managed GPT-OSS service? https://console.cloud.google.com/vertex-ai/publishers/openai/model-garden/gpt-oss-120b-maas
I tried your workaround with this, but couldn't get it to work, I think because I still can't pass 
{"model":"openai/gpt-oss-120b-maas"
in the request body
2 Views
PreviousNext
Powered by