https://www.boundaryml.com logo
I am not sure whether openai updated their api and...
# help
Hold on -- ignore ticket pls, did not add @description to this, eventhough was working perfectly fine for 200+ users before
v
๐Ÿ‘๐Ÿพ
m
@Vaibhav (BAML ๐Ÿ‘) Sorry for the ping, what is the sota way of handling 529 errors?
https://docs.anthropic.com/en/api/errors 529 - overloaded_error: Anthropicโ€™s API is temporarily overloaded.
Once again may have found issue, model that is struggling with this is CustomSonnet and doesn't have retry_policy set.
Copy code
// Learn more about clients at https://docs.boundaryml.com/docs/snippets/clients/overview

client<llm> CustomGPT4o {
  provider openai
  options {
    model "gpt-4o"
    api_key env.OPENAI_API_KEY
  }
}

client<llm> CustomGPT4oMini {
  provider openai
  retry_policy Exponential
  options {
    model "gpt-4o-mini"
    api_key env.OPENAI_API_KEY
  }
}

client<llm> CustomSonnet {
  provider anthropic
  options {
    model "claude-sonnet-4-20250514"
    api_key env.ANTHROPIC_API_KEY
  }
}


client<llm> CustomHaiku {
  provider anthropic
  retry_policy Constant
  options {
    model "claude-3-haiku-20240307"
    api_key env.ANTHROPIC_API_KEY
  }
}

// https://docs.boundaryml.com/docs/snippets/clients/round-robin
client<llm> CustomFast {
  provider round-robin
  options {
    // This will alternate between the two clients
    strategy [CustomGPT4oMini, CustomHaiku]
  }
}

// https://docs.boundaryml.com/docs/snippets/clients/fallback
client<llm> OpenaiFallback {
  provider fallback
  options {
    // This will try the clients in order until one succeeds
    strategy [CustomGPT4oMini, CustomGPT4oMini]
  }
}

// https://docs.boundaryml.com/docs/snippets/clients/retry
retry_policy Constant {
  max_retries 3
  // Strategy is optional
  strategy {
    type constant_delay
    delay_ms 200
  }
}

retry_policy Exponential {
  max_retries 2
  // Strategy is optional
  strategy {
    type exponential_backoff
    delay_ms 300
    multiplier 1.5
    max_delay_ms 10000
  }
}
I assume this is the problem. Fuck me sorry boys ahahaha
https://docs.boundaryml.com/ref/llm-client-strategies/retry-policy Maybe I am wrong - is 529 classed as a
network error
? or is this a fetch() response.ok vs throwing at network error situation
AI is saying 529 is a
server error
does that mean it is not included with the retry_policy?
v
oh we should be handling it
are you finding that we aren't falling back?
we apply a fallback on all non-200 status codes
sorry for the delay in messages here!
m
hm, I have added the retry policy to my claude calls but i'll add a discord webhook to debug and see if it's not doing the fallback
all working now though but with retry policy + try catch exponential backoff
hm @Vaibhav (BAML ๐Ÿ‘) do you have any idea why I'm getting Error: BamlValidationError: Failed to parse LLM response: Failed to coerce value: : Failed while parsing required fields: missing=4, unparsed=0 - : Missing required field: title_of_entire - : Missing required field: song_title - : Missing required field: method_type - : Missing required field: strategy_content now? Was not getting this for a very long time with BAML, as I thought the @description sorts it all out?
v
That usually happens when the LLM returns something that isnโ€™t of the shape you typed.
Can you share what raw string the LLM had?
m
Can I move this over to dms? @Vaibhav (BAML ๐Ÿ‘) Not wanting to show off my prompt publicly too much!