https://www.boundaryml.com logo
Powered by
🟡 Questions about Handling Errors in BAML TypeScr...
# questions
z
Hello there! I have a few questions about BAML TypeScript. 1. Is there a way to get the built prompt without calling the LLM? Currently I believe the only way is to trick BAML to throw an error to get 
error.prompt
. 2. Can 
message
be split into errors instead of combine multiple things? Especially a way to get the LLM Client JSON response and then a way to get BAML error message by itself. Code snippet:
Copy code
ts
            try {
                const stream = b.stream.Generate(..., { signal });

                for await (const chunk of stream)
                    yield chunk;

                const final = await stream.getFinalResponse();
                yield final;
            } catch (error) {
                if (signal.aborted) return console.debug("Generation aborted");

                if (error instanceof BamlValidationError) {
                    const until = error.message.indexOf(" - LLM client");
                    const message = error.message.slice(
                        0,
                        until === -1 ? 400 : until,
                    );
                    throw new Error(message);
                }

                throw error;
            }
Current error:
Copy code
json
{
  "name": "BamlValidationError",
  "message": "BamlValidationError: LLM Failure: Request failed with status code: 400 Bad Request, \n{\"error\":{\"message\":\"This model's maximum context length is 6114 tokens. However, your request has 10147 input tokens. Please reduce the length of the input messages. None\",\"type\":\"BadRequestError\",\"param\":null,\"code\":400}} (Unspecified error code: 400) - LLM client \"AiClient\" failed with status code: Unspecified error code: 400\nMessage: Request failed with status code: 400 Bad Request, \n<REPEATED JSON>",
  "raw_output": "",
  "prompt": "<PROMPT>",
  "detailed_message": "..."
}
v
you can use the modular API! what do you mean by split?
z
Oh the modular API sounds good, thanks! I mean by splitting the error: to extend 
BamlValidationError
with another property 
llm_client_error
for example, that has the response from the LLM Client.
v
oh shoot! i see how. raw_output is wrong! also what version are you on? I think we made a patch in a later version where we fixed this error! BAML validation error should actually only happen if we produce a parsing error. So the LMs respond and then you've gotten the wrong answer. There should be a different error called BAML client error that actually comes up when an LM client actually failed via HTTP and we give you the raw response.
z
oh ok! I'm on @boundaryml/baml 0.217.0
v
oh no! def a bug on our end! we will take care of this tmrw!!
and ship is in our next release
sorry about this regression!
z
ok thanks! no worries thanks a lot for the effort and links!
the documentation also says 
BamlError
but in TypeScript all BAML errors extend 
Error
instead. > When BAML raises an exception, it will be an instance of a subclass of BamlError. This allows you to catch all BAML-specific exceptions with a single except block.
it might be good to add 
BamlError
as well instead of 
type
Copy code
ts
export type BamlErrors = BamlClientHttpError | BamlValidationError | BamlClientFinishReasonError | BamlAbortError | BamlTimeoutError;
export declare function isBamlError(error: unknown): error is BamlErrors;
export declare function toBamlError(error: unknown): BamlErrors | Error;
v
this is kinda how we've been doing validation in TS
TS error inheritance is a bit odd sadly 🙁
oh shoot
i see what you mean
z
yeah, i would expect something like:
Copy code
ts
class BamlError extends Error

class BamlClientError extends BamlError
class BamlClientHttpError extends BamlClientError

class BamlValidationError extends BamlError
what do you think?
v
I agree 🙂
we're currently in the middle of doing some internal working that is employing the code underlying engine.
I'll share this stack with the team and make sure we take a stab at error handling correctly as a part of this
sadly swaping to the model you suggested would be a breaking change, but i think its wroth it
and better to break today than tmrw
z
thanks a lot for the effort! oh really? wondering why would it be a breaking change? is it because an error is missing (
BamlClientError
)? this is current 
errors.d.ts
Copy code
ts
export declare class BamlClientFinishReasonError extends Error {
}
export declare class BamlValidationError extends Error {

}
export declare class BamlClientHttpError extends Error {

}
export declare class BamlAbortError extends Error {
}
export declare class BamlTimeoutError extends BamlClientHttpError {
}
export type BamlErrors = BamlClientHttpError | BamlValidationError | BamlClientFinishReasonError | BamlAbortError | BamlTimeoutError;
export declare function isBamlError(error: unknown): error is BamlErrors;
export declare function toBamlError(error: unknown): BamlErrors | Error;
because just changing the class hierarchy should be fine, no? might be wrong people who did 
error instanceof BamlClientHttpError
nothing would break on their side
v
oh wait i see all were're doing is swapping Error -> BamlError, you're right!
z
for now this is what i see + adding 
BamlClientError
if you'd want to match the rust sdk (this is what i think the tests assert but as part of the message instead of .toBeInstanceof)
i mean if you decided that as long as you're fixing error handling, you'd want to match TypeScript error hierarchy with BAML core
v
yea exactly
z
havent looked at core error handling yet, but from what i understood:
Copy code
diff
+export declare class BamlError extends Error {
+}
+
-export declare class BamlClientFinishReasonError extends Error {
+export declare class BamlClientFinishReasonError extends BamlError {
}
-export declare class BamlValidationError extends Error {
+export declare class BamlValidationError extends BamlError {
}
+export declare class BamlClientError extends BamlError {
+}
-export declare class BamlClientHttpError extends Error {
+export declare class BamlClientHttpError extends BamlClientError {
}
-export declare class BamlAbortError extends Error {
+export declare class BamlAbortError extends BamlError { // or correct parent from CORE
}
export declare class BamlTimeoutError extends BamlClientHttpError {
}
-export type BamlErrors = BamlClientHttpError | BamlValidationError | BamlClientFinishReasonError | BamlAbortError | BamlTimeoutError;
-export declare function isBamlError(error: unknown): error is BamlErrors;
+export declare function isBamlError(error: unknown): error is BamlError;

// one issue here: this method accepts unknown but returns `Error` which is incorrect
// might be better to return "BamlError|null", had to look at source code to know it returns same error if it's NOT a valid BamlError
-export declare function toBamlError(error: unknown): BamlErrors | Error;
// either
+export declare function toBamlError<T>(error: T): BamlError | T; 
// or
+export declare function toBamlError(error: unknown): BamlError | null;
v
yea i think something like that will hopefully "just work" XD
someone should take this on tmrw i'm hoping!
z
ok thanks a lot! my note on `toBamlError`is incomplete, but there's definitely something to improve there! might take a stab at it and contribute if you dont mind!
if the circumstances are right for a first time contribution haha!
v
oh yea ofc
we're always happy to take contributions!
tests are pretty well described for this part:
Copy code
sh
git clone
cd baml
# assuming you already have mise installed
mise install
pnpm i
cd integ-tests/typescript
pnpm build:dev
pnpm test <specifier to run the test you want>
^That + ai coding agent should be enough XD
AI is magic 😂
there's also a properly written Contributing.md that is a bit more explanative than me at 10 pm 🙂
z
haha been really struggling to set up the repo
Copy code
mise i
mise WARN  Failed to resolve tool version list for npm:pnpm: [~/dev/BoundaryML/baml/mise.toml] npm:pnpm@10.14.0: timed out after 20.00s
mise WARN  Failed to resolve tool version list for npm:pnpm: [~/dev/BoundaryML/baml/mise.toml] npm:pnpm@10.14.0: timed out after 20.00s
was trying many different combinations of deleting and installing stuff, modifying mise.toml, mise use rust, but always get errors like > @baml/engine@0.1.0 typecheck /Users/zouabi/dev/BoundaryML/baml/engine > cargo check --workspace sh: cargo: command not found  ELIFECYCLE  Command failed. error: component download failed for rust-std-aarch64-apple-darwin: could not rename downloaded file from '/Users/zouabi/.rustup/downloads/ea619984fcb8e24b05dbd568d599b8e10d904435ab458dfba6469e03e0fd69aa.partial' to '/Users/zouabi/.rustup/downloads/ea619984fcb8e24b05dbd568d599 b8e10d904435ab458dfba6469e03e0fd69aa': No such file or directory (os error 2)  ELIFECYCLE  Command failed with exit code 1.  WARN  Local package.json exists, but node_modules missing, did you mean to install?  ELIFECYCLE  Command failed with exit code 1.  WARN  Local package.json exists, but node_modules missing, did you mean to install? command finished with error: command (/Users/zouabi/dev/BoundaryML/baml/engine/baml-schema-wasm/web) /Users/zouabi/dev/BoundaryML/baml/node_modules/.bin/pnpm run typecheck exited (1) └─ @gloo-ai/baml-schema-wasm-web#typecheck ── @gloo-ai/baml-schema-wasm-web#typecheck: command (/Users/zouabi/dev/BoundaryML/baml/engine/baml-schema-wasm/web) /Users/zouabi/dev/BoundaryML/baml/node_modules/.bin/pnpm run typecheck exited (1)
v
Do you have cargo and rust installed?
(Outside of mise)
z
no, had to 
mise use rust
to fix first error "cargo: command not found" then got the other error: 
error: component....
a
Hey @Zouabi I got a PR up for this [here](https://github.com/BoundaryML/baml/pull/2993), but I am not 100% convinced this is not a breaking change.
I think these were failing before the change so probably not indicator of breaking change
z
thanks! will review sorry i dont know how to use discord and im not sure why im not getting notifs
2 Views
PreviousNext
Powered by