https://www.boundaryml.com logo
BAML struggles to parse perfectly valid JSON
# help
l
These are my model definitions
Copy code
// ===== Shared Enums =====

enum IntegrationType {
  Automatic
  OneClick
  API
  Webhook
  ViaThirdParty @alias("Via third-party")
  Manual
}

enum EntityStatus {
  Confirmed @alias("confirmed")
  Hypothetical @alias("hypothetical")
}

// ===== Helper Classes =====

class Position {
  x float @description("X coordinate (0-2000)")
  y float @description("Y coordinate (0-1500)")
}

class Dimensions {
  width float @description("Width in pixels")
  height float @description("Height in pixels")
}

// ===== UPSERT Operations =====

class UpsertNodeOperation {
  type "upsert_node"
  product_id string @description("UUID of the product from catalog")
  label string? @description("Optional custom label for the node")
  position Position @description("2D coordinates on canvas (0-2000 x, 0-1500 y)")
  parent_environment_id string? @description("ID of parent environment container (if any)")
  status EntityStatus @description("confirmed = user owns, hypothetical = recommendation")
  product_name string @description("Human-readable product name")
  tagline string? @description("Product tagline from catalog")
}

class UpsertEdgeOperation {
  type "upsert_edge"
  // DO NOT include edge_id - generated server-side
  source_product_id string @description("UUID of source node")
  target_product_id string @description("UUID of target node")
  label string @description("Edge label: '{integration_type}: {data_type}' (e.g., 'API: Customer data')")
  description string @description("ONE sentence describing data flow from source to target")
  bidirectional bool @description("true if data flows both ways")
  integration_type IntegrationType @description("How the integration works")
  status EntityStatus @description("confirmed or hypothetical")
}
I'm attaching the raw output from the LLM. It is parseable JSON, yet BAML is returning
operations
with only one element (the 8th element). I'm having this with many examples https://cdn.discordapp.com/attachments/1436167435878273207/1436167436008161393/message.txt?ex=6913e4a7&is=69129327&hm=5d93eedb4ec5d33759d709e89fd0e2767b55242f9f38539fb3961f7985014b7e&
It just happened again with this raw output
Copy code
json
[
  {
    "type": "upsert_node",
    "product_id": "d2c7b88a-3b6b-4b1b-8f1a-9e2b0a1b2c3d",
    "product_name": "Google Meet",
    "status": "confirmed",
    "position": {
      "x": 300,
      "y": 500
    }
  },
  {
    "type": "upsert_node",
    "product_id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "product_name": "Granola",
    "status": "confirmed",
    "position": {
      "x": 600,
      "y": 500
    }
  },
  {
    "type": "upsert_node",
    "product_id": "f1e2d3c4-b5a6-7d8c-9b0a-1c2d3e4f5a6b",
    "product_name": "Notion",
    "status": "confirmed",
    "position": {
      "x": 900,
      "y": 500
    }
  },
  {
    "type": "upsert_edge",
    "source_product_id": "d2c7b88a-3b6b-4b1b-8f1a-9e2b0a1b2c3d",
    "target_product_id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "label": "API: Audio stream",
    "description": "Google Meet provides the audio stream for Granola to record.",
    "integration_type": "API",
    "status": "confirmed",
    "bidirectional": false
  },
  {
    "type": "upsert_edge",
    "source_product_id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
    "target_product_id": "f1e2d3c4-b5a6-7d8c-9b0a-1c2d3e4f5a6b",
    "label": "Automatic: Meeting notes",
    "description": "Granola saves transcribed meeting notes into Notion.",
    "integration_type": "Automatic",
    "status": "confirmed",
    "bidirectional": false
  }
]
The parser output is
Copy code
{
  "operations": [
    {
      "type": "upsert_edge",
      "source_product_id": "d2c7b88a-3b6b-4b1b-8f1a-9e2b0a1b2c3d",
      "target_product_id": "a1b2c3d4-5e6f-7a8b-9c0d-1e2f3a4b5c6d",
      "label": "API: Audio stream",
      "description": "Google Meet provides the audio stream for Granola to record.",
      "bidirectional": false,
      "integration_type": "API",
      "status": "Confirmed"
    }
  ]
}
a
what's the function output type
?
l
Copy code
class DiagramCompositionResult {
  operations (UpsertNodeOperation | UpsertEdgeOperation | UpsertEnvironmentOperation | DeleteNodeOperation | DeleteEdgeOperation | DeleteEnvironmentOperation)[] @description("List of operations to apply")
}

function ComposeDiagram(
  request: DiagramCompositionRequest
) -> DiagramCompositionResult {
a
can you try commenting out
position
from your schema and rerun it?
l
Let me try
a
also parser issue aside, you will also get better results if you dont use uuids, but shorter int ids
weirdly enough this one works: https://www.promptfiddle.com/BAML-Examples-x3FbW but i dont have the rest of your classes
what version of the extension / baml are you on?
l
I removed
position
and it works. Why? Version
0.202.1
a
can you try using 0.213.0?
generator file also needs to say 0.213.0
l
Let me try
The extension stopped working and tests hang forever. Running on the CLI, it keeps failing ~50% of the time
a
on the CLI, it fails because the items are missing still?
when you run
baml-cli version
it also says 0.213.0?
l
Keeps on parsing one element instead of all 5 in
operations
baml-cli 0.213.0
a
if you can repro like in this link I can see more closely why it happens: https://www.promptfiddle.com/BAML-Examples-MqjY3
l
Let me try to reproduce it there
a
or even if yous end me the baml file or DM it to me i can repro
in Cursor
it's just that I haven't been able to get the same error
l
Hold on. Let me send you the files then
a
Update -- for everyone else to see: It's not a baml parser issue, but a problem with the prompt + schema. The prompt has examples that are in the wrong format. GPT4o corrects this, Gemini does not (so our parser fails).
b
That's one of the things I found working with BAML that makes it very awesome. It validates the schema, not just JSON Parsability. Working with smaller models I found that adjusting my prompt and even hand coding some details about the schema can make it output reliably in the exactly correct schema. Schema validation is way more useful than just JSON parsability
Valid Schema >>>> Valid JSON