https://www.boundaryml.com logo
Am hating the LinkedIn/HN feedback on
# general
p
Am hating the LinkedIn/HN feedback on the constrained generation vs. parsing blog post. Is there any public benchmark that actually shows constrained generation objectively works better than BAML does? I'm always at odds with the constrained generation folks. My take is they've been burned by pydantic and other ways of doing parsing in the past, so they refuse to try BAML with the belief that parsing will always suck. But I'm in the midst of some parsing experiments and BAML's quality of outputs astound me. And this is without even optimizing the prompts.
v
haha appreciate that! but in all reality, its very hard to believe that a parser can be that good πŸ™‚
p
Am writing a blog on this soon
v
i don't think there's a benchmark where BAML performs worse as far as I know, but people seem to be open to trying it out, so thats sufficient for now!
p
Lots more BAML education needed out there πŸ’ͺ
m
Is there a benchmark where BAML performs better? From 2025?
v
I have a few internal benchmarks from larger companies that use us (on their data). We were going to invest a week or two to write a new one, but honestly we opted out mostly because once people try it and measure on their own results, they often just get better outputs. that said, we likely should find some time to share a real benchmark re long horizon tasks!
though i think @prashanth has also run time benchmarks as well! (but may not be this year)
I actually wrote a slightly longer (more reasoning tokens) one here! https://discord.com/channels/1119368998161752075/1452820539465797713/1452822700254040154
r
What’s constrained output vs BAML? Link me. I keep hearing try
textract
not from users but from Google hype. It required few shot and I assume it uses structured output?
p
Constrained decoding, or "constrained generation" (basically outlines by dottxt, which you've already been reading about). They published benchmarks that show (apparently) that constrained decoding doesn't adversely impact generation quality for structured outputs, in fact, they claim it sometimes improves it. A simple Gemini deep research should get you all their claims and findings
But I have trouble believing anything published by a team whose future so strongly depends on the outcome speaking in their favour
That said, an outside observer say the same about BAML (the blog post by Sam clearly favours parsing over constrained decoding).
So I'm complaining about the community, where nobody on the outside has done the work comparing these two radically different approaches on diverse enough benchmarks
I have a DSPy vs. BAML benchmark suite here on structured outputs. Would be cool if somebody sent a PR to add outlines results to the mix, too πŸ˜„ https://github.com/thedataquarry/structured-outputs
My concern is that using proprietary LLMs in outlines is a pain - it needs some vLLM magic to be able to use with an arbitrary LLM that it doesn't natively support. BAML is awesome for how easy it makes to just get going
r
Constrained decoding = structured output from the LLM?
I do want UUIDs / function calling, but then again LLMs suck at them so I don't care.
I don't want my data stolen sequentially scanned, but that's what UUIDMapper is for πŸ˜„
Copy code
class UUIDMapper:
    """Maps UUIDs to integer IDs for BAML processing and back."""

    def __init__(self) -> None:
        """Initialize the UUID mapper."""
        self.uuid_to_int: dict[str, int] = {}
        self.int_to_uuid: dict[int, str] = {}
        self.next_id = 1

    def add_uuid(self, uuid: str) -> int:
        """
        Add a UUID to the mapping and return its integer ID.

        Args:
            uuid: The UUID string to map

        Returns:
            The integer ID for this UUID
        """
        if uuid not in self.uuid_to_int:
            self.uuid_to_int[uuid] = self.next_id
            self.int_to_uuid[self.next_id] = uuid
            self.next_id += 1
        return self.uuid_to_int[uuid]
...
I should probably use a UUID class, now that I think of it, I occasionally get shortened UUIDs back and need to validate.
p
Yeah I don't know how useful constrained decoding would be here in your case
It's mostly for standard NER-style structured outputs
8 Views