New World Anthropic Mythos Preview https://www.anthropic.com/glasswing + https://red.anthropic.com/2026/mythos-preview/

Yeah, this is very impressive. The SWE-Bench scores seem too high though (indicating potential data leakage). We may need to start looking at different benchmarks. SWE-Bench pro is good in that it has both public and private datasets, but they didn’t report the numbers separately on them I think

What concerns me more is that this is essentially a precedent — a model being restricted to use only by US companies. I hope that once the infosec adaptation is complete, the model will be made available to everyone else, like Anthropic's other models.

Of course I hope they make it available, but even if they don’t there should be lots of other good options. Anthropic is typically 3-6 months ahead with their best models and the number of months is decreasing

Yeah, that makes sense

@limited-student-10747 what makes you say that about swe bench?

In my experience with SWE-bench verified, despite it being “verified” there’s a certain number of issues (over 6%) that have non-obvious design decisions that you need to get right in order to pass the tests. Separately, if you ask older versions of Claude about the PR name, it can give you a description of the method used in the PR some percentage of the time, indicating that it already knows the solution.

Super interesting. As models get better and better do you think it will be harder and harder to make reliable benchmarks?

Oh yes, it’s already happening! Using public data is tricky, and also it’s harder to find annotators that can reliably annotate tasks that are hard for language models. You also need to capture non-accuracy things, like the ability of the model to answer clarifying questions