https://docs.all-hands.dev/ logo
Join Slack
Powered by
Scores for Anthropic’s Mythos Preview model (a pre...
# proj-agent
l
Scores for Anthropic’s Mythos Preview model (a preview of their next model, not general release) are very good. But honestly I think a fair amount of this is due to memorization, e.g. on swe-bench verified: https://www.anthropic.com/glasswing
b
Thanks for sharing this! The highest improvement is on SWE Bench Multimodal, where scores improve by ~32% 🤯 - so the abilities seem have been improved if the evals were done honestly and assuming memorization was not the major contributor for this benchmark
l
I think there is probably data leakage honestly
👍 1
(but there’s no way to prove it since they’re not releasing the model or the results)
👍 1
b
Understood, thanks for the additional context! I wonder why don't Anthropic researchers ask Mythos to check the npm package before releasing claude code given its superior cyber security vulnerability detection/coding abilities lolsob
l
I think maybe they will now 🙂
😂 1
b
This model is too classified to be handed out for use on such trivial things 🙂
😆 1
m
Are there established techniques to detect data contamination or memorization in benchmark results?
I'm thinking of a procedure that would provide a confidence interval for how many instances were contaminated, as well as a contamination index. Two models might have similar scores on a benchmark, but if one has a significantly lower contamination index, it would imply the model is actually better at the task, as it is using its reasoning capabilities rather than just memorization.
I’ve looked into it, and while there is research in this direction, it seems like hardly anyone is actually calculating these metrics in practice. Or at least it is difficult to spot them on dashboardes.
l
Yeah, there are a lot but often they require access to the model logits which Anthropic doesn’t expose externally AFAIK. You can also sample many times from the model. Also: https://openhands-ai.slack.com/archives/C06PB3T5ZK6/p1775648067934549?thread_ts=1775587793.280989&cid=C06PB3T5ZK6
2 Views
Open in Slack
PreviousNext
Powered by