Meta benchmarks for their new patterns are a little cheated

One of the New Flags Ai Models Meta released Saturday, Maverick, Second Ranks on LM ArenaA test that has humans raters compare models of models and choose they prefer. But it looks the version of meaverick that meta has engaged in LM Arena Sphere from version that is very available for developers.

As well as Different Ai researchers pointed on x, meta noted in their announcement that maverick on LM Arena is a “experimental chat version.” A chart on the Official website llamaMeanwhile, they are disclose of the Arena of the Speece Arena test has been conducted to use “Llama 3 Maverick optimized for conversation.”

As we wrote beforeFor different reasons, lm Arena has never been the most reliable measurement of a pattern of the model. But companies usually do not customize or otherwise tuned their models to score better on LM Arena – or have not admitted to do so, at least.

The problem with a single model a pattern, carrying a variant “and then” “Vanilla” is that makes the model that is to be model in particular contexts. Is still cheated. Ideal, benchmarks – inadeide inadeive as they are – Provide a snapshot of the strengths of a single model and weakness in a range of jobs.

Indeed, researchers to x have Stark observed Differences of the behavior of the publicly download maveric compared to the host model on LM Arena. The LM Arena version seems to use very emojis, and give incredibly long answers.

Ok llama 4 is def a cooked lol, what is this city yap pic.twitter.com/y3gvhbvz65

– Nathan Lambert (@natolamppert) April 6 1725

For some reason, the llama pattern 4 in arena uses a lot more emojis

together. Ay, look better: pic.twitter.com/f74odx4ztt

– Tech Devic Notes (@techdevnotes) April 6 1725

We’ve Arrived Meta and Chatbot Arena, the Organization keeping the Arena, for comments.