Benchmarking Debacts A Reached Pokémon

Not Pokémon is safe from the back of Benchmarking AI.

Last week, a Post on x It went viral, claiming that the ultimate Gemini Gemini Gemini Gemini Clay Claims in the original Pokémon video of videos. Informat, gemini had reached the lavender city in a flow of a developer; Claude was Bow to the mount moon by the end of February.

Gemini is literally in front of Claude ATM in Pokemon after reaching the city of Lavender

119 Direct live only BTW, incredibly underlined flow pic.twitter.com/8avsovai4x

– Tough (@ tuch1a8) April 10 1755

But that the post failed to mention is that the gemins had an advantage.

As well as users on reddit has reported, the developer who keeps the gemini roar built a personalized minimap that helps the “tile” model in the game as cranable trees. This reduces the needs of Gemini to analyze Screenshots before making play decisions.

Now, Pokémon is a semi-serious benchmark to the best – few approaches is a very informative test of a model’s ability. But it is is it A structured example of as different deployments of a benchmark can influence the results.

Eg anthropic reported Two pointers for their recent anthropic 3.7 Sleep Model on Benchmark Swen-bow, which is designed to evaluate the capacity of the model. Claude 3.7 Sonnet accurate 62.3% accuracy to Sw-ban bank, but 70.3% with a “personal scalffold” that the anthropic developed.

Most recently, meta fine-tune A version of one of his most recent patterns, llama 4 maverick, to make it right on a particular benchmark, lm arena. The one’s The version of Vanilla of the model scroll significantly worse on the same evaluation.

Date you are benchmarks – Pokémon included – I am imperfect measures To start with custom and non-standard implementation threaten to the water mud even more. It is to say, it seems likely to compare models as they are released.