When an organization releases a brand new AI video generator, it’s not lengthy earlier than somebody makes use of it to make a video of actor Will Smith consuming spaghetti.
It’s grow to be one thing of a meme in addition to a benchmark: Seeing whether or not a brand new video generator can realistically render Smith slurping down a bowl of noodles. Smith himself parodied the pattern in an Instagram put up in February.
Google Veo 2 has executed it.
We are actually consuming spaghett finally. pic.twitter.com/AZO81w8JC0
— Jerrod Lew (@jerrod_lew) December 17, 2024
Will Smith and pasta is however one among a number of weird “unofficial” benchmarks to take the AI neighborhood by storm in 2024. A 16-year-old developer constructed an app that provides AI management over Minecraft and exams its means to design constructions. Elsewhere, a British programmer created a platform the place AI performs video games like Pictionary and Connect 4 in opposition to one another.
It’s not like there aren’t extra educational exams of an AI’s efficiency. So why did the weirder ones blow up?
For one, lots of the industry-standard AI benchmarks don’t inform the common particular person very a lot. Companies usually cite their AI’s means to reply questions on Math Olympiad exams, or determine believable options to Ph.D.-level issues. Yet most individuals — yours really included — use chatbots for issues like responding to emails and fundamental analysis.
Crowdsourced {industry} measures aren’t essentially higher or extra informative.
Take, for instance, Chatbot Arena, a public benchmark many AI lovers and builders observe obsessively. Chatbot Arena lets anybody on the internet fee how effectively AI performs on explicit duties, like creating an online app or producing a picture. But raters have a tendency to not be consultant — most come from AI and tech {industry} circles — and solid their votes based mostly on private, hard-to-pin-down preferences.
Ethan Mollick, a professor of administration at Wharton, lately identified in a put up on X one other drawback with many AI {industry} benchmarks: they don’t examine a system’s efficiency to that of the common particular person.
“The undeniable fact that there usually are not 30 completely different benchmarks from completely different organizations in medication, in legislation, in recommendation high quality, and so forth is an actual disgrace, as individuals are utilizing methods for this stuff, regardless,” Mollick wrote.
Weird AI benchmarks like Connect 4, Minecraft, and Will Smith consuming spaghetti are most definitely not empirical — and even all that generalizable. Just as a result of an AI nails the Will Smith check doesn’t imply it’ll generate, say, a burger effectively.
One knowledgeable I spoke to about AI benchmarks prompt that the AI neighborhood concentrate on the downstream impacts of AI as a substitute of its means in slender domains. That’s smart. But I’ve a sense that bizarre benchmarks aren’t going away anytime quickly. Not solely are they entertaining — who doesn’t like watching AI construct Minecraft castles? — however they’re straightforward to know. And as my colleague Max Zeff wrote about lately, the {industry} continues to grapple with distilling a expertise as advanced as AI into digestible advertising and marketing.
The solely query in my thoughts is, which odd new benchmarks will go viral in 2025?
