A Meta exec on Monday denied a rumor that the corporate educated its new AI fashions to current effectively on particular benchmarks whereas concealing the fashions’ weaknesses.
The govt, Ahmad Al-Dahle, VP of generative AI at Meta, stated in a submit on X that it’s “merely not true” that Meta educated its Llama 4 Maverick and Llama 4 Scout fashions on “take a look at units.” In AI benchmarks, take a look at units are collections of information used to guage the efficiency of a mannequin after it’s been educated. Training on a take a look at set may misleadingly inflate a mannequin’s benchmark scores, making the mannequin seem extra succesful than it really is.
Over the weekend, an unsubstantiated rumor that Meta artificially boosted its new fashions’ benchmark outcomes started circulating on X and Reddit. The rumor seems to have originated from a submit on a Chinese social media website from a person claiming to have resigned from Meta in protest over the corporate’s benchmarking practices.
Reports that Maverick and Scout carry out poorly on sure duties fueled the rumor, as did Meta’s resolution to make use of an experimental, unreleased model of Maverick to realize higher scores on the benchmark LM Arena. Researchers on X have noticed stark variations within the habits of the publicly downloadable Maverick in contrast with the mannequin hosted on LM Arena.
Al-Dahle acknowledged that some customers are seeing “blended high quality” from Maverick and Scout throughout the completely different cloud suppliers internet hosting the fashions.
“Since we dropped the fashions as quickly as they have been prepared, we count on it’ll take a number of days for all the general public implementations to get dialed in,” Al-Dahle stated. “We’ll hold working by way of our bug fixes and onboarding companions.”
