By OpenAI's own testing,?? ?? ??? its newest reasoning models, o3 and o4-mini, hallucinate significantly higher than o1.
First reported by TechCrunch, OpenAI's system card detailed the PersonQA evaluation results, designed to test for hallucinations. From the results of this evaluation, o3's hallucination rate is 33 percent, and o4-mini's hallucination rate is 48 percent — almost half of the time. By comparison, o1's hallucination rate is 16 percent, meaning o3 hallucinated about twice as often.
SEE ALSO: All the AI news of the week: ChatGPT debuts o3 and o4-mini, Gemini talks to dolphinsThe system card noted how o3 "tends to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims." But OpenAI doesn't know the underlying cause, simply saying, "More research is needed to understand the cause of this result."
OpenAI's reasoning models are billed as more accurate than its non-reasoning models like GPT-4o and GPT-4.5 because they use more computation to "spend more time thinking before they respond," as described in the o1 announcement. Rather than largely relying on stochastic methods to provide an answer, the o-series models are trained to "refine their thinking process, try different strategies, and recognize their mistakes."
However, the system card for GPT-4.5, which was released in February, shows a 19 percent hallucination rate on the PersonQA evaluation. The same card also compares it to GPT-4o, which had a 30 percent hallucination rate.
In a statement to Mashable, an OpenAI spokesperson said, “Addressing hallucinations across all our models is an ongoing area of research, and we’re continually working to improve their accuracy and reliability.”
Evaluation benchmarks are tricky. They can be subjective, especially if developed in-house, and research has found flaws in their datasets and even how they evaluate models.
Plus, some rely on different benchmarks and methods to test accuracy and hallucinations. HuggingFace's hallucination benchmark evaluates models on the "occurrence of hallucinations in generated summaries" from around 1,000 public documents and found much lower hallucination rates across the board for major models on the market than OpenAI's evaluations. GPT-4o scored 1.5 percent, GPT-4.5 preview 1.2 percent, and o3-mini-high with reasoning scored 0.8 percent. It's worth noting o3 and o4-mini weren't included in the current leaderboard.
That's all to say; even industry standard benchmarks make it difficult to assess hallucination rates.
Then there's the added complexity that models tend to be more accurate when tapping into web search to source their answers. But in order to use ChatGPT search, OpenAI shares data with third-party search providers, and Enterprise customers using OpenAI models internally might not be willing to expose their prompts to that.
Regardless, if OpenAI is saying their brand-new o3 and o4-mini models hallucinate higher than their non-reasoning models, that might be a problem for its users.
UPDATE: Apr. 21, 2025, 1:16 p.m. EDT This story has been updated with a statement from OpenAI.
Stocksfc: The Football Stock Market Where Goals Earn You Crypto
Pixel 8 Pro has a new temperature sensor. Can it 'read' an overheated iPhone 15 Pro?
Is it OK to leave a group chat? Why you should, and how to do it.
macOS Sonoma: How to share passwords—even Netflix
Rafu Offering Free Tickets for Exclusive Screening of ‘Yo
Everything to know before seeing 'The Exorcist: Believer'
'Monster' review: Hirokazu Kore
John Oliver responds to Reddit protests on 'Last Week Tonight'
Web3 Corporate Innovation Strikes Geneva, STORM Partners Announces Lightningbox?
Microsoft ends free upgrade from Windows 7 to 11
INTO THE NEXT STAGE: MIS Stories Spotlighted in Bill Kubota's Next Documentary
Prime Day Echo Show deal: Save on Echo Show 10, more
Twitter/X just fired half of its election integrity team
Elon Musk sued by college grad he falsely accused of being a 'fed' posing as a neo
Dukascopy Revolutionizes p2p Crypto
Wordle today: Here's the answer and hints for October 2
Twitter/X CEO didn't seem to know about Elon Musk's mandatory fee plan
'Only Murders In the Building' Season 3 ending explained: We know who killed Ben
coL through; VG to face n!faculty
Elon Musk sued by college grad he falsely accused of being a 'fed' posing as a neo
The rise of edible plastics, from whiskey pods to potato forksMicrosoft, not Amazon, wins $10 billion contract from the PentagonRussian trolls on Instagram focus on Joe BidenHere's Elon Musk's latest selfAdobe exposed nearly 7.5 million Creative Cloud accounts to the publicFacebook's sins haunt Mark Zuckerberg at Libra congressional hearingSnapchat is still growing where it matters mostTwitter quietly killed its secret adJessica Biel did not care for *NSYNC at all in 1999 (and still doesn't)Twitter is working on a policy to fight deepfakes and it wants users' help IT PAYS TO KNOW: Build Muscle Before Surgery Wong Makes Emmys History as ‘Beef’ Wins for Acting, Directing, Writing DA Announces Charges of Murder, Attempted Murder in Fatal Sword Attack 20 Legacy Businesses to Visit in Little Tokyo Glenn Osaki to Lead USC U.S. Screening of ‘The Crimson Kimono’ at Occidental Band of Brothers: Veterans March in 2024 Nisei Week Parade CoinEx Launches New Insight Page for In Bitget Meetup in Croatia HLTV.org live from IEM6 WC
0.1683s , 9824.421875 kb
Copyright © 2025 Powered by 【?? ?? ???】OpenAI's o3 and o4,Feature Flash
