AI Fact-Checks the Fact-Checkers—Verifying Claims Directly

Separating Hype from Reality: How AI Helped Verify Performance Claims in Real Time

Introduction

An AI consultant working in the growth strategy space came across a provocative article on Medium titled “Apple Calls Bullshit On The AI Revolution”. The article claimed Apple scientists had found that both OpenAI and Meta’s models failed a basic logic and math test—casting doubt on the intelligence of today’s most advanced AI tools.

Rather than taking the article’s claims at face value, the consultant decided to run their own test using OpenAI’s GPT-4o and GPT-o1 mini — the same tools Apple scientists allegedly claimed had failed.

The goal was simple: Test the exact question and see if the AI could, in fact, solve it correctly.

The Challenge

The broader challenge highlighted in the article is one every business relying on AI should care about:

• Can AI handle basic logic and arithmetic reliably?
• Are public claims about AI failures rooted in real data or selective narratives?
• How can users distinguish between real AI limitations and sensational headlines?

For professionals incorporating AI into their daily work, understanding the actual limits of AI capabilities—rather than media spin—matters.

The Turning Point

The consultant directly tested the exact kiwi math problem cited in the article using both GPT-4o and o1 mini:

Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday. Five of them were a bit smaller than average. How many kiwis does Oliver have?

Both models gave the correct answer.

This simple exercise exposed a critical gap: sensational articles often fail to account for rapid improvements in AI models, updates between versions, and differences in how questions are phrased or framed.

The AI Solution

The consultant used AI itself not only to:

• Solve the problem — Verifying if current versions of GPT could pass the test.
• Explain the reasoning — AI showed its step-by-step logic, making it clear where the article’s claims were outdated or inaccurate.
• Frame the lesson — AI helped summarize why independent validation is essential whenever someone makes sweeping claims about AI’s shortcomings.

Remarkable Progress

This quick reality check delivered:

• Instant Verification – The consultant confirmed in under 5 minutes that the claim was no longer valid.
• Critical Insight – The consultant learned to treat all third-party claims about AI performance with caution.
• Process for Future Validation – This simple test approach can now be reused whenever media reports exaggerate AI limitations.

“This was a great reminder that direct experience beats second-hand claims every time.”

A Defining Moment

The defining moment came when both models confidently delivered the correct answer — and explained their reasoning.

“AI isn’t perfect, but watching both models get it right in seconds proved how fast these tools evolve — and how easily bad information spreads when people don’t check for themselves.”

Impact and Inspiration

This experience highlighted the importance of directly testing claims instead of blindly trusting headlines, especially in fast-moving fields like AI.

• Greater Trust in AI Tools – Direct testing built confidence in the actual capabilities of GPT-4o and o1 mini.
• Critical Thinking Shift – The consultant now views all AI performance claims with healthy skepticism.
• Faster Fact-Checking Process – A repeatable, AI-assisted fact-checking method is now part of the consultant’s workflow.

Protagonist’s Own Words

“AI moves too fast to trust outdated reports — if you want to know what’s real, test it yourself. This was a great reminder that direct experience beats second-hand claims every time.“

Conclusion

This case highlights how direct AI testing is essential for anyone relying on AI-powered tools in their work. Instead of assuming AI is either miraculous or broken based on secondhand reports, users can quickly validate performance themselves — separating fact from fiction.

Call to Action

Want to become confident in your AI tools and cut through the noise? Learn how Simpleacademy.ai’s training teaches you to independently test, validate, and trust AI performance in real-world scenarios — so you know exactly what AI can (and can’t) do.