GPT-6 Astra was launched less than a week ago!
GPT-6 Astra was launched less than a week ago! I have been following the news behind it - and vicariously enjoying people being blown away by it. It was one-shotting "so many cool use cases" (I ran through my credit limit before I could finish my "cool" use case). Jensen Huang was congratulating Sam Altman on the arrival of AGI with GPT-6 Astra, has to be the peak of that hype train.
That praise, however, was not unsubstantiated. GPT-6 Astra blew past 14 benchmarks. In ARC-AGI-3, it scored 99.9%. I mean, it has AGI in its name, so it must test for AGI, right? right? So, like, every good AI enthusiast, I started digging into these numbers and saw that it actually had 2 scores - 99.9% when it used OpenAI's own setup, but when forced to use ARC's own standard, provider-blind harness, it scored just 62.7%.
This number is huge, especially when compared to Claude's 30ish % and OpenAI's own GPT 5.6 Sol's abysmal 7.8%. Well, back when i was younger, if i came back with 62.7% in a school test, I would have lost TV and candy privileges, but I believe OpenAI will continue to let its models binge through Internet and have all the GPU candy it wants after this result. Sigh, double standards, I say.
Amazing ratings aside, do these numbers really matter to people like my clients, in the supply chain space? None of my clients is going to burn through their revenues trying to one-shot their own ERP/Planning/Reporting systems with Astra. Most of their problems are either sequential decision problems, or optimization problems, or some weird mix of both. They are building agents that can help them surface different options to solve a small subset of problems - do they need AGI for that? Anyone building agents for supply chain problems know that "What model to use" is only a small part of the question (everyone's first answer here is Claude, and as soon as they say that, they open the data residency/sovereign AI/governance Pandora box).
So what SHOULD we do with these benchmarks? Are they meaningless? Not really - they were built by researchers for researchers. The ARC Prize organization, true to its "Meesaiyil man ottavillai" (tamil joke meaning - mud not sticking to mustache - Google it) form, says that scoring 99.9% doesnt mean AGI is achieved. But the rest? They don't really care that much! The rest of the Enterprise AI world, who actually build narrow-scoped, bound, guardrailed, etc. etc. agents that help solve real-world problems - they really care about the cost. At scale. They care about the reliability, traceability, explainability - all those abilities. If someone is building a benchmark around that, that is the one that would be worth tracking for my clients and me.