Ask GPT-4o, Gemini, or Claude to count how many times two overlapping circles intersect, or to trace which of two tangled lines connects two dots, and there's a good chance the answer will be wrong. Not because the underlying math is hard — a five-year-old could solve it by looking — but because the model doesn't actually "see" the image the way a person does. That gap is the subject of a new benchmark that, According to The Decoder, confirms leading multimodal AI systems still perform poorly at basic visual perception, even as they post impressive scores on other benchmarks.

This isn't a new complaint. Researchers have been flagging the same pattern for over a year: models that can pass bar exams, write production code, and summarize dense academic papers routinely fail at tasks that require nothing more than looking closely — counting overlapping shapes, judging whether two lines cross, or reading the position of a clock's hands. What makes the latest finding notable is less the discovery itself than the confirmation that, despite several model generations and heavy investment in multimodal training, the underlying weakness hasn't gone away.

For teams building products on top of vision-capable models, that's a practical problem, not an academic one. A lot of real-world AI use cases — document processing, UI testing, quality inspection, accessibility tools — depend on exactly the kind of precise, pixel-level perception these benchmarks say current models still struggle with.

Why perception lags behind reasoning

The disconnect comes down to how vision-language models actually process images. Rather than perceiving pixels the way a human retina does, most models convert an image into a grid of patch embeddings and feed those into the same transformer architecture used for text. That approach is very good at capturing the general gist of a scene — what objects are present, what a chart roughly shows, what mood a photo conveys — because that's the kind of signal present in the image-caption pairs these models are trained on.

It's much weaker at tasks requiring exact spatial relationships: counting, measuring distances, tracking overlapping edges, or judging precise alignment. Captions rarely describe "the two lines cross at approximately this pixel," so the model never learns to represent that kind of detail. The result is a system that can describe a photo eloquently while getting the number of objects in it wrong.

What the new benchmark adds

Where prior academic benchmarks like MMMU or ChartQA tend to reward the kind of holistic, gist-level understanding models are already good at, benchmarks built specifically around low-level perception — counting, geometry, spatial reasoning — keep exposing the same blind spot across model families and generations. The latest results add to that body of evidence rather than overturning it: newer, more capable models are not closing the gap at the rate their gains on reasoning and coding benchmarks would suggest.

That distinction matters for how the AI industry talks about progress. A model can post record scores on math olympiad problems or software engineering benchmarks and still fail a task a toddler manages easily, because these two families of skills — abstract reasoning and raw visual perception — appear to improve on different curves, or at different rates, inside the same model.

Practical implications for AI builders

If you're shipping features that rely on a model "looking" at an image, this pattern has concrete consequences:

AiiN's takeaway

The consistency of this finding across benchmarks and model generations suggests it isn't a bug that will simply disappear with the next scaling round — it likely reflects something more structural about how current vision-language architectures represent images. Until model builders address that directly, whether through different training data, different architectures, or hybrid pipelines that pair language models with dedicated vision systems, teams building on top of these models should treat visual perception as a known weak point to design around, not an edge case to ignore.