Researchers from MIT, Harvard, and the University of Chicago have proposed the term “potemkin understanding” to describe a newly identified failure mode in large language models that ace conceptual benchmarks but lack the true grasp needed to apply those concepts in practice.
It comes from accounts of fake villages – Potemkin villages – constructed at the behest of Russian military leader Grigory Potemkin to impress Empress Catherine II.
The academics are differentiating “potemkins” from “hallucination,” which is used to describe AI model errors or mispredictions. In fact, there’s more to AI incompetence than factual mistakes; AI models lack the ability to understand concepts the way people do, a tendency suggested by the widely used disparaging epithet for large language models, “stochastic parrots.”
Computer scientists Marina Mancoridis, Bec Weeks, Keyon Vafa, and Sendhil Mullainathan suggest the term “potemkin understanding” to describe when a model succeeds at a benchmark test without understanding the associated concepts.
“Potemkins are to conceptual knowledge what hallucinations are to factual knowledge – hallucinations fabricate false facts; potemkins fabricate false conceptual coherence,” the authors explain in their preprint paper, “Potemkin Understanding in Large Language Models.”
The paper is scheduled to be presented later this month at ICML 2025, the International Conference on Machine Learning.
Keyon Vafa, a postdoctoral fellow at Harvard University and one of the paper’s co-authors, told The Register in an email that the choice of the term “potemkin understanding” represented a deliberate effort to avoid anthropomorphizing or humanizing AI models.
Here’s one example of “potemkin understanding” cited in the paper. Asked to explain the ABAB rhyming scheme, OpenAI’s GPT-4o did so accurately, responding, “An ABAB scheme alternates rhymes: first and third lines rhyme, second and fourth rhyme.”
Yet when asked to provide a blank word in a four-line poem using the ABAB rhyming scheme, the model responded with a word that didn’t rhyme appropriately. In other words, the model correctly predicted the tokens to explain the ABAB rhyme scheme without the understanding it would have needed to reproduce it.
The problem with potemkins in AI models is that they invalidate benchmarks, the researchers argue. The purpose of benchmark tests for AI models is to suggest broader competence. But if the test only measures test performance and not the capacity to apply model training beyond the test scenario, it doesn’t have much value.
If LLMs can get the right answers without genuine understanding, then benchmark success becomes misleading
As noted by Sarah Gooding from security firm Socket, “If LLMs can get the right answers without genuine understanding, then benchmark success becomes misleading.”
As we’ve noted, AI benchmarks have many problems, and AI companies may try to game them.
So the researchers developed benchmarks of their own to assess the prevalence of potemkins, and they turn out to be “ubiquitous” in the models tested – Llama-3.3 (70B), GPT-4o, Gemini-2.0 (Flash), Claude 3.5 (Sonnet), DeepSeek-V3, DeepSeek-R1, and Qwen2-VL (72B).
One test focused on literary techniques, game theory, and psychological biases. It found that while the models evaluated can identify concepts most of the time (94.2 percent), they frequently failed when asked to classify concept instances (an average of 55 percent failure rate), to generate examples (40 percent), and to edit concept instances (40 percent).
As with the previously noted ABAB rhyming blunder, the models could reliably explain the literary techniques evident in a Shakespearean sonnet, but about half the time had trouble spotting, reproducing, or editing a sonnet.
“The existence of potemkins means that behavior that would signify understanding in humans doesn’t signify understanding in LLMs,” said Vafa. “This means we either need new ways to test LLMs beyond having them answer the same questions used to test humans or find ways to remove this behavior from LLMs.”
Doing so would be a step toward artificial general intelligence or AGI. It might be a while. ®
