Generative AI Is an Engineering Disaster
· The Atlantic
Editor’s note: This work is part of AI Watchdog, The Atlantic’s ongoing investigation into the generative-AI industry.
Visit asg-reflektory.pl for more information.
As they scramble to keep their systems online, AI companies are making things expensive for the rest of us. Large language models such as ChatGPT and Claude are so resource-hungry that tech companies may be purchasing 70 percent of the world’s supply of high-end computer memory, causing a shortage. As a result, the prices of computer memory and storage are skyrocketing: Hard drives that I bought for my reporting two years ago for $350 each were $800 when I checked two weeks ago, and are now out of stock. The prices of some laptops have gone up as much as 50 percent, and low-cost computers are being hit the hardest. Affordable entry-level computers may “disappear by 2028” according to one forecast. And the memory shortage is expected to continue for years.
The memory is being put into data centers, which tech firms are expanding at incredible speed. They are planning to multiply total U.S.-data-center capacity by a factor of eight over the next few years. The demand for electricity at these sites is already so great that some companies are repurposing jet engines to power them.
The problem is not simply that AI is being deployed so widely or quickly. Other computer technologies have seen similarly massive growth without triggering such a large spike in electricity or a shortage of computer components: Video and music are now streamed around the globe, accounting for many terabytes of internet traffic daily; the smartphone boom required the manufacturing of billions of devices that are now transferring huge amounts of data; billions of household devices are also now part of the Internet of Things; and whole industries have moved their operations to cloud software, which is hosted not in the sky but in, yes, data centers.
[Read: The $10,000 MacBook Pro is here]
The problem with generative AI, in the industry’s own jargon, is that it does not scale. The cost of growing from, say, a thousand users to a million is a key factor that venture capitalists examine when they evaluate start-ups. They want to see that the cost of adding each new user decreases over time, so that the company can support millions of users and make increasing profits. This is achieved partly through the careful engineering of computer systems that can efficiently handle more users who want to post photos, hail Ubers, or stream music.
With generative AI, the work of building efficient, scalable systems has not been done. And the problem is exacerbated by the ever-larger generative-AI models, which have grown from 175 billion parameters in 2020 to more than 1 trillion today, according to independent estimates (the actual sizes of the models powering products such as Claude and ChatGPT are secret). The large in large language model should not be a selling point. But the industry’s observation that bigger models tend to outperform smaller ones has given rise to a totemic belief in “scaling laws” that suggest any problem can be solved by simply making models bigger. “Maybe with 10 gigawatts of compute, AI can figure out how to cure cancer,” OpenAI CEO Sam Altman wrote on his blog in September.
Yet the returns are diminishing. The bigger an AI model is, the less it improves with each added parameter, and so it must be made bigger at a faster rate just to sustain steady progress. I asked a few AI researchers whether they could name any other real-world software that scales so poorly. None of them could think of any. Even outside the world of software, it’s hard to find a comparable example, given that economy of scale is the principle that has made light bulbs, cars, and clothing so affordable. By economic and engineering measures, generative AI might be the worst technology ever deployed.
[Read: Welcome to a multidimensional economic disaster]
But with the massive investment behind the current bloated approach, there may not be much will to change. Ilya Sutskever, a co-founder and former chief scientist at OpenAI, said in a November interview that companies take the brute-force approach “because it gives you a very low-risk way of investing your resources.” It’s harder, he argued, to invest in research that would reengineer a product currently accruing trillion-dollar valuations. Those who suspect we are in an AI-driven bubble economy have pointed out that the profitability of these companies remains an open question, largely because of the high cost and inefficiency of the technology.
Efficiency is a core principle of computer science. One of the first things undergraduates learn is that writing a program that sorts a list of 50 words is easy. But if you give that program 50 million words, it will likely run out of memory or take hours to finish. Much of computer science is learning the clever coding techniques that prevent this from happening. Many of these techniques take advantage of repeating patterns in the data so that as the program receives more input, it takes less time or memory to process each additional bit. Such efficiency is one reason that modern smartphones and computers are so capable and affordable. This is called logarithmic scaling, and it looks like this when you graph it:
Large language models do not scale logarithmically. When they’re given more words to process, they get slower and use more memory—the time and resources increase faster as the input grows. In technical terms, LLMs scale quadratically. Any computer-science student knows that this is very bad.
Epoch AI, an organization that tries to determine the costs of operating AI models, published a graph last year, which is reproduced here with permission. It shows the exponentially increasing costs of serving more “tokens”—the words users type to chatbots—with several public AI models.
AI does not have to be built this way. Traditionally, the goal of AI was to solve problems in ways that simulated human mental processes. Researchers observed their own thinking and tried to implement their mental habits in code. This approach has mostly been abandoned, partly due to the difficulty of discerning and articulating the rules of human thought, but it did have the benefit of consuming far fewer resources and data.
Today’s approach to AI doesn’t try to describe the rules of human thought; instead, it gives a computer millions of examples to imitate. The huge quantity of examples is one reason that large models can perform better than small ones when generating language, images, and music—they have more material to draw from. Some researchers want to bring back the old, more efficient approach and combine it with the modern approach, but so far these projects have not drawn nearly as much attention or funding as the models that power chatbots.
Chatbot companies are aware that their products are inefficient. Some have found techniques for improving performance, but they have not yielded significant gains. Occasionally, companies claim to have made breakthroughs—Anthropic CEO Dario Amodei has called them “compute multipliers”—but they are usually described in vague terms, and there is no evidence that the basic problems of quadratic scaling and exploding model size have been overcome. (Anthropic declined to comment on the record when I reached out to ask about this.)
Some researchers are working on extremely small models that require fewer data and less computing power. I spoke with Alexia Jolicoeur-Martineau, an AI researcher at Microsoft who independently designed one of these, and asked her about the industry’s brute-force approach. “It’s a bit insane,” she told me. “At some point you have to learn to be a bit more efficient.”
Last year, Jolicoeur-Martineau won a $50,000 prize for her paper on a “tiny recursive model” that doesn’t consume huge quantities of computing resources. “The idea that one must rely on massive foundational models trained for millions of dollars by some big corporation in order to achieve success on hard tasks is a trap,” she wrote. Her model isn’t a substitute for an LLM—it’s designed to solve logic problems in fields such as biology and electrical engineering, rather than generate language—but it can do some of the tasks that much larger AI models are currently being used for.
Yet we seem stuck with LLMs, perhaps because they have been so aggressively marketed. They are now being added to everything, whether you want them or not. In 2024 and 2025, they were integrated into both Windows and MacOS, which means that more computing power is now needed to run a basic personal computer. Smartphones are also being sold with upgraded hardware as companies anticipate new AI features. Inefficient AI is also being added to common programs such as Adobe Photoshop and Microsoft Word, meaning that computers need to be more powerful to run this software.
[Ted Chiang: No, artificial intelligence is not conscious]
This is all especially bad because computers are no longer improving at the rate they used to. Since the 1950s, manufacturers have learned to make microchips steadily faster, smaller, and cheaper, a trend known colloquially as Moore’s Law. But in the past few years, components have gotten so small that manufacturers have run into molecular-level limitations on shrinking them any further, which has slowed progress significantly.
Instead of shrinking components, manufacturers have been focused on developing new hardware that is tailored to AI. This has provided occasional incremental improvements in performance, but none has come close to keeping up with the exponential curve of AI’s increasing demands.
Ultimately, inefficiency may be of little concern to the people within the tech industry who believe that they are replicating intelligence itself. There is an almost-religious conviction among many in Silicon Valley that something mindlike could arise from LLMs, which are ultimately just statistical language-generating software—this, despite the software’s inability to recall basic facts, its lack of common sense, and its complete dissimilarity to a biological brain. Even Yann LeCun, one of AI’s “godfathers,” told The New York Times recently that “LLMs are not a path to superintelligence or even human-level intelligence.” But the mythological lure of AI is so strong that many engineers believe that nothing should stand in their way. Not even the basic task of writing efficient software.