A 1985 Cray-2 had enough memory to hold a GPT-2-sized model and enough speed to generate a token every second or two. Training that model on it would have taken between 1,501 and 25,017 years. This page is the arithmetic, the year it stopped being impossible, and the machine nobody built.
This analysis grew out of our measured work running transformers on 1980s and 1990s game consoles — see Transformers on Four Retro Consoles for the cycle counts that prompted the question.
Consumer hardware is one budget; a national laboratory is another, and the distinction lands somewhere surprising.
A Cray-2 (1985) could be configured with up to 4 GB of memory. Fifteen hundred million 8-bit weights occupy about 1.5 GB, so they fit. Inference costs roughly two operations per parameter per token, which at that machine's ~1.9 GFLOPS works out to a second or two per token. A GPT-2-sized model could plausibly have been run — not trained — on mid-1980s supercomputing hardware.
Training is where it stops. GPT-2's training run is on the order of 1021 operations. At 1.9 GFLOPS that is roughly seventeen thousand years.
So the era could have executed a large model and could not have produced one — which makes the same point from the other end. The missing ingredient was never the machine. It was the architecture, and the compute to fill it.
"Impossible" and "expensive" are different claims, and the boundary between them has a date. Below is the fastest computer on Earth in each year, against a single GPT-2 training run. The range reflects two credible estimates of that run's cost; the exact token count is debated.
| Year | Fastest machine | Sustained | Years to train GPT-2 |
|---|---|---|---|
| 1985 | Cray-2 | 1.9 GF | 1,501 – 25,017 |
| 1993 | Fujitsu Numerical Wind Tunnel | 124 GF | 23 – 383 |
| 1994 | Intel Paragon XP/S 140 | 143 GF | 20 – 332 |
| 1996 | Hitachi CP-PACS | 368 GF | 7.7 – 129 |
| 1997 | ASCI Red | 1,068 GF | 2.7 – 45 |
| 1999 | ASCI Red (upgraded) | 2,380 GF | 1.2 – 20 |
Roughly 27 Cray-2 systems were ever manufactured. All of them, running flat out for a year, complete somewhere between 1.8% and 0.1% of one training run. The 1980s could not have done this for any amount of money — the hardware did not exist in sufficient quantity on Earth.
By 1993 that collapses from twenty-five thousand machine-years to about twenty. By 1999 a single existing machine does it in a year.
What follows is speculative engineering. Every component shipped, every figure is arithmetic on published specifications, and the machine itself never existed. We are labelling that boundary rather than blurring it.
Nobody had to wait for a single fast computer. The render farm was already the state of the art: Beowulf clustering of commodity hardware was published in 1994, and television 3D animation was being produced on banks of Amigas by 1992. Shard the work instead.
| Node | Per CPU | CPUs for a 2-year run | Blades @ 32/blade |
|---|---|---|---|
| Pentium Pro 200 (1995) | 50 MF | 28,519 | 891 |
| PowerPC 604e 233 (1996) | 100 MF | 14,260 | 446 |
That design already existed. ASCI Red, delivered to Sandia in 1997, was 9,298 Pentium Pro 200 processors on a custom mesh. A GPT-2 run on it works out to 2.7 years.
The catch is not arithmetic, it is the network. Data-parallel training must all-reduce the entire gradient every step — 1.5 billion parameters at 32-bit is 6 GB. Over 10 Mbit links that is 9,600 seconds per step, and blades do not help: they reduce the number of slow links, not the bytes each one carries. A hundred thousand steps is thirty years of pure communication.
The lever is gradient accumulation, which is neither modern nor subtle — take fewer, larger steps.
| Accumulation | Steps | Time spent communicating |
|---|---|---|
| K = 1 | 100,000 | 30.4 years |
| K = 100 | 1,000 | 0.30 years |
| K = 1000 | 100 | 0.03 years |
At K = 100 the network cost falls to about four months, comfortably inside a two-year compute budget on 10 Mbit links.
Roughly 900 blades of 32 commodity CPUs, a 10 Mbit interconnect, gradient accumulation, and about two years. Every component shipped by 1996.
It was expensive rather than impossible — a national-laboratory budget, not a miracle. Nobody built it, because the architecture it would have trained was not published until 2017.
Every figure is arithmetic on published specifications: sustained (not peak) FLOPS for each machine as reported in the TOP500 list and contemporary vendor documentation, a GPT-2 training cost on the order of 1021 operations bracketed by two credible estimates of the token count, and the 27-unit Cray-2 production figure from Cray's own history. No machine on this page was benchmarked by us; the consoles on the companion page were.
Plausibly yes, for inference only. A Cray-2 (1985) could be configured with up to 4 GB of memory, enough to hold 1.5 billion 8-bit weights (about 1.5 GB). At roughly two operations per parameter per token and ~1.9 GFLOPS sustained, generation works out to a second or two per token. Running a GPT-2-sized model on mid-1980s supercomputing hardware was physically possible; training one was not.
No. A GPT-2 training run is on the order of 10^21 operations, which is 1,501 to 25,017 years on a single Cray-2. Roughly 27 Cray-2 systems were ever manufactured, so every one of them running flat out for a year completes between 0.1% and 1.8% of one run. The hardware did not exist in sufficient quantity on Earth.
Around 1997 to 1999. ASCI Red (1997, 9,298 Pentium Pro processors, 1,068 GFLOPS sustained) would take about 2.7 years in the optimistic estimate; the 1999 upgrade brings that to 1.2 years. A hypothetical 1996 cluster of about 900 blades of 32 commodity CPUs with 10 Mbit links and gradient accumulation (K=100) could have done it in roughly two years at national-laboratory cost. Nobody built it because the transformer architecture was not published until 2017.
Data-parallel training must all-reduce the full gradient every step: 1.5 billion 32-bit parameters is 6 GB. Over 10 Mbit links that is 9,600 seconds per step, and 100,000 steps is about 30 years of pure communication. Taking fewer, larger steps (K=100 accumulation, 1,000 steps) cuts communication to about 0.3 years, inside a two-year compute budget.