We Ran Transformers on Four Retro Consoles and Counted Every Cycle

The interesting part is that we can tell you exactly how many cycles each one takes, and we can tell you which of our own conclusions turned out to be wrong.

Every number comes from a real ROM running under an accurate emulator. Where a figure is an extrapolation rather than a measurement, we say so.

Three things that transfer

Low-bit weights help where the multiplier is scarce — and can hurt where it isn't

Ternary weights (−1, 0, +1) measured 53.6% faster than int8 on the Game Boy's SM83, which has no multiply instruction at all, and 57.0% faster on the N64's MIPS core. On the N64's RSP vector unit, the same weights measured 12% slower.

That is not a contradiction, and we tested both candidate explanations rather than picking one. The density advantage was real: ternary moved 3.74× fewer weight bytes. The unit was not bandwidth-bound either, running at roughly a fifth of RDRAM throughput. The cost was unpacking — 0.656 issue slots per weight against int8's 0.375.

On a vector unit whose vmadh performs eight 16×16 multiply-accumulates in a single instruction, there is no multiply to delete, so you pay only the unpack. On the 6502 the same question resolved a third way: a 4-bit weight with a lookup table is quick to read, but building the table index requires the accumulator, and the 6502 has no way to build it elsewhere. The design is structurally forced to spill.

Three platforms, three different deciding constraints: instruction set, issue slots, register pressure.

When you quantize matters more than how far

Post-training ternarization of our N64 model produced a fit loss of 6.63. The corpus is byte-level, so ln(256) = 5.545 — the model was worse than uniform random over bytes. Trained with the quantizer inside the loop, the identical 2,031,628-byte artifact reached 0.1012 against a full-precision control's 0.1007, and produced byte-identical output on every test prompt.

We checked whether quantization-aware training works by finding weights that happen to quantize well. It does not. Mean absolute weight 0.0748 against 0.0826, kurtosis 3.85 against 3.86, and both ternarize to the same 42.5% zeros. The distributions are indistinguishable; the rest of the network absorbs the error during training instead.

The practical consequence is direct: quantization-aware training let a 2.80× larger model fit the same console memory at no measured difference in training loss, where the int8 version of that shape overruns it by 2.4×.

Build the instrument first, then distrust it

Every genuine gain here appeared only after a measurement tool existed — and every first tool was wrong in a way that silently reversed a conclusion.

• Static instruction counting swung from −17% to +1% on a no-op line reorder, and separately called a measured 14% improvement a regression.
• A three-token output check passed two genuinely broken changes. One deleted live range checks and diverged at position 5. The other narrowed cache entries to 16 bits while values peaked at 110,944 — overflowing 44 times inside the three-token window it passed.
• A benchmark ROM that omitted the interface, tokenizer and vocabulary could not see a memory-bank overflow. We submitted a pull request upstream that broke the build, and found it only when we finally linked the real ROM.
• One N64 emulator priced an uncached cartridge read at approximately zero cycles. An accurate one measures 268.6. The same optimization measures −7.47% on the first and −14.2% on the second — and the first is deterministic to the last digit across runs.

The counterfactual

The Ricoh 2A03 shipped in 1975. Backpropagation was published in 1986. Cartridge bank switching, battery-backed memory, and every other primitive our NES port depends on were shipping consumer technology by 1985.

Our 6.36-million-parameter model represents on the order of 10¹⁵ training operations. A Cray-1 sustained roughly 160 MFLOPS in 1976, putting that at three to four months of machine time — expensive, and the kind of allocation researchers actually received.

The transformer architecture was published in 2017.

So the constraint was never the silicon. Small-scale neural inference was physically available for decades while the field pursued symbolic approaches: expert systems, LISP machines, hand-built ontologies. The hardware sat there the entire time.

A footnote on supercomputers

Consumer hardware is one budget; a national laboratory is another, and the distinction lands somewhere surprising.

A Cray-2 (1985) could be configured with up to 4 GB of memory. Fifteen hundred million 8-bit weights occupy about 1.5 GB, so they fit. Inference costs roughly two operations per parameter per token, which at that machine's ~1.9 GFLOPS works out to a second or two per token. A GPT-2-sized model could plausibly have been run — not trained — on mid-1980s supercomputing hardware.

Training is where it stops. GPT-2's training run is on the order of 10²¹ operations. At 1.9 GFLOPS that is roughly seventeen thousand years.

So the era could have executed a large model and could not have produced one — which makes the same point from the other end. The missing ingredient was never the machine. It was the architecture, and the compute to fill it.

When did it become buildable?

"Impossible" and "expensive" are different claims, and the boundary between them has a date. Below is the fastest computer on Earth in each year, against a single GPT-2 training run. The range reflects two credible estimates of that run's cost; the exact token count is debated.

Roughly 27 Cray-2 systems were ever manufactured. All of them, running flat out for a year, complete somewhere between 1.8% and 0.1% of one training run. The 1980s could not have done this for any amount of money — the hardware did not exist in sufficient quantity on Earth.

By 1993 that collapses from twenty-five thousand machine-years to about twenty. By 1999 a single existing machine does it in a year.

The machine nobody built

What follows is speculative engineering. Every component shipped, every figure is arithmetic on published specifications, and the machine itself never existed. We are labelling that boundary rather than blurring it.

Nobody had to wait for a single fast computer. The render farm was already the state of the art: Beowulf clustering of commodity hardware was published in 1994, and television 3D animation was being produced on banks of Amigas by 1992. Shard the work instead.

That design already existed. ASCI Red, delivered to Sandia in 1997, was 9,298 Pentium Pro 200 processors on a custom mesh. A GPT-2 run on it works out to 2.7 years.

The catch is not arithmetic, it is the network. Data-parallel training must all-reduce the entire gradient every step — 1.5 billion parameters at 32-bit is 6 GB. Over 10 Mbit links that is 9,600 seconds per step, and blades do not help: they reduce the number of slow links, not the bytes each one carries. A hundred thousand steps is thirty years of pure communication.

The lever is gradient accumulation, which is neither modern nor subtle — take fewer, larger steps.

At K = 100 the network cost falls to about four months, comfortably inside a two-year compute budget on 10 Mbit links.

It was expensive rather than impossible — a national-laboratory budget, not a miracle. Nobody built it, because the architecture it would have trained was not published until 2017.

Proof of Antiquity

Elyan Labs runs a blockchain that rewards vintage hardware for participating. Older silicon earns a higher multiplier, verified by hardware fingerprinting that emulators cannot fake. The premise is that old machines have worth beyond nostalgia.

This work is the other half of that argument. Not "old hardware deserves credit for existing", but "old hardware can do the thing everyone assumes requires a datacenter". A Game Boy Color generating a sentence is a stronger claim about antiquity than any multiplier we could assign it.

What we got wrong

We had two frontier models adversarially review these conclusions. They removed several, and the ones that survived are stated at the strength the evidence supports.

• None of the quantization techniques are new. Ternary weight networks, BitNet-style training, accumulator-aware training and cache quantization all predate this work. What is new is the measurement on these targets.
• Our "phase change" framing was confounded. Comparing 3-level to 8-level quantization changed the level count and the quantizer design together.
• We have not run on real hardware. Two independent emulators agreeing is not silicon. Our Game Boy assembly disables interrupts while the stack pointer walks an activation array, and a real console with a live display controller is less forgiving.
• Our corpora are saturated. Every model we trained lands at essentially the same fit loss. We can show a 2.80× larger model fits; we cannot show it is better, because our data has no headroom left. That is a data problem and we are not pretending otherwise.

Reproduce it

All four ports, the measurement harnesses and the findings journals are public.

• Scottcjn/gbc-transformer — Game Boy Color, 10.09×
• Scottcjn/legend-of-elya-genesis — Sega Genesis, exact 68000 cycles
• Scottcjn/legend-of-elya-n64 — Nintendo 64, RSP vector matmul
• Scottcjn/elya-nes — NES / Famicom, 0.681 s/token, token-exact

The harnesses matter more than the ports. If you take one thing from this, take the habit: build the instrument, calibrate it against something you can derive by hand, and check it against a second implementation before trusting any number it gives you.