The Ricoh 2A03 in an NES is built on a 6502 core designed in 1975. Backpropagation was published in 1986. The transformer architecture was published in 2017. The chip is older than the mathematics it is running.
NESFamicom6502Ricoh 2A03MMC5ternary weightsretro computing
A byte-level transformer running natively on NES hardware. The weights are on the cartridge and the 6502 does the arithmetic — nothing is streamed in from a PC, and there is no lookup table. It generates a token every 0.63 seconds, and across a 64-seed survey 2,432 of 2,432 generated tokens matched the reference implementation exactly.
You do not. You put it on the cartridge.
The port uses the MMC5 mapper and 32 KB of battery-backed PRG-RAM — the same class of trick The Legend of Zelda used to save your game. Weights are ternary (−1, 0, +1) and stored as sign-separated index streams, so the inner loop is add and subtract with no multiply at all. The 6502 has no multiply instruction.
The port also has no video output. It writes generated tokens into battery-backed save RAM, so the output survives a power cycle. A cartridge that appears completely dead has been quietly thinking the whole time.
Two findings came out of the NES that changed the other ports.
Longer context made it worse. Extending the window from 20 to 85 tokens raised the loss, and the attention had demonstrably learned to reach further — mean distance went from 0.91 to 7.82. The ceiling was capacity, not context. Buying capacity instead, through a mixture of experts spread across spare cartridge banks, moved the loss by 0.214 nats per character for 1.5% more cycles.
An 8-bit accumulator is 5× cheaper than 16-bit on this CPU, and the driver is register residency rather than add width: an 8-bit value lives in the accumulator, and anything wider spills to memory.
Not yet. The ROM is built and verified under a cycle-exact emulator, but the NES port is the one platform in the series still waiting on a working cartridge reader. The Sega Genesis and Nintendo 64 ports have run on real silicon.
github.com/Scottcjn/elya-nes The full measurements
The ROM, the training code, the cycle-measurement harness and the findings journal are all public, including the six times our own instruments gave us confident wrong answers.
Part of a five-console series from Elyan Labs. See also: NES · Sega Genesis · Super Nintendo · Nintendo 64 · Game Boy Color · the full measurements, including the conclusions we had to retract.