An IBM POWER8 S824 — a 2014 enterprise server you can buy secondhand for a fraction of one datacenter GPU — can run modern large language models entirely on CPU. Elyan Labs runs LLM inference on IBM POWER8 daily: a 16-core, 128-thread S824 with 512 GB of RAM serves a 120B-parameter model as our in-lab reasoning endpoint. No GPU, no cloud account, no per-token bill. This page covers how, and why we call it sovereign CPU inference.
LLM on IBM POWER8ppc64 llama.cpp VSX AltiVecNUMA inference CPU-only LLMsovereign AI
The POWER8 S824 pairs two 8-core POWER8 CPUs (SMT8, 128 hardware threads) with 512 GB of DDR3 across NUMA nodes. That memory capacity is the point: models that need expensive multi-GPU rigs to fit in VRAM simply load into system RAM here. Our production instance serves GPT-OSS 120B (Q4 quantized, two GGUF shards) with a 20B draft model for speculative decoding, on ppc64 under Ubuntu.
Stock llama.cpp runs on POWER8 but leaves most of the machine idle. The
llama-cpp-power8 tree
adds POWER-specific work: VSX/AltiVec vector kernels (vec_perm,
vec_msum) for quantized matmul, IBM MASS math library integration,
dcbt cache-resident prefetch hints that keep hot weights in
L2/L3, and hardware entropy from the POWER timebase register. On our S824,
TinyLlama 1.1B prompt processing went from 16.7 tokens/sec on the stock scalar
build to 147.5 tokens/sec with the full optimization stack — about
8.8×. A counterintuitive finding along the way: 64 threads beat all 128;
past that, SMT contention costs more than it pays.
Large-memory CPU inference lives or dies on memory locality. The ram-coffers project treats each NUMA node as a distinct “coffer” of weight storage: model tensors are banked per node, threads are bound to the node holding the weights they read, and prefetch is planned per coffer. Our locality benchmarks showed roughly a 2× bandwidth spread between the best and worst node placements on the same machine — placement is free performance. Colibri, our inference-engine work, carries these ideas forward with sparse-attention experiments targeting big-memory CPU hosts.
“Sovereign” means the model, the weights, and the compute are yours: on your premises, on hardware with no vendor lock, no API key, and no usage meter. Decommissioned POWER8 servers are abundant and cheap precisely because the cloud moved on. For labs, homelabs, and anyone who cannot ship data to a third party, a big-RAM POWER box is a practical path to running serious models privately. It is the same thesis as our llama.cpp port for PowerPC G4/G5 Macs and the transformer we run on a Nintendo 64, scaled up to server class: proof of physical AI. The same machine also attests to RustChain, our proof-of-antiquity chain, earning its keep twice.
llama-cpp-power8 on GitHub ram-coffers on GitHub colibri on GitHub Non-bijective collapse paper (DOI) Elyan Labs home
By Elyan Labs. Benchmarks measured on our own S824; your numbers will vary with model, quantization, and NUMA layout. Every CPU has a voice.