A 744-Billion-Parameter Model on Hummingbird Rations
What if you could run one of the largest open-weight AI models in existence — a 744-billion-parameter Mixture-of-Experts model — on a laptop with just 25 GB of RAM and no GPU? That's exactly what colibrì, a new open-source inference engine, claims to do. And remarkably, it appears to work.
Colibrì runs GLM-5.2, a 744B-parameter MoE model, using a single ~2,400-line C file with zero external dependencies: no BLAS, no Python at runtime, and no GPU required. The project name is fitting — like a hummingbird hovering effortlessly while weighing almost nothing, colibrì keeps a giant model "alive" on a fraction of the resources you'd expect it to need.
The Core Idea: Stream What You Don't Need to Keep
The trick behind colibrì isn't magic — it's an elegant exploitation of how Mixture-of-Experts (MoE) models actually work. Although GLM-5.2 has 744 billion total parameters, only about 40 billion parameters are activated per token. Of those, just ~11 GB worth of routed experts actually change from token to token.
Colibrì splits the model into two tiers:
- The dense part (attention layers, shared experts, embeddings — roughly 17B parameters) stays resident in RAM at int4 precision, taking up about 9.9 GB.
- The 21,504 routed experts (spread across 75 MoE layers × 256 experts, plus the multi-token-prediction head) live on disk, occupying roughly 370 GB, and are streamed on demand using a per-layer LRU cache, an optional pinned "hot store," and the OS page cache as a free secondary layer.
In short: keep the small, always-needed part in memory, and treat the disk as a giant, lazily-loaded parameter warehouse for everything else.
What's Actually Under the Hood
This isn't a toy proof-of-concept — colibrì implements a surprisingly complete inference stack:
- Faithful GLM-5.2 architecture, validated token-exact against a Hugging Face Transformers reference implementation.
- MLA attention with a compressed KV-cache that's 57× smaller than a naive implementation.
- Native MTP speculative decoding, using GLM-5.2's own multi-token-prediction head to draft tokens the main model verifies in a single batched pass — lossless, even under sampling.
- Integer-dot kernels (int8/int4) tuned with AVX2, chosen per-shape based on actual measured performance rather than assumptions.
- DSA sparse attention, GLM-5.2's "lightning indexer" for top-2048 causal key selection, validated to reproduce dense attention exactly when forced to keep every key.
- KV-cache persistence, letting conversations reopen exactly where they left off — even after restarting the engine — with zero re-prefill.
- A learning cache that tracks which experts your usage actually routes to and automatically pins the hottest ones in spare RAM, meaning colibrì genuinely gets faster the more you use it.
There's even an experimental, opt-in CUDA backend for pinning hot experts in VRAM, and a native Windows 11 port built with MinGW-w64 — no WSL required.
The Honest Numbers
Colibrì's creator is refreshingly transparent about performance. On the reference development machine (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX), cold decoding costs roughly 11 GB of disk reads per token, which translates to somewhere between 0.05 and 0.1 tokens per second when the cache is cold. That's slow by any conventional standard.
But speed isn't really the point here. As the project description puts it: this is a 744B frontier-class model answering correctly on hardware that costs less than one H100's cooling fan. Warm caches, pinned "hot" experts, and speculative decoding push useful-response latency down considerably — the raw physics of disk I/O handles the rest.
Real Numbers from Real Machines
Community-submitted benchmarks paint an interesting picture of where the actual bottleneck lies:
- On a machine with only 24 GB of RAM, the expert cache gets capped small, keeping decode "cold" even with a fast disk — RAM, not disk speed, becomes the binding constraint.
- An Apple M5 Max with 128 GB of unified memory and a fast internal SSD reached about 1 token/second — a genuinely usable pace for a 744B model on a laptop.
- A Framework 13 with a Ryzen AI 9 chip improved from 0.29 to 0.37 tokens/second purely by giving the learning cache room to work, with expert cache hit rates climbing from 28% to 66%.
- On a Ryzen 9950X, swapping to a faster PCIe 5.0 SSD nearly tripled throughput, flipping the performance bottleneck from disk-bound to compute-bound.
The overall pattern is clear: once disk bandwidth passes roughly 5 GB/s, the bottleneck shifts from storage to CPU matmul throughput — at which point better kernels, more cores, or a CUDA-accelerated expert tier start to matter more than raw disk speed.
Why This Matters
Colibrì is a striking demonstration of how much headroom still exists in how we run large models, independent of the models themselves. By treating disk storage as a legitimate extension of memory — rather than assuming everything must fit in RAM or VRAM — a single developer working on a 12-core laptop with 25 GB of RAM has made it possible to run a model that would normally demand a rack of enterprise GPUs.
It's not fast. It's not meant for production serving. But as a statement about the accessibility of frontier-scale AI, colibrì is a compelling one: a 744-billion-parameter model, alive and answering questions, on hardware nearly anyone could own.
Try It Yourself
The project is open source under the Apache 2.0 license (GLM-5.2's weights are released separately by Z.ai under MIT). Getting started involves building the engine, converting the model weights from FP8 to its int4 container format, and pointing the CLI at the resulting directory:
cd c
./setup.sh # checks gcc/OpenMP, builds, self-tests
./coli convert --model /nvme/glm52_i4
COLI_MODEL=/nvme/glm52_i4 ./coli chatFor anyone curious about the limits of consumer hardware, or simply fascinated by clever systems engineering, colibrì is well worth a look — and the project's creator is actively seeking benchmark data from more powerful machines to help push those limits further.