We just hit the local LLM tipping point

A 744 billion parameter frontier model, GLM 5.2, is now running on an ordinary laptop with 25 GB of RAM and no GPU. The tool making it possible is Colibrì, an open source inference engine written by a solo Italian engineer, JustVugg, in roughly 1,300 lines of pure C with zero dependencies: no Python, no CUDA, no BLAS, no Docker.

The important part is not the headline alone. Colibrì treats RAM and disk as a managed memory hierarchy and uses Mixture of Experts sparsity to stream only the parts of the model that a token actually needs. That is a concrete path for local models at frontier scale. It is also an honest look at the gap that remains: physics at the NVMe layer set a hard floor on speed. The future is local, but we are not there yet.

How Colibrì makes a 744B MoE fit on a laptop

GLM 5.2 is a Mixture of Experts model. Although the model totals 744B parameters, each token activates only around 40B of them. Of those, only about 11B change token to token. The rest are shared and dense. Traditional inference paths load the full model into VRAM, which is cost prohibitive. Colibrì’s README describes a different split that keeps the model’s dense backbone resident and streams the routed experts from disk as needed.

The dense part covers attention, the shared expert, and embeddings. Quantized to int4, this is about 17B parameters that sit in approximately 9.9 GB of RAM. The routed experts, 19,456 of them at roughly 19 MB each when quantized to int4 for a total of about 370 GB on disk, are pulled on demand from an NVMe drive. A per layer LRU cache keeps hot experts in memory, the OS page cache acts as a second tier for free, and there is an optional pinned hot store for the experts that show up constantly. In practice, this turns disk bandwidth and locality into the primary performance budget.

This routing split only works if the engine speaks the model’s dialect. Colibrì implements the GLM 5.2 forward pass faithfully, and the README reports token exact matches against a transformers oracle on a tiny random model in both teacher forcing, 32 of 32, and greedy decoding, 20 of 20. The router is DeepSeek V3 style with a sigmoid gate, a shared expert, and the first three layers treated as dense, so the MoE decisions line up with the reference. Integer matmuls run through hand tuned AVX2 maddubs kernels in a Q8_0 style format.

Engineering choices that make it run

A few techniques carry a lot of weight here. They reduce RAM pressure, hide disk latency, and turn GLM 5.2’s native features into real throughput gains. The README calls out several of them, and they are tightly coupled to the MoE streaming design.

MLA attention compresses the KV cache to 576 floats per token rather than 32,768, a roughly 57 times reduction. That keeps long contexts feasible on machines that only have a few tens of gigabytes free.
Router lookahead prefetch exploits the fact that expert selection is often predictable even before the router has formally run. The project reports it can anticipate a large fraction of choices, on the order of seventy percent. It is marked experimental, and it gives the engine time to pull likely experts from disk early and mask latency.
Native MTP speculative decoding uses GLM 5.2’s multi token head to draft 2.2 to 2.8 tokens per forward pass. With an int8 MTP head, the README shows 39 to 59 percent acceptance. With an int4 MTP head, acceptance drops to zero, a sharp edge that is documented with a clear fix.
Integer dot kernels accelerate the hot path. The author notes int8 matmuls run 1.4 to 2.5 times faster than float in this setup, which matters when every unnecessary millisecond becomes disk idle.

Several other features shore up correctness and usability rather than raw speed. Grammar forced speculative drafts via a GBNF subset make constrained output practical without writing custom decoders. Async expert readahead uses WILLNEED to nudge the kernel pager. DSA sparse attention is implemented in a way the README calls faithful to the reference. The KV cache persists across restarts so a crash does not wipe a long interactive session. A batch union MoE prefill merges requests to reduce duplicate expert fetches. Finally, RAM safety auto sizes the expert cache from MemAvailable to avoid out of memory events on tight machines.

The tradeoffs and honest numbers

The README is clear about performance envelopes on a basic setup. On WSL2 with 12 CPU cores, 25 GB RAM, and an NVMe drive mounted via VHDX, a cold start loads in about 30 seconds and the resident footprint during chat peaks around 20 GB. The cold decode cost is dominated by expert loads: roughly 11 GB of disk reads per token. If the SSD sustains around 1 GB per second, that yields about 0.05 to 0.1 tokens per second on a fully cold cache. On a faster machine with a high end SSD, the author reports around one token per second. Those numbers move once the cache warms and pinned hot experts are in RAM. With MTP enabled, end to end response latency falls considerably. The README does not give a single figure warm speed for the 25 GB RAM config.

Speculation is not a free lunch on cold caches. Each verified draft touches more experts than a single greedy step. The project reports that routing can jump from about 660 expert loads per token to roughly 1,100 until the cache settles. That is a net loss if the SSD is the bottleneck. Correctness has its own caveats too. In exact arithmetic the engine is lossless, but integer quantization and shape dependent rounding mean the non speculative byte stream is not guaranteed to match the single token greedy path. Near argmax ties, a slightly different rounding can flip a token. The README emphasizes that every emitted token is still the argmax of a valid forward pass, so the continuation remains correct. If byte exact reproducibility is required, set DRAFT=0 and also IDOT=0 and COLI_CUDA=0.

There is also a ceiling defined by residency. A GPU experiment in the README notes that six RTX 5090 cards, with enough combined VRAM plus CPU RAM to keep all experts resident, reach 6.84 tokens per second in a single request decode. That shows where the disk floor comes from. It also highlights the benefit of Colibrì’s optional GPU tier, which is loaded at runtime from coli_cuda.dll when available and otherwise falls back to CPU without error. On the platform side, the engine runs natively on Windows 11 via MinGW w64 and a _WIN32 compatibility layer, and it ships an OpenAI compatible API server behind coli serve along with coli plan and coli doctor to check a machine before loading. The README adds a note on SSD wear: read heavy streaming is not a meaningful wear risk, while swap traffic and sustained thermals deserve more attention under heavy use.

Why it still marks a shift for local first work

Speed is the honest limiter today, not feasibility. Colibrì proves that disk streamed MoE inference works on commodity hardware. It shows that VRAM is not a hard requirement if the model design is sparse and the engine can make locality predictable enough. That existence proof matters for a few immediate reasons, even at 0.1 tokens per second on a cold cache.

Privacy critical workloads where data cannot leave the machine gain a path to frontier class behavior without a cloud dependency.
Batch and overnight jobs, from document analysis to targeted corpus summarization, can trade latency for cost and control.
Research and cost experiments on open weight frontier models become possible at zero marginal cost per token after the one time download.

The remaining gap is mostly physics and format. NVMe keeps getting faster, and routing plus prefetch keep getting smarter. If more frontier models continue to ship as MoE, the distance between proof of life and usable keeps shrinking. A frontier class model answering correctly on hardware that costs less than a single H100 fan is the kind of milestone that looks obvious only in hindsight.

Getting started locally

The repository is open source under Apache 2.0 at https://github.com/JustVugg/colibri and ships as a single C engine with small headers. There are no runtime dependencies, and the repo builds with a single setup script. The binary includes an OpenAI compatible API so it can drop into existing tooling. The coli serve process exposes standard endpoints, which makes quick integration straightforward without custom adapters.

A pre converted int4 GLM 5.2 is available on Hugging Face under the mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp repository name and it takes about 370 GB on disk. The README flags one critical detail for speculative decoding: the MTP head must be int8 or draft acceptance will be zero. There is also a single command pipeline from FP8 to int4 for conversions. Before loading, coli plan and coli doctor will check the machine and report whether the RAM budget and disk layout will work for the expert cache and streaming path.

What to watch next for local models

I read this as a clean proof that the local future arrived on the correctness axis, and that throughput is the remaining price. The README does not hide the costs, and that is useful. The work shows people are finding the levers that matter: sparsity, locality, and predictable routing. If those keep improving, and if more models expose the hooks that Colibrì uses, frontier class answers on a private laptop will inch from hours to minutes and then to interactive. We are not ready yet, but the path is visible.

References

Colibrì: Pure C inference engine for GLM 5.2 MoE