MTPLX versus oMLX on a MacBook Pro M5 Max

The DGX Spark is Round 3 of my private engine tournament. Before the Spark, the local engine slot lived on a MacBook Pro M5 Max with 128GB of unified memory, running the same Qwen3.8-Flash-Next mixture of experts. Two MLX-native serving engines were in the running: MTPLX, and oMLX. One of them lasted a week.

This is also the post where I explain what the benchmarks are for, because engines that just sit there and win contests are trophies, not equipment. The router at the end spends these numbers every day, and it is the reason they had to be honest.

MTPLX versus oMLX on a MacBook Pro M5 Max

MTPLX got the first shot, and its startup numbers looked great. Warmup decode readings of 42 to 43 tokens per second, self-reported at every boot. What it also brought: a serve loop that crashed enough that I wrapped it in a retry harness and left an attempts log like a bad neighborhood, and a one-request-at-a-time server, which is a hard ceiling the moment you want a second thought while the first one is still talking. That warmup number did not survive contact with real traffic either.

oMLX was messier to set up and better to live with. The first week was a config sweep, most of it negative results:

• MoE expert offload sounded like exactly what a mixture-of-experts model wants. Measured: decode fell from about 61 to 24 tokens per second, because the engine streamed 88GB of experts from SSD on every single request. MoE experts are touched every decode token; offloading them is a survival feature for RAM-starved boxes, not a 128GB one.
• Keeping the model's persistent layer resident while offloading experts is forbidden by the engine itself, and the arithmetic explains why: resident, that layer alone needs about 111.6GB, and the server's memory ceiling sits at 115.4GB. The machine's whole life is a negotiation between those two numbers.
• KV quantization had a toggle war with MTP. The UI literally says "disable TurboQuant KV first." With this much RAM, the fight is pointless anyway.
• MTP depth four was tempting and measurably worse than depth three (2.58 to 2.88 accepted tokens per cycle versus 2.84 to 3.18), because the backbone cost per cycle is flat and a rejected fourth draft token is a full model pass for nothing. Depth three, in other words, was the ceiling. Optimize accepted tokens per cycle, never acceptance rate.

The keeper was an INT8 activation prefill path, worth a few percent on prefill for nothing, and a persistent-layer-on-SSD mode that accepts a scary-sounding warning to free the RAM that experts need. Settled config runs 46 to 61 tokens per second on decode, with real-world traffic in the 35 to 55 band depending on prompt size and cache hits. Prefill on genuinely cold prompts: about 1,800 tokens per second at 6k, about 2,145 at 12k, scaling sub-linearly.

MTPLX was disabled on October 11, its launch script renamed, its model left on disk. oMLX took the slot and the scorecard has been boring ever since: 4,215 requests, 293.9 million prompt tokens served against 280.7 million cached, two million tokens of output. In the engine business, boring is the trophy.

The benchmark traps from that week still apply to every engine since: a "fresh read" seconds after the last one is not cold, it is a cache lookup with a hat on; a 66-token-per-second short warm prompt and a 45-token-per-second real traffic run are different sports; and tiny prompts measure fixed overhead, not the engine.

The quiet round: the plumbing

These engines are not trophies. Something spends their hours every day, and it is the reason their numbers have to be honest.

I grew a small router out of Claude Code. One process, one port, and it decides per request where each message goes. The planning lane stays on the Claude models. The execution lane is the local engines from the rounds above. A day looks like this. Names and log lines are invented; the rhythm is not:

[orchestrator] plan: 4 steps, 2 parallel, gates: cargo+pytest
[subagent  step-2] local-flash-next   14:02:11
[subagent  step-2] edit crates/ledger/policy.rs (+41 -8)
[subagent  step-2] cargo test --workspace    64 passed, 0 failed
[subagent  step-2] report: step done, contract C7 unchanged
[orchestrator review] step-2 accepted  14:04:57
[orchestrator] dispatch step-3 (judgement call: retry semantics)
[subagent  step-3] local-flash-next   14:05:02
[subagent  step-3] report: BLOCKED : two callers disagree, 3 options
[orchestrator review] option B, reason: C7 forbids the silent path
[subagent  step-3b] local-flash-next  14:11:40
[subagent  step-3b] cargo test --workspace   71 passed, 0 failed
[subagent  step-3b] report: step done
[orchestrator] group merged, pushed, closing plan

Opus does the thinking, the plans, the review of work nobody has looked at yet. The smaller models do the drafting, the edits, the mechanical writing of a plan already made. When a subagent hits a real decision, like the two callers disagreeing above, it stops and reports instead of guessing. The expensive model answers, the cheap model executes the answer. Intelligence gets spent where it changes outcomes. Where the plan is already made and the tests are already written, a smaller engine on good silicon is enough engine. A big model writing a function and a small model writing the same function against a contract and a test suite are closer than anyone wants to admit, and the bill and the latency know it.

The router itself dispatches on the model id, fails over a dead hop before the client sees a byte, keeps a circuit breaker so a sleeping machine stays skipped for a while, and rewrites nothing else. It has run long enough to earn opinions: machines that come and go, thousands of real agent runs, enough failover nights that the breaker stopped being theoretical. It taught me the same lesson the benchmarks taught, one level up. You cannot route on advertised numbers. You route on what the client actually measured, and every number you have never re-derived is a rumor.

The next round is already queued, the one after that has a quantization in its eye. Measure the thing the client feels, keep the old engine one command away, and distrust whichever side of the race is winning too comfortably.