We Stopped Trusting the AI Leaderboards and Ran Our Own Tests | Edition 278

Chip stocks had an ugly Friday: the Philadelphia Semiconductor Index closed into a bear market, and Apple briefly overtook Nvidia as the world's most valuable company — roughly $4.88 trillion to Nvidia's $4.84. Analysts kept pointing at a leaderboard.

On July 16, Moonshot AI released Kimi K3 — 2.8 trillion parameters, open weights. It debuted at #1 on LMArena's Frontend Code Arena at 1,679 Elo, ahead of Claude Fable 5 and GPT-5.6 Sol; its predecessor sat at #18. The full weights are promised by July 27 — a promise on the calendar, not a file you can download.

Leaderboards are a number someone else calculated. So we sent three models identical prompts, ran what came back, and scored it.

One disclosure: our Claude column is Claude Opus 4.8, not the newer Fable 5. Opus 4.8 ranks below all three models above here.

Test 1 — the one the leaderboard measures

Build a full-screen animated particle flow field, mouse-reactive, no libraries. Claude Opus 4.8 and GPT-5.6 Sol delivered on attempt one. Kimi K3's first two runs burned their reasoning budget and returned nothing; our scored output was its third attempt. That reliability gap is part of the finding — the Elo score does not show it.

Test 2 — where we proved nothing

Preference tests measure taste; we wanted one right answer. So: a 16x16 pixel-art spec fixing all 256 cells by geometric formula, each model's code run in a sandbox, every cell diffed. All three scored 100%. That is a floor check, not a ranking — all three clear the bar; we draw no ranking from it.

Test 3 — six pixels

So we built a harder test: render the Mandelbrot set across 160,000 pixels to an exact spec — coordinate mapping, escape condition, color formula.

Claude Opus 4.8 and GPT-5.6 Sol: 160,000 of 160,000. Kimi K3: 159,994.

Six wrong pixels — 99.9963%. The signature matches a loop that skips the final escape check: six points that should be pure white came out black, in mirror-symmetric pairs — a genuine mathematical boundary, not noise. Daylight between an open model and the closed leaders now hides at the fourth decimal place — and this is one prompt, one run, one edge case a differently worded spec would not have caught.

Test 4 — what image models cannot count

We described a well-known painting — fifteen flowers, pale yellow ground, two-tone vase, 3:4 portrait — and asked two image models to produce it photorealistically from text alone.

Both nailed the atmosphere. The one machine-checkable requirement split them: Gemini 3 Pro returned 896x1200 — an ~3:4 portrait, as asked; GPT-5 Image returned a 1024x1024 square.

That is the image test's pattern, and why we built deterministic specs at all: models follow atmosphere instructions convincingly. The only way to catch them is to count something.

One admission: our first prompt said twelve flowers on deep ochre; the National Gallery says fifteen sunflowers on pale yellow. We rebuilt the test; the scores here are from the corrected version. Most benchmarks fail the same way: the test was wrong, not the model.

Can you just run it yourself?

Almost certainly not. K3 activates only 16 of its 896 experts per token, which cuts compute — but inference still needs the whole checkpoint in memory.

Moonshot's own serving guidance calls for a "supernode" of 64 or more accelerators, and the weights alone run past every consumer ceiling — even aggressive quantization lands between 650GB and 1TB, past a 512GB Mac Studio. Our rough estimate, and Moonshot hasn't published memory tables yet, puts a full-precision self-host north of $80,000 in hardware. Data-center gear, not a workstation purchase — if you're not running cluster-class hardware, plan on the API.

What open weights buy you: provider competition, no lock-in, and distillation into smaller models that do run on ordinary hardware.

What this means for you

Leverage. On July 1, GitHub added Kimi K2.7 Code to Copilot's model picker — GitHub calls it the first open-weight model offered in the picker; off by default on Business and Enterprise plans until an admin enables it. A credible alternative now sits in the same dropdown as the thing you are paying for.

Be skeptical of single benchmarks, including ours. K3 tops the frontend coding board but sits at #3 among distinct models on Artificial Analysis's Intelligence Index and, as of this writing, #8 on the general Text Arena. Moonshot roughly tripled its input price (about $1 to $3 per million tokens) and nearly quadrupled output ($4 to $15, per Artificial Analysis) versus K2.6. On Artificial Analysis's AA-Omniscience test it answers more questions correctly than its predecessor (46% versus 33%) — but when it is wrong, it is now likelier to answer confidently than admit it does not know: that hallucination rate rose from 39% to 51%.

Eighteen months ago, top-tier AI cost whatever the biggest labs charged. This weekend, on four tasks, one run each, a near-free open model matched two expensive closed ones almost pixel for pixel — when it answered at all. Bet on that trend continuing.