The Cost You Can't See

Every company that self-hosts a large language model makes the same bet twice a year: which accelerators to buy or rent, and how many. It is usually the largest line item in the budget, and it is almost always decided on aggregate numbers - tokens per second, cost per million tokens, a benchmark leaderboard.

Aggregate numbers are fine for choosing a vendor. They are useless for the question that follows: if this is too expensive, what exactly do we change? That question needs itemisation. And for DeepSeek-V4 - a 284-billion-parameter model released open-weight this year - one of its three architectural pillars had no itemised cost at all.

DeepSeek-V4 replaced the standard residual connection with something called manifold-constrained hyper-connections. The mechanism is elegant and the research case for it is strong. What matters commercially is simpler: it runs for every token, in every layer, on every request you serve, forever.

DeepSeek published its cost during training. Nobody had published its cost during inference - which is where the money actually goes, because you train once and serve continuously. So I built the accounting: a structural cost model in Rust, an instrumentation harness in Python, everything open source.

I derived the model from the shipped inference code rather than from the paper that describes it. That decision turned out to be the whole exercise.

The published description says the block runs once per layer. The shipped code runs it twice - once around attention, once around the feed-forward network, with separate parameters. Any capacity model built from the paper is wrong by a factor of two on this component.

Open to read

Two more differences point in the same direction. The three separate learned projections are one packed weight. And the block computes in 32-bit precision while everything around it runs in 16- and 8-bit - which means, on the released 284B model, this one component reads about 129 MiB of parameters for every single decode step. That is roughly the memory traffic of loading a 130-million-parameter model's entire weights, per token, forever.

None of this is an error in the paper. A mechanism description is not an implementation specification, and it was never meant to be one. But if your infrastructure plan was derived from the description, it inherited the gap. Technical due diligence that reads architecture papers and not shipped code produces plans that are confidently wrong in ways nobody notices until the invoice arrives.

Two findings changed how I would advise someone running this model.

The same model costs differently on different stacks - structurally, not incidentally. Serving engines implement this component with varying degrees of kernel fusion. The unfused path issues roughly thirty-seven times as many GPU instructions as the fused one to move essentially identical data. Same weights, same arithmetic, same output. The difference is entirely in how the work is scheduled.

That reframes a hardware decision. The question is not only which accelerator is fastest, but how mature a vendor's implementation is for the specific parts your model actually spends time in. That is a question you can only ask if someone has itemised the parts.

Cost share moves with concurrency, and not in the direction people assume. I expected this component's share of decode traffic to rise steadily with batch size. My own model contradicted me: it is U-shaped, worst at very low and very high concurrency, for two unrelated reasons.

Open to read

The operational consequence is direct. Batch size is the main lever teams tune for cost per token. If you characterise this component at a single batch size, you will be wrong - and the direction of your error depends on which one you happened to pick. That kind of mistake survives review because the number looks reasonable and nobody asks what it is a function of.

The model predicts 12,728 sequential operations in a single decode step. I could multiply that by a plausible per-operation cost, publish a latency figure, and make it look authoritative. The repository ships no default for that constant, because doing so would convert a structural count into a measurement that never happened.

I would rather publish a smaller claim I can defend than a larger one I cannot. That is not modesty - it is the entire value of the exercise. An itemisation you cannot trust is worse than no itemisation, because people budget against it.

Open to read

What I did instead was test the model against reality where I could. The prediction was 148 operations per site. I built an executable version of the same computation and measured 148. Six per iteration, predicted and measured. Every variant exact. It runs in under a second on a laptop, with no model weights required, and anyone can reproduce it.

Mid-project I also re-ran my own literature survey and it invalidated my original novelty claim - part of this had already been done by someone else. I narrowed the claim in the repository rather than hoping nobody checked. It sharpened the result: the part that was already studied turns out to be 0.017% of the component's memory traffic. The expensive part is the part nobody had looked at.

Two phases, both already instrumented. The first calibrates the model on real hardware: operations mapped to actual GPU work, and that per-operation constant measured rather than assumed.

The second is the number that matters commercially - this component's share of a live serving workload, across stacks with different implementation maturity. That is the number a capacity plan can actually use.

The structural half is complete, reproducible, and public. The rest is a hardware budget away.