Your security team says this fix is free. It costs between $721 and $61,000 a year.

A security review lands on your desk. Multi-tenant inference, shared prefix cache, and a recommendation: turn on per-tenant cache_salt.

You look up the cost. Vendor documentation, summaries, and the cost tables that circulate in this space all say roughly the same thing. The salt only changes the hash input. Overhead is approximately zero.

So you approve it. It is free, and it closes a finding.

That number is wrong. On a mid-size cluster the real figure is $32,712 a year, and it can reach $61,000. On a small deployment it is $721. The interesting part is not any of those numbers. The spread is driven by a metric most teams are not measuring, which means most teams cannot price this decision at all.

The claim is true about the wrong thing. Hashing a block with a salt costs the same as hashing it without one - call it 100-200 nanoseconds per token, which on a 50,000-token context is a few milliseconds against a prefill measured in hundreds. Genuinely negligible.

But that is the cost of computing the mitigation, not the cost of having it. Per-tenant salting is per-tenant partitioning. That is the mechanism. The salt protects you by making one tenant's cache entries unreachable from another tenant's requests. Tenant B prefills the same system prompt again, on every request, forever.

Those destroyed cache hits are the cost. Not the hashing. Pricing the mitigation at zero tells a decision-maker that the strongest practical control is also the cheapest one available. That can get a control approved in principle, deployed to one service, and quietly abandoned when the first prefill bill arrives.

You cannot look this up, because it depends on your traffic. But it is straightforward to derive. The cost is the compute you spend re-prefilling tokens you used to get for free:

extra prefill tokens/day = requests/day x prompt tokens x hit rate x cross-tenant share
extra GPU-seconds/day    = extra tokens / prefill throughput
prefill throughput       = GPU FLOPs x MFU x GPU count / (2 x parameters)
extra $/day              = GPU-seconds x $/GPU-hour / 3600

Four inputs you already know - requests, prompt length, model size and GPU rate - one you can read off a dashboard - cache hit rate - and one you almost certainly cannot: cross-tenant share.

The same exercise on a small deployment - 2 L4s, an 8B model and 50,000 requests a day at 2,000 tokens - produces a very different table:

Same control. Same threat. Two orders of magnitude apart. There is no industry number for this, and anyone who quotes you one - including zero - is quoting you a number from someone else's cluster.

Hold the cluster constant and sweep only the share of cache hits that come from cross-tenant reuse:

The entire decision lives on that axis. Your cache hit rate is probably already on a dashboard; both vLLM and SGLang export it. The composition of that hit rate is not. Nobody is measuring how much comes from a tenant reusing its own multi-turn history versus landing on a block another tenant paid for.

Those two cases are indistinguishable in the metric you have, and they behave completely differently under isolation. Within-tenant reuse survives salting untouched. Cross-tenant reuse is exactly what salting destroys.

Two ways to get it, neither of which requires new infrastructure:

• Tag requests by tenant and compute hit rate twice - once normally, once with matching restricted to the same tenant. The difference is your cross-tenant share.
• Run the experiment directly: apply a per-tenant salt to a small slice of traffic and watch what happens to hit rate on that slice. That is a production-safe A/B and gives you the number alongside a partial rollout.

Either takes a sprint. Both are cheaper than being wrong about $61,000, and both are cheaper than deferring the decision indefinitely because nobody can size it.

Cost alone does not make this decision. You also need the value of what you are buying, and improved security posture is not a number you can put next to $32,712.

The threat here is a timing side channel. Shared cache means a cache hit skips prefill and returns sooner, so an observer who can time responses learns which prefixes are already resident. That work is established in the literature and needs no hash collision; it works against a fully patched server.

What that research does not give you is the exploitation cost on your system. Measure how reliably a timing observation distinguishes a cached prefix from an uncached one, express it as the adversary's success probability in a two-way forced choice, and a Hoeffding bound converts it into the number of measurements they need:

Now the risk is denominated in queries, and queries are something you already control. At 231 measurements per bit of inferred cache state, a per-tenant rate limit is a real, quantified mitigation. At 15, it is not.

Not as a binary. As a portfolio, in this order:

• Do the $292 thing first. SHA-256 block hashing is a rounding error and closes an entirely different attack class. If you are on vLLM 0.11 or later you already have it; the work is confirming nobody explicitly downgraded it for throughput.
• Then measure the cross-tenant share. One sprint converts a $0-to-$61,000 uncertainty into a number. Until you have it, every subsequent conversation is speculation.
• Then price rate limiting against salting. If the timing gap puts an adversary at hundreds of queries, a per-tenant rate limit buys most of the mitigation for a fraction of the cost. If the gap is wide, rate limiting is theatre and you pay for isolation.
• Then salt selectively, not globally. Salt the tenants whose prompts are commercially sensitive; leave shared-nothing tenants on the common cache. Buy risk reduction where it is worth something and keep cache economics where it is not.

The thing I would push back on in any version of this conversation: a recommendation to enable isolation that arrives without a cost, and a decision to defer it that arrives without one either. Both are the same failure. The number is derivable in an afternoon.

I built the tooling for this - KVSentry, Apache-2.0 - and I want to be precise about what is established and what is not, because a per-component verification column is cheaper than a bigger claim.

Every calibration constant in the cost model carries a provenance level. The model-size scaling anchors are marked SECONDARY, meaning transcribed rather than read off source tables, and any estimate resting on them says so in its own output instead of presenting itself as measured. Operators are expected to recalibrate against their own cluster; that path is a first-class API, not an escape hatch.

The detector is characterized against a simulator whose timing gap is exact by construction: zero false positives on a true null, 100% detection at a 4 ms gap, and observed effect sizes matching a closed-form expectation derived independently of the tool.

I also killed one of my own claims. I had written down, as an open question, that SGLang appeared to lack vLLM's cache_salt equivalent. I verified it against source. The hypothesis was false; SGLang has parity through cache_salt and extra_key on its request path. That was the most useful hour of the project, and it is why this note does not contain a wrong claim about SGLang.

What survived verification is narrower and applies to both frameworks: the isolation key is client-supplied. It protects you only where a gateway injects it server-side and strips whatever the client sent. Otherwise the attacker simply omits the field, lands in the shared default namespace, and every configuration check reports the deployment as clean.

Validation against live vLLM and SGLang clusters. The matrix is written; it needs a few hours of rented GPU. The question is narrow and specific: whether a real engine's timing lands where the simulation says it should.

Until then the cost model stands on its own, because it does not depend on the detector. Lost cache hits are lost cache hits whether or not anyone is measuring your latency.