Load Testing Language Models

I occasionally need to run load tests for language models. Here are a few checks that prevent a clean looking benchmark from measuring the wrong thing.

When comparing two runs, check that their input and output token length distributions are roughly the same. A bug that produces longer generations can increase end to end latency even when the serving system itself has not regressed.

Inspect samples of the actual input and output text as well. Corrupted prompts, garbage generations, or other data pipeline bugs can make the latency numbers meaningless. Also verify that the run completed without request or server errors.

Speculative decoding makes representative traffic especially important. Its performance depends on the drafter's acceptance rate, which depends on the input and output distribution. A synthetic dataset may produce a very different acceptance rate from production traffic and therefore a misleading result. Benchmark on data that resembles the distribution used to train or deploy the system.

Do not stop at averages. Check tail percentiles such as P95 and P99. Speculative decoding can improve average latency while worsening the tail on some workloads. In an unfavorable step, the drafter spends time proposing tokens, the verifier scores them, and the draft may still be rejected after little or no progress. The system then pays the speculative overhead while advancing by only one corrected or bonus target model token. In that case, generating the token without speculative decoding would have been cheaper.

Whether that tradeoff is acceptable depends on the product. A large average improvement with a small P99 regression may be reasonable for casual chat. If a model participates in a latency sensitive autonomous transport pipeline, however, an unexpected obstacle or dangerous road event may require perception and planning to respond within a fixed deadline. A rare P99 delay can miss that deadline, so predictable, bounded tail latency matters more than a better average.