Discrete-event simulation benchmark measuring the serving-side overhead and system-wide recovery value of request cancellation handling strategies in LLM inference.
When a user cancels a streaming request, the server may continue decoding tokens, hold KV cache capacity, and keep batch slots occupied. This benchmark measures how much is wasted, how quickly each policy recovers, and how cancellation handling affects the latency of surviving requests.
How much does fast cancellation detection improve system-wide efficiency, and what is the right detection mechanism?
Prior benchmarks in this portfolio covered the full lifecycle of a serving request except its abrupt termination:
- token-streaming-latency-bench: SSE streaming pipeline
- kv-cache-aware-scheduler: KV budget management
- dynamic-batch-size-controller: batch dynamics under load
- adaptive-batching-policy-bench: batch composition changes
In production, cancellations are frequent:
- user closes the browser tab
- mobile network drops
- load balancer timeout
- frontend SSE disconnect
Without proper handling, the server continues decoding tokens for canceled requests, wasting compute and holding KV capacity that could serve other users.
Survivor p95 completion time improves by 250 to 1,550ms in high-cancel workloads through batch compaction and KV release.
Polling overhead stays below 0.4% of benefit at all intervals. The gap vs push-based detection doubles with every doubling of poll interval, tracking directly with post-cancel tokens generated.
| Poll interval | Waste saved | Overhead | Overhead % of waste | Gap vs push |
|---|---|---|---|---|
| 1 token | 36,693 ms | 98.4 ms | 0.37% | 422 ms |
| 2 tokens | 36,272 ms | 49.3 ms | 0.19% | 844 ms |
| 4 tokens | 35,428 ms | 24.8 ms | 0.10% | 1,688 ms |
| 8 tokens | 33,742 ms | 12.5 ms | 0.05% | 3,374 ms |
Delayed KV cleanup strands 450 to 2,691 request-ms of capacity after cancellation and reduces survivor p95 by 9 to 89ms, scaling with output length.
All cancellation handling strategies tested deliver positive net recovery at cancel rates as low as 6%. The choice between policies affects magnitude, not whether there is benefit.
| Cancel rate | push score | poll_4 score | poll_8 score |
|---|---|---|---|
| 6% | 659 | 617 | 584 |
| 18% | 1,965 | 1,857 | 1,759 |
| 35% | 3,887 | 3,667 | 3,453 |
Wasted compute, stranded KV, and survivor benefit are all largest when expected output length is high, consistent with the decode-dominates-cost principle from prior benchmarks.
| Policy | Kind | Detection lag |
|---|---|---|
| no_cancel_handling | baseline | never |
| poll_every_1 | poll | 1 token |
| poll_every_2 | poll | 2 tokens |
| poll_every_4 | poll | 4 tokens |
| poll_every_8 | poll | 8 tokens |
| push_cancel | push | 0 tokens |
| push_cancel_lazy_kv | push | 0 tokens, KV freed after delay |
| push_cancel_no_repack | push | 0 tokens, slot not repacked |
| perfect_immediate_cancel | oracle | 0 tokens, upper bound |
| Workload | Cancel rate | Description |
|---|---|---|
| low_cancel_rate | 6% | minimal cancellations |
| moderate_cancel_rate | 18% | realistic production rate |
| high_cancel_rate | 35% | heavy churn |
| long_output_cancel_heavy | 25% | long expected outputs |
| kv_pressure_cancel_heavy | 22% | high arrival rate, KV-constrained |
| bursty_disconnects | 20% base | periodic burst cancel spikes |
cd ~/dev/request-cancellation-overhead-bench
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -u run.py
Runtime: approximately 30-60 seconds. No GPU required.
results/
summary.csv
decision_map.csv
effect_isolation.csv
cancel_rate_sweep.csv
plots/
01_wasted_compute_*.png
02_survivor_p95_*.png
03_kv_waste_*.png
04_effect_isolation_*.png
05_cancel_rate_sweep_*.png
06_recovery_score_*.png
07_polling_frontier_*.png
request-cancellation-overhead-bench/
+-- src/
| +-- config.py
| +-- workload.py
| +-- server.py
| +-- bench.py
| +-- analysis.py
+-- results/
+-- plots/
+-- run.py
+-- README.md
+-- summary.txt
+-- design.md
+-- LICENSE
+-- requirements.txt
This project is a calibrated discrete-event simulation, not an end-to-end serving benchmark.
Each decode tick represents one token step. Batch step latency scales linearly with active batch size, so canceling a request early accelerates decode for all surviving requests in wall-clock time.
Three push variants isolate individual effects:
- push_cancel: stop decode + free KV + repack batch
- push_cancel_lazy_kv: stop decode + delayed KV free + repack
- push_cancel_no_repack: stop decode + free KV, no batch repack
For full design details, see design.md.
- Python 3.10+
- NumPy >= 1.26.0
- Pandas >= 2.0.0
- Matplotlib >= 3.8.0
No GPU required.
This project does not run real LLM inference.
Main limitations:
- synthetic arrival and cancel distributions
- batch step latency modeled as linear in batch size
- single model profile evaluated (7B)
- no preemption or priority scheduling
- no network-level cancel propagation delay
These are acceptable because the benchmark targets relative comparison of cancellation handling strategies, not absolute production forecasting.
See design.md for full discussion.
If you need a simple default:
- use push-based cancellation as the primary strategy
- if push is not available, poll_every_4 is the best polling fallback
- avoid coarse polling (every 8+ tokens) in long-output workloads
- always free KV immediately after cancel detection
- monitor wasted_compute_frac and kv_wasted_request_ms as production signals
- design.md -- detailed design rationale and modeling assumptions
- summary.txt -- concise high-level summary of findings
- LICENSE -- MIT License
MIT License -- Copyright (c) 2026 Joao Felipe De Souza
Joao Felipe De Souza 2026