BudSimulator — Optimal, Math-Grounded Remediation Plan

Companion to bugs.md (39 findings) and audit_microbench_data.md (measurements on this NVIDIA GB10). Governing principle: no hardcoded magic numbers. Every constant traces to (a) a vendor datasheet, (b) a microbenchmark measured on this box, (c) a published benchmark with citation, or (d) a closed-form derivation. Where a value must be fit, it is fit by a documented, generalizable procedure (the differential_evolution calibration harness) against measured/published data — never hand-typed per case.

Per-unit solutions (RCA → solution → impact → regression → verification) are in §3, produced by the solution-design workflow and reconciled against the measurements below. §1–§2 are the empirically-derived backbone; §4 is the cross-coupling/rollout/regression plan.

1. The empirical foundation (measured, not assumed)

Run on this box (NVIDIA GB10, bf16, torch CUDA). Full data + method: audit_microbench_data.md.

Quantity	Measured	Consequence
Memory bandwidth (streaming)	206–211 GB/s = 73–77% of 273 peak	`eta_mem` ≈ 0.75 (LPDDR5X), not 1.0
GEMM achievable peak	85 TFLOPS = 34% of configured 250	`eta_compute` must derate to measured achievable
Roofline shape	`throughput = min(AI·205, 85)`, ridge AI≈419	per-op `max(compute,mem)` already encodes shape → dead TC-eff code is removable, not patchable
Kernel-launch floor	8.3 µs/op	grounds `t_launch` (GB10 decode calib 4.56µs ✓)
Power	idle 7 W → 96%-util 73 W	power is `idle + util·(TDP−idle)`, not fixed 0.41·TDP

Keystone end-to-end validations (1.5B Llama, real weights, on GB10) — the regression anchors

Decode (memory-bound): measured 70% MBU, TPOT≈15.9 ms. Sim default eff=1.0 → −30% (too fast). Calibrated eta_mem → ±7%. Confirms C2; proves the fix and its magnitude.
Prefill (compute-bound): measured ~27% MFU (60–68 TFLOPS). Sim now (250×0.85) → −68 to −72% (≈3× too fast). eta_compute fit to prefill → matches. Confirms C3.
Coupling proof: at 8k ctx, toggling full-S² vs causal-S²/2 attention shifts the error by ~11 pts — the calibrated eta_compute absorbs the FLOP-counting convention. ⇒ structural fixes first, then calibrate against the corrected structure (see §4).

These two anchors (decode −30%, prefill −3×) are the headline accuracy bugs and are now reproduced on real silicon with a real model. They are the keystone tests in the master regression suite (§4.3).

2. The master fix is one coherent mechanism, not 39 patches

The dominant errors (C1, C2, C3, C4, M4b, L0, L3, M14) are all facets of the same thing: the roofline is physically correct in structure but runs with peak specs and unity efficiency. The optimal, generalizable fix is a layered, calibration-based correction — every layer grounded:

Correct the structural inputs (so the roofline reflects real work):
- C1/C6/C7: dense bf16 per-chip FLOPS from datasheets (H100 989, H100-PCIe 756, Max-1550 ~839…), single source of truth shared by engine + catalog + training.
- M4b: causal attention counts the S(S+1)/2 triangle, not S² (exact, derived — not 0.5 magic).
- C4/L3/M3: delete the ad-hoc get_tensor_core_efficiency_factor / get_shape_dependent_op_intensity / OPERATION_CHARACTERISTICS and the flat 0.85 — the roofline already encodes shape via AI (§1).
- L0: tf32 compute multiplier 2 (A100: 156 vs 312 TFLOPS, datasheet).
Apply per-device efficiency eta_mem, eta_compute, t_launch, per_stream — fit by the existing validation/inference_calibration.py harness against a measured/published MBU·MFU grid, per device. Non-measured devices anchored by memory technology (HBM3 ~0.85, HBM2e ~0.8, LPDDR5X ~0.75 measured, GDDR ~0.75) and published cuBLAS/CUTLASS large-GEMM MFU (~0.7–0.85 of dense peak), with the citation recorded. No per-device hand-typed constants.
Calibration is fit AFTER step 1 (the coupling proof): otherwise eta silently absorbs the structural error and won't generalize to other models/contexts.

This is why C5 (18 unsimulatable devices) and the source-of-truth split fall out for free: once the engine builds a System from the same (dense, single-source) hardware row the catalog serves, every listed device is simulatable and shows the spec it computes with.

3. Per-unit solutions

Unit: hardware-single-source-dense-flops-full-simulatability

Approach. The original unit conflated three separable problems and got the physics wrong on two of them; this revision splits and re-grounds it. VERIFIED against the actual code and DB: (1) the compute path is compute_time = num_ops·bit_mult/op_per_sec with op_per_sec=flops/2 (system.py:100) and GEMM.get_num_ops = B·M·N·K MACs (operators.py:123-125) — so compute_time = 2·MACs/flops_raw, exactly as the critique states, NOT the inverted form the original wrote; (2) the engine today resolves Flops from STATIC configs only (set_hardware_db_path is never called at runtime — grep-verified), so H100_GPU=1979 (sparse) is what the engine actually simulates, yielding a physically-impossible implied 155% MFU vs the true 989.5 TFLOPS dense bf16 peak on a measured Llama-2-7B prefill; (3) get_tensor_core_efficiency_factor returns a CONSTANT 0.85 in the real GEMM/Logit/Attend path (the shape-dependent branches are dead code — get_dimensions() returns a list for GEMM and a 3-tuple for Logit, both failing the isinstance(tuple)/len>=4 guards — empirically verified across skinny and square shapes), so there is a fixed 1/0.85=1.176x double-correction, not the variable 0.4–0.95 the critique feared. The fix therefore is: (F1) audit the DB to dense bf16 per-chip peaks with a per-device sourced correction table and DEMOTE every real_values=0 placeholder out of the simulatable set rather than enabling it; (F2) make the engine resolve from the DB as the single source by calling set_hardware_db_path at startup, AFTER F1, so no shared name silently regresses; (F3) delete the dead tc_eff/shape-intensity double-correction so the roofline is clean before any eta calibration; (F4) repoint parallelism_optimizer.HARDWARE_CONFIGS at the same DB. Every numeric constant traces to a vendor datasheet; verification uses the engine's own closed-form (2·MACs/flops on a pure large-M GEMM) and asserts a full pre/post spec snapshot over ALL active devices, not a hoped-for MFU.

Net effect. Decode (bs=1, memory-bound): UNCHANGED (byte-identical) on every device — the FLOPS/tc_eff fixes touch only the compute term that loses max(compute,memory) in decode. Prefill / compute-bound (the corrected NVIDIA datacenter parts H100/H200/GH200): latency increases ~2.35x (F1 2.0x dense + F3 1.176x dead-code removal), driving implied MFU from a physically-impossible ~155% down to a measured ~66.5% vs the 989.5 dense peak — now plausible vs published cuBLAS/CUTLASS large-GEMM MFU (0.7-0.85) net of attention/norm overhead. RTX4090 prefill unchanged at the engine (stays 330.3 dense; the bug would have been INTRODUCED by F2-without-F1, now prevented). 19 DB-only placeholder devices: change from 'silently simulatable with garbage flops' to 'explicitly gated (raise/estimated)'. Memory footprint and KV-cache math: unchanged (no FLOPS dependence). Power/energy on compute-bound prefill: rises toward realistic (higher compute_utilization), magnitude to be re-snapshotted, not assumed.

Couplings. F1 and F2 are strictly ordered and MUST ship together: F2 (wire DB to engine) is only safe AFTER F1 (correct DB to dense + gate placeholders), otherwise wiring the DB regresses RTX4090 (330->661) and activates Cerebras/Groq placeholders in the engine. F1 and F3 COMPOUND on compute-bound latency: F1 doubles H100 prefill (1979->989.5) and F3 multiplies by a further 1/0.85=1.176x; combined ~2.35x — both are correct and both must be measured in the post snapshot (do not attribute the whole move to one). F3 is a HARD PREREQUISITE for the separate eta-calibration unit (per audit_microbench_data.md sec 6 plan-of-record): if eta_compute is fit by differential_evolution while the dead 0.85 is still applied, every fitted eta absorbs the 0.85 and stops being device-portable. F1's placeholder gate COUPLES to the power/cost endpoints: gating real_values=0 devices out of simulatability means power/energy/recommendation for those devices now error (or are marked estimated) instead of returning fabricated numbers — a deliberate, surfaced behavior change. The power model (serving.py estimate_power consuming compute_utilization) is coupled to F1+F3: doubling+1.176x prefill compute_time raises compute_utilization on compute-bound prefill, which raises energy/token; the power endpoint output must be re-snapshotted and is the subject of a separate finding (not silently assumed unchanged). F4 depends on F1/F2 (it delegates to the corrected central source).

F1-dense-flops-audit-and-placeholder-gating

Root cause: Two distinct defects share the FLOPS field. (a) Sparse-vs-dense unit error: several real_values=1 devices store the 2:4-sparse tensor-core peak instead of dense bf16. Verified: H100_GPU static=1979, GH200_GPU=1979, H200_GPU=1979 (NVIDIA H100 dense bf16 = 989.5 TFLOPS, the 1979 is the sparse figure); RTX4090 DB row=661 (Ada 2:4 sparse; dense bf16=330.3); B100 DB=3500 / GB200 DB=4500 are FP8 peaks, not bf16. Engine-resolved H100=1979 produces an impossible 155.1% MFU vs the 989.5 dense peak on a measured Llama-2-7B/2048-token prefill (achieved 1533.7 TFLOPS > 989.5 peak). (b) Unverified placeholders: 19 DB-only rows AND 4 shared rows have real_values=0 (A100_GPU, MI300X_GPU=1600, MI250X_GPU=957, Gaudi2=432, L40_GPU, V100_GPU=157, TPU_v2/v3/v4, Trainium_Trn1, Inferentia2_Inf2, Graphcore_IPU=250, Cerebras_WSE2/WSE3, Groq_LPU, etc.). These were never intended to be load-bearing simulation inputs — real_values=0 is the existing in-schema flag for 'estimated/unverified'. TPU_v3=420 > TPU_v4=275 is physically backwards (v3 dense bf16/chip is ~123 TFLOPS; 420 is a board/pod aggregate), and Cerebras_WSE2 static=7500 vs DB=50000 disagree 6.7x with no citation — these are guesses.
Solution: Two-part, no magic numbers. PART A (correct the real_values=1 rows to DENSE bf16 PER-CHIP, vendor-sourced): set H100_GPU=989.5, GH200_GPU=989.5, H200_GPU=989.5 (NVIDIA H100 datasheet dense bf16 989.5 TFLOPS); RTX4090=330.3 (NVIDIA Ada dense tensor bf16); B100=1750, B200_GPU/GB200=2250 (NVIDIA Blackwell datasheet dense bf16 per-GPU). Convention stated once in a module docstring: 'Flops = dense bf16 tensor TFLOPS per single chip; NEVER the sparse (2:4) or FP8 peak.' FP8 is handled by System.compute_multiplier['fp8']=0.5 (system.py:17) applied to the bf16 base — verified correct for Blackwell whose FP8:bf16 ratio is exactly 2:1; the multiplier model means we MUST store bf16, not FP8, for fp8 requests to scale right. PART B (gate placeholders out of simulatability, do NOT invent values): keep real_values=0 as the simulatability gate. Add a one-line guard in get_inference_system (utils.py, dict branch ~line 144) that, when real_values is False, refuses to fabricate performance and raises a clear 'unverified hardware spec — calibrate before simulating' error (or, if the caller passes allow_estimated=True, proceeds but stamps the result estimated=True). This makes F2 (wiring the DB) SAFE: enabling 19 DB-only rows does not silently feed garbage flops into latency, because each is gated until someone supplies a vendor/measured value and flips real_values=1. Devices that ARE confidently dense-bf16-per-chip sourceable (e.g. MI300X already correct at 1307 dense bf16, real_values=1) need no change. This is generalizable: every future import is gated the same way; nothing prevents a sparse/FP8 peak from entering, but it cannot become load-bearing until reviewed and flagged real.
Constants & sources: (1) H100/GH200/H200 dense bf16 = 989.5 TFLOPS — SOURCE: NVIDIA H100 datasheet (Tensor Core FP16/BF16 'dense' 989.5 TFLOPS; the 1,979 figure is the 'with sparsity' number in the same table). (2) RTX4090 dense bf16 = 330.3 TFLOPS — SOURCE: NVIDIA Ada Lovelace GPU architecture whitepaper / RTX 4090 spec (165.2 TF FP16 tensor FP32-accumulate dense doubling to 330.3; 661 = 2:4 sparse). (3) B100 = 1750, B200/GB200 = 2250 dense bf16 TFLOPS — SOURCE: NVIDIA Blackwell datasheet (per-GPU dense BF16; 3500/4500 are the FP8 dense peaks). (4) MI300X = 1307 dense bf16 TFLOPS — SOURCE: AMD CDNA3 / MI300X datasheet (Matrix bf16 1307.4 dense; already in DB, leave as-is). The DB-only MI300X_GPU=1600 row is real_values=0 and is GATED, not used. (5) FP8 compute_multiplier = 0.5 — SOURCE: system.py:17 (existing model). Verified self-consistent for Blackwell (FP8:bf16 = 2:1). (6) real_values flag — SOURCE: existing DB column and configs.py key (NOT a new constant). Used as the simulatability gate; 0 = estimated/unverified.
Code changes: [1] BudSimulator/data/prepopulated.db hardware table: UPDATE flops for real_values=1 rows H100_GPU 1979->989.5, GH200_GPU 1979->989.5, H200_GPU 1979->989.5; set DB row B100 3500->1750, GB200 4500->2250 (B200_GPU 2250 already bf16); ensure RTX4090 row = 330.3 — apply via a checked-in migration script, not a hand-edited binary. [2] llm-memory-calculator/src/llm_memory_calculator/hardware/configs.py:14-16 (and H200/GH200/B100/GB200 entries): set Flops to the dense bf16 per-chip values above; add a module-level docstring stating the 'dense bf16 per-chip, never sparse/FP8' convention. [3] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py:~144 (dict branch of get_inference_system): after NUM_FLOPS resolution, if system_name.get('real_values', True) is False and not kwargs.get('allow_estimated'), raise ValueError('hardware spec is unverified (real_values=0); supply a vendor/measured Flops and set real_values=1, or pass allow_estimated=True'); otherwise proceed and mark estimated. [4] NO code change writes a fabricated value for any of the 19 placeholders — they stay real_values=0 and gated.
Impact: Prefill (compute-bound) latency on H100/H200/GH200 DOUBLES (1979->989.5) — this is the intended correction (155% MFU was impossible). Decode bs=1 (memory-bound, max(compute,memory)=memory) is UNCHANGED because compute_time never wins there. Coupled computations: (1) get_best_parallelization_strategy ranks strategies partly on compute roofline — H100 strategies shift toward more parallelism, in the correct direction. (2) serving.py estimate_power consumes compute_utilization; doubling prefill compute_time raises prefill compute_utilization (was artificially low because flops were 2x too big), which RAISES energy/token estimates for compute-bound prefill — direction correct (more realistic), magnitude must be re-snapshotted. (3) Hardware recommendation rankings comparing H100 vs MI300X (1307, unchanged) narrow the H100 lead on prefill. The placeholder gate makes any caller that previously simulated A100_GPU/Gaudi2/Cerebras/etc. via the DB overlay now get an explicit error instead of fabricated numbers.
Regression risks: RTX4090: the DB row is 661 (sparse) but the engine today resolves the STATIC 330 (correct). If F2 wires the DB without fixing the DB row, RTX4090 would regress 330->661 (2x too optimistic). MITIGATION: F1 PART A fixes the DB row to 330.3 BEFORE F2 lands; the regression test asserts the post-DB-wiring engine-resolved RTX4090 Flops == 330.3. B100/GB200: storing bf16 (1750/2250) is correct ONLY if every fp8 request scales by 0.5; verified true for Blackwell but the regression test must include a bits='fp8' B200 case asserting effective peak == 2x the bf16 (i.e. 4500). Gating placeholders could break a currently-shipping demo that simulated a real_values=0 device; allow_estimated=True is the escape hatch and the error message names it.
Verification: BEFORE applying, run the engine's OWN closed form on a pure large-M GEMM to isolate the FLOPS fix from everything else: build GEMM(['g',1,8192,8192,8192,3]) and get_roofline on System(flops=989.5,bits='bf16',compute_efficiency=1) with tc_eff REMOVED (F3) — assert compute_time == 2*(8192**3)/(989.5e12) within 1e-9 (closed-form 2MACs/dense_flops). Then end-to-end: prefill_moddeling(Llama-2-7B, bs1, 2048 tok, H100=989.5) and assert implied MFU = (2Ptok)/(Latency989.5e12) <= 1.0 AND within [0.55,0.80] (cuBLAS large-GEMM published MFU 0.7-0.85, end-to-end with attention lands lower). Measured pre-fix value was 155.1% (FAIL); post-fix with tc_eff removed is ~66.5% (PASS). For each of the 19 placeholders, assert prefill_moddeling RAISES (gated), NOT 'returns finite>0'. REGRESSION CHECK: snapshot engine-resolved {Flops,Memory_BW,Memory_size} for ALL active (real_values=1) device names BEFORE F2 and assert post-F2 BYTE-IDENTICAL except the F1 PART A intentional changes (H100/H200/GH200 989.5, B100 1750, GB200 2250, RTX4090 330.3); assert decode bs1 TPOT on H100 byte-identical pre/post (memory-bound, must not move); assert prefill H100 latency post = 2.000x +/-0.5% of pre (the 1979/989.5 ratio) holding F3 constant.

F2-single-source-engine-resolves-DB

Root cause: There are THREE divergent FLOPS tables and the engine silently uses the WRONG one. (1) configs.py HARDWARE_CONFIGS (static, 73 entries) — what the engine resolves TODAY because set_hardware_db_path is never called at runtime (grep-verified: only exported in init.py, no runtime caller). (2) the DB hardware table (92 rows) — wired into BudHardware for the recommendation LISTING (commit 65ffe10, hardware.py:60-71) but NOT into the engine's module-level _default_manager. (3) parallelism_optimizer.HARDWARE_CONFIGS (7 rows, different keys like 'H100_80GB'=989). So get_best_parallelization_strategy can pick a strategy using H100=989 while the engine then simulates with H100=1979 — a self-inconsistent pipeline.
Solution: Make the DB the single source for the ENGINE too, but ONLY after F1 so nothing regresses. Call set_hardware_db_path(prepopulated.db) once at BudSimulator API startup so the engine's _default_manager overlays the (now-corrected, placeholder-gated) DB. Because manager merges {**static, **db} (manager.py:86, DB wins), the corrected DB rows define engine behavior. F1 guarantees the 8 shared names that differ are intentional (H100/H200/GH200 dense, B100/GB200 bf16, RTX4090 dense, Groq/Cerebras gated by real_values=0). This collapses sources (1) and (2) into one. For source (3), see F4.
Constants & sources: (1) No new numeric constants — this finding only changes WHICH table is authoritative. (2) DB path = BudSimulator/data/prepopulated.db — SOURCE: existing health.py:71 resolution; reuse the same Path.
Code changes: [1] BudSimulator API startup (apis/main.py or run_api.py lifespan/startup): from llm_memory_calculator import set_hardware_db_path; set_hardware_db_path(str(prepopulated_db_path)) — call ONCE, AFTER F1 migration has run. [2] llm-memory-calculator/src/llm_memory_calculator/hardware/manager.py:86 — keep the {**base, **hw} merge (correct now that DB is the authority); add a comment that DB is the single source of truth and static is the fallback/metadata layer. [3] No change to get_inference_system resolution order (utils.py:128 already calls get_hardware_config first).
Impact: After F1+F2, get_best_parallelization_strategy, the engine, and the recommendation listing all read the same corrected DB. The 19 DB-only devices become VISIBLE to the engine but GATED (F1 PART B), so they raise instead of fabricating. Power/cost endpoints that read manager flops now agree with the engine. This is the actual 'single source of truth' the unit promised — previously unmet because the engine used static.
Regression risks: If F2 is deployed WITHOUT F1, every shared-name change in the DB takes effect uncorrected: RTX4090 330->661 (regress), H100 1979->989 (correct), Cerebras/Groq placeholders become engine-active (regress). HARD ORDERING: F1 migration MUST be in the same commit/release as F2; the regression snapshot test FAILS the build if any active device's engine-resolved Flops differs from the intended post-F1 value.
Verification: After F1+F2, assert get_inference_system('H100_GPU').flops corresponds to 989.5 TFLOPS (op_per_sec == 989.5e12/2). Assert the engine-resolved spec dict for all 73 static names equals the corrected DB values (the snapshot). Cross-check: get_best_parallelization_strategy('H100_80GB') hardware flops == engine-resolved H100_GPU flops (989.5) after F4. REGRESSION CHECK: full spec snapshot over ALL active devices pre/post must match the intended post-F1 table exactly (this test replaces the original false 'all 73 unchanged' claim with 'changed exactly as F1 specifies'); build fails on any unlisted delta.

F3-remove-dead-tensor-core-double-correction

Root cause: operator_base.py:465-469 divides compute_time by effective_compute_efficiency = system.compute_efficiency * tensor_core_eff. EMPIRICALLY VERIFIED that get_tensor_core_efficiency_factor returns a CONSTANT 0.85 for every real GEMM/Logit/Attend op: for GEMM get_dimensions() returns a LIST (operators.py:102-103) so the isinstance(dims,tuple) guard at operator_base.py:416 is False; for Logit it returns a 3-element tuple so len(dims)>=4 is False; both fall through to the 0.85 default (line 433). The shape-dependent get_shape_dependent_op_intensity (operator_base.py:110) is likewise bypassed at line 453 because get_dimensions() is a list for GEMM. So both 'Phase 2' shape models are DEAD CODE that nonetheless impose a flat 1/0.85 = 1.176x compute speedup on top of the FLOPS — a hidden, undocumented efficiency multiplier with no physical source. The GB10 microbench (audit_microbench_data.md sec 3) proves the real roofline max(compute,memory) per-op ALREADY encodes shape dependence, so this layer is redundant, not just dead.
Solution: Delete the tc_eff and shape-intensity application from the compute path (per HARD RULE 5: remove redundant complexity the physical model already covers). At operator_base.py:464-470 stop dividing by tensor_core_eff; effective_compute_efficiency = system.compute_efficiency only. At operator_base.py:452-460 use op_intensity = num_ops/num_data (the simple ratio) — the shape branch never fired anyway. Keep get_tensor_core_efficiency / get_shape_dependent_op_intensity functions for the reported 'TC Effcy' diagnostic column if desired, but they must NOT alter latency. This makes per-device achievable efficiency come ONLY from the calibrated eta_compute/eta_mem (the inference_calibration block, already wired in utils.py:149-155 and GB10-validated), which ARE measured ratios — never a hidden 0.85.
Constants & sources: (1) 0.85 — the value being REMOVED. It had no source (HARD RULE 1 violation); it is the literal default-return at operator_base.py:434. Removing it is the fix. (2) No new constants introduced.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:464-470: remove tensor_core_eff multiplication; set effective_compute_efficiency = system.compute_efficiency; compute_time /= system.compute_efficiency. [2] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:452-460: drop the get_shape_dependent_op_intensity call (dead for GEMM); use op_intensity = num_ops/num_data if num_data else 0. [3] Keep f'TC Effcy' in the ret dict as a constant 1.0 (or drop the column) to signal the double-correction is gone.
Impact: Every compute-bound op's latency increases by 1/0.85 = 1.176x. Combined with F1 (H100 1979->989.5 = 2.0x), total H100 prefill latency moves 2.0/0.85 = 2.35x vs today; implied clean MFU drops from the impossible 155% to ~66.5% (measured). Decode bs1 unchanged (memory-bound). This is a prerequisite for ANY eta calibration: per the audit_microbench_data.md plan-of-record (sec 6), structural fixes (dense FLOPS + dead-code removal) MUST land before differential_evolution fits eta_compute/eta_mem, else the 0.85 is double-counted into every fitted eta and the etas stop being device-portable.
Regression risks: Memory-bound ops (decode, small-M) are unaffected (compute_time loses max()). Any test that snapshotted prefill latency WITH the 0.85 baked in will change by 1.176x — these snapshots must be regenerated; the change is uniform and explainable. The critique's worry that tc_eff in [0.4,0.95] could push MFU>1 on skinny GEMMs is MOOT: tc_eff is provably constant 0.85, never <0.5, so removing it can only INCREASE latency (reduce MFU), never create MFU>1.
Verification: Closed-form: GEMM(B=1,M=N=K=8192) on System(flops=989.5,ceff=1,bf16) — assert get_roofline compute_time == 2*(8192**3)/989.5e12 EXACTLY (no 1/0.85 factor). Pre-fix this is off by 1/0.85; post-fix exact. Sweep M in {1,16,128,2048,8192} and assert achieved-TFLOPS curve is monotone and capped at 989.5 (clean roofline, matches the GB10 measured-curve shape in audit sec 2). REGRESSION CHECK: assert decode bs1 TPOT byte-identical pre/post (memory-bound); assert ALL prefill latencies increase by exactly 1/0.85 vs pre-F3 (holding flops fixed) — a single uniform factor proves no shape-dependent behavior was lost.

F4-repoint-parallelism-optimizer-table

Root cause: parallelism_optimizer.py:244-296 defines its OWN HARDWARE_CONFIGS (7 rows, keys 'A100_80GB','H100_80GB','MI300X','GB10','TPUv4','TPUv5e' — different names than configs.py/DB). get_hardware_config (line 299) reads only this table. So get_best_parallelization_strategy can choose a strategy with H100=989 (it happens to be dense-correct here, line 260) while other devices/keys drift. The unit's 'single source of truth' is unmet while this third table exists.
Solution: Repoint parallelism_optimizer.get_hardware_config at the central manager (the DB, post-F1/F2). Map its short keys ('H100_80GB'->'H100_GPU', 'A100_80GB'->'A100_80GB_GPU', etc.) to canonical names and delegate to llm_memory_calculator.hardware.get_hardware_config, returning {Flops,Memory_size,Memory_BW,ICN,real_values}. Delete the inline 7-row dict. Now all three former sources are one.
Constants & sources: (1) No numeric constants — this is a delegation/aliasing change. The key-alias map (H100_80GB->H100_GPU etc.) is a naming map, not a physics constant; each target name's Flops comes from the F1-corrected DB.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/parallelism_optimizer.py:244-318: replace the inline HARDWARE_CONFIGS dict and get_hardware_config with a thin wrapper that maps legacy short names to canonical names and calls hardware.get_hardware_config; raise the same ValueError for unknown names (preserve API).
Impact: get_best_parallelization_strategy now uses identical flops to the engine and the recommender. For H100 the value is unchanged (both 989). For any device where the inline table drifted from the DB, strategy selection now matches simulation. Removes a whole class of 'optimizer picks a strategy the engine then re-prices differently' inconsistencies.
Regression risks: If a legacy short key has no canonical DB counterpart, the wrapper must still resolve it or raise — covered by keeping the ValueError path. Strategy outputs for the 7 supported devices must not change EXCEPT where the inline table was already wrong vs the DB (none for the dense-correct H100; verify the other 6).
Verification: For each of the 7 legacy keys, assert get_hardware_config(legacy)['Flops'] == engine-resolved canonical Flops post-F1/F2. Assert get_best_parallelization_strategy for Llama-2-7B on H100 returns the same strategy pre/post (H100 value unchanged at 989). REGRESSION CHECK: snapshot get_best_parallelization_strategy output for all 7 devices pre/post; assert unchanged for devices whose inline flops already equaled the DB, and changed-in-known-direction otherwise.

Unit: Inference efficiency calibration (the core optimism fix)

Approach. The original unit tried to fix GenZ's optimistic inference roofline two ways: (a) per-device/per-arch eta_compute/eta_mem DEFAULTS (C2/C3) applied to all 92 devices, and (b) removing dead shape-efficiency code (C4) plus a power model rework (M14) and a tf32 multiplier flip (L0). The critique is correct that the C3 eta_compute=0.80 default, the C2 eta_mem defaults, the M14 0.4125*TDP, and the tf32 flip are all un-anchored magic numbers, that the only measured device (GB10) end-to-end fits eta_compute≈0.27 (not 0.34, not 0.80), that the same (85,250) pair was re-labelled three ways, and that Op Intensity is consumed by budevolve. The corrected approach makes the per-hardware inference_calibration block (already in the codebase, already MEASURED-and-fit for GB10 by the differential_evolution harness) the SINGLE, generalizable mechanism: every numeric efficiency is a measured/fitted ratio against a documented grid, never a hand-typed default. Uncalibrated devices keep the honest, explicitly-documented eff=1.0 roofline (a known over-prediction we surface, not a fabricated correction). We remove the genuinely dead/double-counting shape code (C4) because the physical per-op roofline already encodes shape dependence (audit section 3 proves this), while PRESERVING the Op Intensity column semantics that budevolve reads. We DROP M14 and the tf32 flip from this unit as unsubstantiated and out of scope. We expand the GB10 grid and refit AFTER the structural FLOP fixes land (audit section 6.3 ordering), so no eta double-counts a structural error.

Net effect. Uncalibrated devices (91 of 92): DECODE latency unchanged (eff=1.0 kept). PREFILL/compute-bound latency DECREASES ~15% (=1/0.85) due to removing the fabricated flat tensor_core_eff (C4) — i.e. uncalibrated prefill gets slightly MORE optimistic, which is honest (we surface it as 'uncalibrated/optimistic' metadata) rather than papered over with an un-anchored 0.80 default. GB10 (the one measured device): after C4 removal + grid expansion + refit, prefill matches measured within ~10% RMS (vs -68% to -72% at today's eff*0.85 baseline) and decode within ~7% (vs -30% at eff=1.0). Memory unchanged everywhere. Power unchanged (M14 dropped). tf32 paths unchanged (L0 dropped). Net: the optimism is FIXED only where it is measured (GB10), and elsewhere the model stays at the honest, documented eff=1.0 roofline instead of acquiring 91 fabricated derating constants — fully compliant with HARD RULE 1.

Couplings. CRITICAL COUPLING (C3<->C4, same-change requirement): C4 removes the flat 0.85 tensor_core_eff that GB10's prefill eta_compute=0.096 was fit ON TOP OF. After C4, uncalibrated prefill is 1/0.85≈1.176x faster, and GB10's bare 0.096 makes its prefill ~15% faster, breaking the validated 5% RMS fit. Therefore C4 (dead-code removal) and the C3 GB10 refit MUST land in the SAME change, never separately. ORDERING (audit section 6.3, decisive): structural FLOP fixes (causal-mask attention, dense-peak FLOPS, C4 removal) land FIRST; THEN eta_{compute,mem}/t_launch are refit by differential_evolution against the corrected structure — because eta_compute ABSORBS the attention FLOP-counting convention (audit lines 100-106: full-S^2 -> -20%, causal-S^2/2 -> -31%), so fitting efficiency before structure double-counts. C2 and C3 share the same single mechanism (the inference_calibration block) and the same harness; neither introduces defaults, so they do not offset each other. M14 and L0 are DROPPED, so they no longer couple to anything in this unit (eliminating the double-counting risk between a model-level utilization scalar and power.py's per-op weights, and the tf32 multiplier/mem-multiplier inconsistency). Op Intensity coupling: C4's change to num_ops/num_data feeds budevolve/roofline_analyzer.py:65 -> OperatorInsight.arithmetic_intensity -> RooflineReport recommendations; this is a real downstream consumer and is handled by pinning the new semantics in a test, not by claiming no consumers.

C2-eta_mem-per-device-defaults

Root cause: Decode latency uses memory_efficiency=1.0 for all 91 uncalibrated devices. Audit section 6 (lines 83-88) measures that eff=1.0 over-predicts real decode throughput by 30% on GB10 (the one device with measured data). The original proposal tried to fix this with per-memory-technology eta_mem DEFAULTS (HBM2e/3 ~0.8-0.9, LPDDR5X ~0.75, GDDR ~0.7-0.8 from audit section 5 line 68-69), but (i) those ranges are qualitative literature bands not tied to a measured streaming-BW ratio for each SPECIFIC device, (ii) configs.py carries memory_type for only 37 of 92 devices (verified: grep -c memory_type = 37, name keys = 57 in the static block, 92 in DB) so 55 devices fall through to a generic band = the same flat-constant antipattern, and (iii) decode achievable MBU is also model/launch dependent (audit line 88: decode MBU 70% < copy MBU 76% because of per-layer launch/attention/sampling), which a static per-tech band cannot capture.
Solution: Do NOT invent per-device eta_mem defaults. Keep memory_efficiency=1.0 as the documented, honest baseline for uncalibrated hardware (it is a known over-prediction, surfaced in the config docstring and API response metadata, not a fabricated correction). The ONLY way a device gets eta_mem<1.0 is via its measured-and-fit inference_calibration.decode.eta_mem block, produced by validation/inference_calibration.py:fit_inference_calibration() against a measured grid in INFERENCE_BENCHMARKS. This is exactly how GB10's eta_mem=0.659 was obtained (configs.py:349, decode RMS ~6.6%, audit section 6.1 confirms 0.659 lands +7% vs measured and the measured copy-MBU ceiling 0.73-0.77 brackets it). Generalizability: the SAME closed-form least-squares procedure (inference_calibration.py:131-151) applies to any device once a measured grid is added; nothing is special-cased to GB10. The harness fits inv_eta_mem, t_launch, c_stream jointly so the launch/attention residual the critique flags is absorbed by t_launch+c_stream, not mis-attributed to eta_mem.
Constants & sources: (1) eta_mem=0.659 (GB10 decode): SOURCE = fitted by inference_calibration.py differential/lstsq against MEASURED llama.cpp grid (configs.py:349); cross-checked vs measured pure-copy MBU 0.73-0.77 (audit section 1, lines 9-16). NOT hand-typed. (2) memory_efficiency=1.0 for all uncalibrated devices: SOURCE = derivation/honest-baseline. This is the unmodified GenZ roofline, explicitly documented as optimistic (audit section 6 quantifies the GB10 gap at 30%). No fabricated number is introduced. (3) No per-memory-technology default bands are used at all — the audit section 5 0.75-0.9 literature ranges are explicitly NOT promoted to code defaults (this is the key correction vs the original proposal).
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py:153 -> UNCHANGED mechanism (meff defaults to passed value=1.0 unless cal['decode']['eta_mem'] present). Confirm no get_default_eta() helper is added; the original design's get_default_eta is REMOVED from scope. [2] llm-memory-calculator/src/llm_memory_calculator/hardware/configs.py:338-353 -> extend the GB10 docstring to state explicitly that uncalibrated devices use eff=1.0 (optimistic) BY DESIGN and list the calibration command. No new config keys. [3] llm-memory-calculator/src/llm_memory_calculator/validation/inference_calibration.py:66-77 -> add the EXPANDED GB10 grid (see C3) so the refit covers a larger-model regime; structure unchanged.
Impact: Decode latency for the 91 uncalibrated devices is UNCHANGED (byte-identical to today, eff=1.0). GB10 decode unchanged unless the grid is expanded and refit (then it moves by <=7% toward measured, in the proven-correct direction). serving.py decode path: unchanged numerically for uncalibrated devices. budevolve RooflineReport: unaffected (it reads per-op utilization/intensity, not the model-level eta_mem).
Regression risks: Lowest-risk option: nothing changes for uncalibrated devices, so no currently-passing test can flip. The only behavioral change is the documentation/metadata flag that a result is 'uncalibrated/optimistic'. Risk: downstream code that asserts a specific decode number for a NON-GB10 device would be unaffected; a GB10 decode assertion must be updated only if the grid is expanded+refit.
Verification: BEFORE: run python -m llm_memory_calculator.validation.inference_calibration GB10 on the CURRENT grid and confirm it reproduces eta_mem≈0.659, decode RMS<=7% (matches configs.py:349 and audit section 6.1). Regression assertion: for any device WITHOUT an inference_calibration block, estimate_decode_performance(...) returns the identical Latency before and after this change (assert equality, not approx). For GB10 after any grid expansion+refit, assert decode RMS over the measured grid stays <=10% and predicted/measured ratio is in [0.93,1.07] at ctx in {512,2048} (audit section 6.1).

C3-eta_compute-default-0.80

Root cause: Prefill latency uses compute_efficiency=1.0 (times the flat tensor_core_eff=0.85, see C4) for all uncalibrated devices, predicting prefill ~3x too fast on GB10 (audit section 6.2: -68% to -72% at the eff=1.0*0.85 baseline; achievable MFU ~27%). The ORIGINAL proposal fixed this with eta_compute=0.80 default for HBM datacenter GPUs and 0.70 for LPDDR/GDDR. The critique is CORRECT and I concede every point: (1) 0.80 has NO measured anchor — the only measured device GB10 fits eta_compute≈0.27 end-to-end (audit line 97-98), its pure-GEMM ceiling is 85/250=0.34 (audit line 37), neither is 0.80; the 0.75-0.85 cuBLAS band (audit line 71-72) is a qualitative literature range for large dense GEMMs vs TRUE dense peak, not a validated end-to-end prefill MFU vs CONFIGURED peak. (2) My old rationale that GB10's 250 label is 'FP4/sparse-flavored' is FALSE — configs.py:316-317 annotates Flops=250 as 'BF16 TFLOPS dense-tensor'; 0.096 is a small-model launch-heavy prefill FIT, not an MFU artifact. (3) The 0.70 LPDDR default re-labelled the same 250 as FP8 to manufacture 0.69 — circular. The same (85,250) pair cannot anchor 0.34, 0.69, 0.70 AND 0.80. (4) t_launch double-counting: GB10's fit SPLIT the launch-heavy regime into eta_compute=0.096 AND t_launch=0.206ms; a device given 0.80 with t_launch defaulting to 0.0 (utils.py:154) gets neither floor, so small-model prefill on uncalibrated HBM devices would be far TOO FAST — the opposite of the optimism fix.
Solution: Eliminate the eta_compute DEFAULT entirely. Like C2: the only mechanism that lowers prefill compute efficiency below 1.0 is a measured-and-fit inference_calibration.prefill block (eta_compute + eta_mem + t_launch jointly fit by differential_evolution over the production prefill path, inference_calibration.py:153-169). Uncalibrated devices keep eff=1.0 (documented as optimistic). For GB10, EXPAND the measured grid (it currently only has 0.5B/3B small models, which is exactly why the fit lands at the launch-heavy 0.096 + 206us split rather than a clean per-arch MFU) by adding larger-model prefill points where the GEMM regime dominates, then refit. This lets the fitter separate the true large-GEMM eta_compute (which should converge toward the measured 0.27-0.34 ceiling for big GEMMs) from the t_launch floor — instead of asserting a literature 0.80 that audit section 6.2 proves does NOT match measured end-to-end behaviour even on the one available device. CRITICAL ORDERING (audit section 6.3): the structural FLOP fixes (causal-mask attention, dense-peak FLOPS labelling, C4 dead-code removal) MUST land FIRST, then eta_{compute,mem}/t_launch are refit against the corrected structure — fitting efficiency before structure double-counts (audit line 100-106 proves eta_compute absorbs the attention FLOP-counting convention: full-S^2 -> -20%, causal-S^2/2 -> -31%).
Constants & sources: (1) eta_compute=0.096 (GB10 prefill, CURRENT): SOURCE = differential_evolution fit over the production prefill path against the MEASURED small-model grid (configs.py:352, prefill RMS ~5%, harness inference_calibration.py:165-169). It is a launch-heavy SMALL-MODEL fit, NOT an MFU default. Honestly labelled as such. (2) Measured GB10 achievable bf16 GEMM ceiling = 85 TFLOPS = 0.34 of configured 250 (audit section 2, lines 33-39) — used ONLY as a sanity bracket for the refit, never written into a config. (3) Measured GB10 end-to-end prefill achievable MFU ≈ 0.27 (audit lines 97-98) — the refit target bracket, fit by the harness, not hardcoded. (4) t_launch=0.206ms (GB10 prefill): SOURCE = fit; order-of-magnitude cross-checked vs measured 8.26us raw 64^2-matmul launch floor (audit section 4, line 63). Per-op launch 5-10us empirically justified. (5) NO eta_compute default (0.80/0.70/0.34/0.69) is written anywhere — this is the central correction. Uncalibrated eta_compute stays 1.0 (honest optimistic baseline).
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py:152 -> UNCHANGED (ceff defaults to passed 1.0 unless cal['prefill']['eta_compute'] present). The original design's eta_compute default lookup is REMOVED. [2] llm-memory-calculator/src/llm_memory_calculator/validation/inference_calibration.py:66-77 -> ADD larger-model prefill benchmark points to the GB10 grid (e.g. a 7B/14B qwen prefill TTFT at ctx in {512,2048,8192}) so the fit separates large-GEMM eta_compute from the launch floor. Bench rows only; fitter unchanged. [3] llm-memory-calculator/src/llm_memory_calculator/hardware/configs.py:346-353 -> after refit, replace GB10 prefill block with the harness output (human-reviewed per the module docstring line 17). Only GB10 changes. [4] DO NOT modify utils.py to add per-arch prefill eta_compute defaults.
Impact: Prefill latency for the 91 uncalibrated devices: UNCHANGED relative to today (eff=1.0) ONCE C4's flat 0.85 is removed — see C4 coupling: today uncalibrated prefill is divided by (1.0*0.85); after C4 removal it is divided by 1.0, making uncalibrated prefill ~15% FASTER. That 15% must be acknowledged (the critique's missed-regression point). GB10 prefill changes by the refit. serving.py prefill_latency_ms flows into total_latency and the power estimate. budevolve RooflineReport per-op compute time changes by removing the 0.85 (see C4).
Regression risks: (1) The 0.85->removal makes ALL uncalibrated-device prefill 1/0.85≈1.18x faster — a known-direction, known-magnitude change (≈+15% throughput) that is PHYSICALLY CORRECT (the 0.85 was a fabricated flat derating with no per-device basis; audit section 3 shows the roofline already encodes shape). Any test asserting an absolute uncalibrated prefill latency will change by this factor and must be updated. (2) GB10 prefill: removing the 0.85 means the OLD 0.096 was fit on top of 0.85, so the effective old factor was 0.096*0.85≈0.0816; after C4 the bare 0.096 makes GB10 prefill ~15% faster, BREAKING the validated 5% RMS fit UNTIL refit. Therefore C4 removal and GB10 prefill refit MUST land in the same change.
Verification: BEFORE: with C4 dead code removed, run the harness on the EXPANDED GB10 grid and confirm prefill eta_compute lands in the measured bracket [0.20,0.40] (audit: 0.27 end-to-end fit, 0.34 pure-GEMM ceiling) and prefill RMS<=10%. Assert the fitted eta_compute is NOT pinned to a bound (i.e. the data, not the prior, determines it). Regression: for any uncalibrated device, assert prefill latency changes by exactly 1/0.85 (=1.1765x faster) relative to pre-change — a deterministic, explainable shift, not arbitrary. For GB10, assert predicted/measured TTFT ratio in [0.90,1.10] at ctx in {512,2048,8192} after refit (audit section 6.2 grid).

C4-remove-dead-shape-efficiency-code

Root cause: operator_base.py:436-469 get_roofline applies TWO ad-hoc shape corrections on top of the physical roofline: (1) get_tensor_core_efficiency_factor() (operator_base.py:397-434) returns a flat 0.85 for almost all ops (the GEMM/Attend branches require get_dimensions() to be a tuple of len>=3, but get_dimensions() returns self.get_tensors() = a list of tensor lists, so isinstance(...,tuple) is FALSE for real ops -> the 0.85 default at line 434 is what actually runs; the 0.4-0.95 shape math is dead). compute_time is divided by (system.compute_efficiency * 0.85) at line 468-469. (2) get_shape_dependent_op_intensity() (operator_base.py:110-165) overrides the Op Intensity reported in the roofline dict (line 454). The critique correctly notes get_tensor_core_efficiency_factor mixes a fabricated 0.85 derating into EVERY device's compute time with no per-device/per-measurement basis, and audit section 3 (lines 41-54) proves the per-op max(compute_time, memory_time) roofline ALREADY encodes shape dependence (small-M -> memory bound, large-M -> compute bound emerges automatically from per-op arithmetic intensity).
Solution: Remove the redundant compute-time derating: delete the tensor_core_eff multiplication in get_roofline (operator_base.py:465-470) so effective_compute_efficiency = system.compute_efficiency only. Per HARD RULE 5, the physical roofline + the measured/fit inference_calibration eta_compute already cover what 0.85 pretended to model, without a fabricated constant. HOWEVER (critique's missed-regression): the 'Op Intensity' field at operator_base.py:454/505 is NOT display-only — budevolve/roofline_analyzer.py:65 reads it into OperatorInsight.arithmetic_intensity. So do NOT silently change the reported Op Intensity. The shape-dependent op-intensity and the bare num_ops/num_data give DIFFERENT values for GEMM/Logit/Attend, and budevolve's RooflineReport/serving config recommendations depend on it. Decision: KEEP the Op Intensity semantics that budevolve consumes UNCHANGED (revert operator_base.py:452-460 to the simple num_ops/num_data ratio ONLY IF we also confirm budevolve's expected intensity; the safer, audit-grounded choice is to compute the TRUE roofline intensity = num_ops/num_data consistently, because that is the quantity the ridge-point analysis in audit section 3 uses: ridge at AI=eff_peak/eff_BW). Concretely: (a) remove tensor_core_eff entirely from the compute path (dead+fabricated); (b) make Op Intensity = num_ops/num_data (the physically-correct flop/byte the roofline itself uses for boundedness), and update budevolve/roofline_analyzer to consume that same definition; (c) keep get_shape_dependent_op_intensity and get_tensor_core_efficiency functions only if any other caller exists (grep shows none in apis/src; the only consumer is operator_base itself + budevolve via the column) -> delete the now-unused helpers.
Constants & sources: (1) 0.85 flat tensor_core_eff: SOURCE = NONE (fabricated). This is the magic number being REMOVED. Its dead shape branches (0.4-0.95, typical_sms=108, wave_penalty=0.15) are also unsourced and unreachable -> removed. (2) Op Intensity = num_ops/num_data: SOURCE = closed-form definition of arithmetic intensity (flop/byte), the exact x-axis of the roofline validated in audit section 3 (predicted vs measured throughput within a few % at AI=1..2048).
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:464-470 -> remove tensor_core_eff usage; set effective_compute_efficiency = system.compute_efficiency; compute_time /= effective_compute_efficiency. [2] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:452-460 -> set op_intensity = num_ops/num_data if num_data else 0 (remove get_shape_dependent_op_intensity call) so the reported flop/byte is the physical roofline intensity. [3] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:509 -> remove the 'TC Effcy' column (now always 1.0/meaningless) OR keep it pinned to 1.0 for schema stability; prefer removal after confirming no reader. [4] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:60-165,397-434 -> delete get_tensor_core_efficiency, get_shape_dependent_op_intensity, get_tensor_core_efficiency_factor and OPERATION_CHARACTERISTICS once grep confirms no remaining caller. [5] llm-memory-calculator/src/llm_memory_calculator/budevolve/roofline_analyzer.py:65 -> no code change needed (still reads 'Op Intensity'); add a test pinning the new num_ops/num_data semantics so the report's arithmetic_intensity is documented.
Impact: (1) Every device's prefill/decode compute_time becomes 1/0.85≈1.176x SMALLER (compute-bound ops faster). Coupled: prefill_moddeling/decode_moddeling latency, serving.py latencies+power, budevolve per-op Latency and Bound classification (boundedness uses com_to_mem_ratio = compute_time/memory_time, so removing 0.85 LOWERS com_to_mem_ratio -> a few borderline ops near ratio=1 may flip Compute->Memory; this is correct, since 0.85 was artificially inflating compute_time). (2) budevolve OperatorInsight.arithmetic_intensity changes for GEMM/Logit/Attend from the shape-formula value to num_ops/num_data; RooflineReport recommendations shift accordingly. This is the critique's correctly-identified consumer (roofline_analyzer.py:65,80,191) — addressed by pinning the new semantics in a test, not by claiming 'no consumers'.
Regression risks: (1) Uncalibrated-device prefill/decode compute-bound latency drops ~15% (known direction/magnitude). (2) GB10 fit breaks until refit (see C3): the old eta_compute=0.096 was on top of 0.85; must refit in the same change. (3) Any test asserting 'TC Effcy' or a shape-dependent Op Intensity value will fail — update to the num_ops/num_data definition. (4) budevolve recommendation tests that depend on arithmetic_intensity may change selection; re-baseline against the physical definition.
Verification: BEFORE: assert get_dimensions() returns a list (not tuple) for a sample FC/GEMM op, proving the 0.85 branch is what runs today (confirms the code IS dead shape math). Reproduce the GB10 GEMM sweep (audit section 2) with the bare roofline (eff_peak=85 TFLOPS via eta_compute, eff_BW=205 via eta_mem) and confirm predicted throughput matches the measured M=1..4096 table within the audit's stated few-% (validates that removing 0.85 + using num_ops/num_data intensity reproduces the measured roofline). Regression: snapshot budevolve RooflineReport.arithmetic_intensity for one model before/after and assert the change equals (num_ops/num_data) vs the old shape formula for each op type (documented delta, not surprise).

M14-power-model-rework-DROPPED

Root cause: The original M14 proposed replacing serving.py:161-162 fallback 0.5 with a 0.4125TDP figure and feeding a model-level compute_utilization/memory_utilization scalar into PowerModel. The critique is CORRECT that this is unsubstantiated and physically confused: (1) 0.4125TDP is never traced — serving.py:161-162 actually falls back to 0.5 for BOTH utils because getattr(prefill_result,'compute_utilization',0.5) ALWAYS hits the 0.5 default (verified: compute_utilization/memory_utilization are set only on budevolve RooflineReport and genz/simulation/results.py, NEVER on ModdelingOutput — grep confirms). (2) The per-op 'Compute Utilization' column (operator_base.py:522) is compute_time/exec_time, a per-op TIME-FRACTION, not a device MFU/occupancy; power.py:25-28 ALREADY multiplies per-op power by power_breakdown weights, so a model-level scalar path would double-handle utilization. (3) audit section 7 (lines 108-113) shows real GB10 power is P_idle + util*(P_max-P_idle) with measured idle≈7.2W, max≈67-73W — which REFUTES a fixed 0.4125*TDP. (4) CPU path (serving.py:139) is not addressed.
Solution: REMOVE M14 from this unit entirely (out of scope and unsubstantiated). Power calibration is a SEPARATE unit that must (a) define a single, sourced utilization quantity, (b) source P_idle and P_max per device from vendor idle/TDP (GB10 measured idle 7.2W / TDP 170W per configs.py:336 give an anchor), and (c) avoid double-counting with power.py per-op weights. Do not touch serving.py:161-167 in this unit. The only defensible interim note: the current 0.5/0.5 fallback and the 0.4125 are both un-anchored; flag them for the power unit. This unit's scope is strictly latency efficiency (eta_compute/eta_mem/t_launch).
Constants & sources: (1) 0.4125*TDP: SOURCE = NONE (asserted, untraced) -> REJECTED, not used. (2) 0.5 fallback (serving.py:161-162): SOURCE = NONE -> left as-is (pre-existing), flagged for the power unit, not changed here. (3) GB10 measured idle=7.2W, GEMM-max=67-73W, util-35%=44W (audit section 7): SOURCE = measured -> handed to the power unit as the anchor; NOT consumed in this latency unit.
Code changes: [1] NO changes to BudSimulator/apis/routers/serving.py in this unit. [2] NO changes to llm-memory-calculator/src/llm_memory_calculator/genz/power.py in this unit. [3] Documentation: record in solutions.md that M14 is descoped to a dedicated power-calibration unit with the audit section 7 anchors.
Impact: By dropping M14, no power computation changes; the serving power estimate is byte-identical to today. This avoids the double-counting (per-op power.py weights vs a model-level scalar) and the wrong-quantity aggregation the critique identified.
Regression risks: None introduced (no change). The pre-existing 0.5 fallback inaccuracy remains, openly flagged rather than replaced by another unsourced number.
Verification: Assert serving.py power output is unchanged before/after this unit (no diff). Confirm via grep that ModdelingOutput defines no compute_utilization/memory_utilization attribute (so the existing getattr falls to 0.5) — documenting the current behaviour for the power unit's design.

L0-tf32-compute-multiplier-DROPPED

Root cause: The original L0 proposed flipping System.compute_multiplier['tf32'] from 1 to 2 (system.py:24). The critique correctly notes this doubles compute_TIME for any path running bits='tf32' (op_per_sec is fixed; get_bit_multiplier type C returns the multiplier and compute_time scales with it, operator_base.py:238), halving tf32 throughput, with mem_multiplier['tf32']=4 (system.py:56) left inconsistent, and no enumeration of which models/tests/precision paths use tf32. This is a cross-cutting precision change with unlisted couplings and no measured anchor in this repo.
Solution: REMOVE L0 from this unit. The tf32 compute_multiplier is a precision-throughput model question, not part of the inference efficiency calibration, and changing it has no measured anchor here (the only measured device GB10 was fit at bits='int8', inference_calibration.py:113). If tf32 peak modelling is wrong it belongs in a precision-model unit with a vendor spec (e.g. A100 TF32 = 156 TFLOPS = half of BF16 312, which WOULD justify multiplier 2) and an explicit consistency pass over mem_multiplier and all tf32 test paths. Not in scope here, and critically: the calibration block is precision-coupled (eta fit at one bits-value does not transfer to another, critique's generalizability point) — so any precision-model change must be coordinated with per-precision calibration, which is future work, not this unit.
Constants & sources: (1) compute_multiplier['tf32']=1 (current, system.py:24): SOURCE = existing; left UNCHANGED in this unit. (2) A100 TF32 156 TFLOPS vs BF16 312 (would imply multiplier 2): SOURCE = NVIDIA A100 datasheet — cited only to show WHERE a future tf32 fix would be anchored; NOT applied here.
Code changes: [1] NO change to llm-memory-calculator/src/llm_memory_calculator/genz/system.py compute_multiplier/mem_multiplier in this unit. [2] Documentation: note in solutions.md that tf32 peak modelling + per-precision calibration is a separate unit; record that the calibration block should be keyed by bits in that future work.
Impact: No change to any tf32 path; training_operators and any tf32 calibration grid are unaffected. Avoids the unlisted cross-cutting regression the critique flagged.
Regression risks: None (no change).
Verification: Assert system.compute_multiplier['tf32']==1 unchanged. Document the precision-coupling limitation: a device's inference_calibration eta is valid only at the bits it was fit at (GB10: int8, inference_calibration.py:113); add an assertion/warning if a calibration block is reused at a different precision (future unit).

Unit: causal-masking-prefill-attention

Approach. Prefill Logit/Attend currently count the FULL S×S (per shard M×N) attention rectangle, but a causal decoder only computes the lower triangle. The correct structural factor is the exact triangle-count ratio (a closed-form derivation, not a fitted constant). The prior proposal had a hard correctness bug: under sequence parallelism the square-prefill op is built with M=input_sequence_length (full logical query length) and N=input_sequence_length//sp (a KEY SHARD, not a causal key budget), so the naive f=1-(M-1)/(2N) goes NEGATIVE for sp>1 (verified: S=4096, sp=4 -> f=-1.0). The revised design (a) derives the factor from the LOGICAL causal geometry per builder, never from raw confounded M,N, and clamps to [0,1] defensively; (b) carries causality as dedicated prefill OpTypes (Logit_PREFILL/Attend_PREFILL) emitted ONLY by causal decoder-LM square/MLA/local prefill builders, so analyse_model gets the causal signal from op_type (it has no model_config.is_decoder), bidirectional/encoder/vision prefill keeps plain Logit/Attend (f=1), and the chunked builder uses its own valid 1-(M-1)/(2N) (M<=N there); (c) scales num_ops AND op_intensity/boundedness consistently in the operator; (d) deletes the dead get_tensor_core_efficiency / get_shape_dependent_op_intensity blocks (proven dead for attention) per the GB10 audit rather than layering the causal factor under unsourced magic constants. Structural FLOP fix lands BEFORE eta_compute is re-fit (audit plan-of-record).

Net effect. Prefill: causal scaling removes ~50% of attention Logit/Attend FLOPs at long context (attention ~56% of prefill compute there) -> attention compute_time ~0.5x, total prefill compute_time down ~0.7-0.8x at 8k+ for causal decoder LMs (smaller at short context where attention share is low). Combined with deleting the flat 0.85 TC multiply (compute_time x1.18 on all compute-bound ops), the NET pre-refit prefill TTFT moves toward the audit-documented -31% (from -20%) gap at ctx=8192, i.e. predicted prefill gets ~13% SLOWER net at long context because the removed-double-count and the 0.85 removal both push the structure toward the measured ~27% MFU that eta_compute will then capture. Decode: unaffected by causal factor (BM_PREFILL/suffix keep OpTypes), but the 0.85 removal slightly speeds decode compute_time (decode is memory-bound so near-zero net). Memory/KV/params: unchanged (canonical raw num_ops untouched). After eta_compute refit on corrected structure, both phases land within the audit's +-7% (decode) / structural-grounded (prefill) targets.

Couplings. HARD ORDERING with the GB10 efficiency calibration unit (eta_compute/eta_mem/t_launch): audit_microbench_data.md lines 100-106 prove eta_compute ABSORBS the attention FLOP-counting convention (ctx=8192: full+eta=0.34 -> -20%; causal+eta=0.34 -> -31%). Therefore this structural causal fix (and the C1 dense-FLOPS fix and the C4/L3 dead-code removal it bundles) MUST land BEFORE eta_compute is fit by the differential_evolution harness; fitting efficiency first, or fitting one structural piece in isolation, double-counts. COUPLED with the dead-code-removal unit: this fix DELETES get_tensor_core_efficiency / get_shape_dependent_op_intensity / get_tensor_core_efficiency_factor (the flat 0.85 multiply and the dead-for-attention intensity), so any unit that assumed those exist must be revised; removing the 0.85 multiply changes every op's compute_time by 1/0.85 and is only valid jointly with the eta_compute refit. COUPLED with sliding-window/SWA attention unit: the exact windowed causal count (R4) is deferred to that unit; until then *_local uses the capped-N approximation which is an upper-bound-safe under-count vs today. OFFSETTING: causal halving (compute_time x~0.5) and TC-0.85 removal (x1/0.85=1.18) partially offset on attention ops; net attention compute_time change ~0.59x at long context before refit.

M4b-causal-prefill

Root cause: mha_flash_attention_prefill (attention.py:23,26), mla_attention_prefill (255,258), mha_flash_attention_prefill_local (54,55) emit Logit/Attend whose get_num_ops = prod(B,H,M,N,D) counts the full M*N score matrix. For a causal decoder, query at position i attends only to keys 0..i, so only the lower triangle is computed. Counting the full rectangle overstates attention FLOPs by ~2x in long-context prefill (where attention is ~56% of prefill compute), making TTFT too fast. Verified the live dim layout: after save_layers (8-col CSV ['Name','M','N','D','H','Z','Z','T']) + analyse_model np.insert(batch_size at idx1), the operator self.dim = [B,H,M,N,D,Hkv,Residency,OpType] (indices 0..5 are B,H,M,N,D,Hkv). For square prefill M=input_sequence_length (full Q), N=input_sequence_length//sp (K SHARD). For chunked prefill (attention.py:161) M=chunk, N=past+chunk.
Solution: Apply the EXACT closed-form causal triangle-count ratio f to num_ops and op_intensity for causal prefill attention only. Two derivations, selected by which builder emitted the op (carried via dedicated OpType), never inferred from raw M,N:

(A) Square/full prefill (mha_flash_attention_prefill, mla_attention_prefill, _local with sw>=S): the LOGICAL attention is S queries x S keys, M=S is the full query length. Causal lower-triangle count = S(S+1)/2 out of S^2 -> f = (S+1)/(2S) = (M+1)/(2M). Under sequence parallelism each of sp ranks holds S/sp keys and the per-rank op counts M(S/sp); the per-rank causal share is [S(S+1)/2]/sp, so the per-op ratio is still ((S(S+1)/2)/sp)/(S*(S/sp)) = (S+1)/(2S) = (M+1)/(2M). It depends ONLY on M (the logical query length), is shard-invariant, and lies in (0.5, 1] for all M>=1 -> NEVER negative. lim_{M->inf}=1/2 (asymptotic anchor; pure derivation).

(B) Chunked prefill (mha_flash_attention_chunked:161-162): the query block is the latest chunk c=M attending over past+c=N keys; query local-index i is at absolute position (N-M)+i and sees (N-M)+i+1 keys. Causal count = sum_{i=0}^{M-1}((N-M)+i+1) = M(N-M) + M(M+1)/2; full = M*N. f = ((N-M)+(M+1)/2)/N = 1 - (M-1)/(2N). Valid because here M<=N always (chunk <= past+chunk) -> f in (0,1].

Mechanism: add OpType.Logit_PREFILL=41 / OpType.Attend_PREFILL=42 mapped to subclasses LogitPrefill(Logit) / AttendPrefill(Attend). The causal-square builders emit these instead of Logit/Attend; the subclasses override get_effective_num_ops() to return self.get_num_ops()f_square where f_square = (M+1)/(2M) computed from self.dim[2] (=M), max(.,0.5)-clamped, min(.,1.0)-clamped. The chunked builder keeps OpType.Logit/Attend but multiplies via the same subclass selecting formula (B) when emitted from the chunked path; concretely the chunked builder also emits Logit_PREFILL/Attend_PREFILL and the subclass detects the chunked regime by M<N (square is M>=N because N=S/sp<=S=M) -> use formula (B) 1-(M-1)/(2N) when M<N else formula (A). Both branches clamp to [0,1]. Encoder/bidirectional/vision prefill (is_decoder False) keeps plain Logit/Attend -> f=1 (unchanged). Decode BM_PREFILL ops have M=1 -> (1+1)/2/1=1 and would be untouched anyway since they keep their existing OpTypes. To keep compute_time, op_intensity, boundedness and reported throughput CONSISTENT, override the intensity in get_roofline by reading the SAME f: since get_shape_dependent_op_intensity is proven dead for attention (see regression notes), remove it and compute op_intensity = effective_num_ops*2/num_data using the scaled effective ops, so AI and compute_time agree by construction.

Constants & sources: (1) f_square = (M+1)/(2M): closed-form count of strictly-lower+diagonal triangle entries (M(M+1)/2) divided by full M^2. Source: (d) derivation. No fitted value. (2) f_chunked = 1-(M-1)/(2N): closed-form causal count M(N-M)+M(M+1)/2 over M*N for a query block of M tokens ending at absolute position N. Source: (d) derivation. (3) asymptotic 0.5 = lim_{M->inf}(M+1)/(2M): derivation, used only as a defensive lower clamp, not a magic operating value. (4) clamp [0.5,1.0] for square / [0,1] for chunked: derivation-bounded ranges (the exact ratios provably live there); the clamp is defensive against degenerate dims (M=0), not a tuned parameter. (5) NO use of get_tensor_core_efficiency constants (typical_sms=108, 0.15, 1e9, 1.1, 0.4/0.95): those unsourced guesses are DELETED, not relied upon (audit_microbench_data.md lines 49-54 prove the roofline already encodes shape via arithmetic intensity).
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/Models/utils.py:13 (after Attend_BM_PREFILL=10) -> add Logit_PREFILL=41, Attend_PREFILL=42 to OpType [2] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:12-14 -> add 41:'Logit', 42:'Attend' to op_type_dicts (so breakdown/boundedness treat them as attention; the causal scaling lives in the subclass, not the name map) [3] llm-memory-calculator/src/llm_memory_calculator/genz/operators.py:161-191 -> add classes LogitPrefill(Logit) and AttendPrefill(Attend) overriding get_effective_num_ops(self, system=None): read M=self.dim[2], N=self.dim[3]; f = (1 - (M-1)/(2N)) if (M < N and N>0) else ((M+1)/(2M) if M>0 else 1.0); f = max(0.0, min(1.0, f)); return self.get_num_ops()*f [4] llm-memory-calculator/src/llm_memory_calculator/genz/analyse_model.py:215 -> getattr(operators, op_type) already resolves 'Logit'/'Attend' via name map; add explicit class dispatch for OpType 41/42 to LogitPrefill/AttendPrefill (e.g. a small {41:operators.LogitPrefill,42:operators.AttendPrefill} override keyed on dim[-1] before the getattr fallback) [5] llm-memory-calculator/src/llm_memory_calculator/genz/Models/attention.py:23,26 -> OpType.Logit->OpType.Logit_PREFILL, OpType.Attend->OpType.Attend_PREFILL (causal square prefill) [6] llm-memory-calculator/src/llm_memory_calculator/genz/Models/attention.py:54,55 -> same for *_local (sliding-window prefill); see regression note on SWA [7] llm-memory-calculator/src/llm_memory_calculator/genz/Models/attention.py:161,162 -> chunked Logit Pre/Attend Pre -> Logit_PREFILL/Attend_PREFILL (formula B applies, M<N) [8] llm-memory-calculator/src/llm_memory_calculator/genz/Models/attention.py:255,258 -> MLA prefill Logit/Attend -> Logit_PREFILL/Attend_PREFILL [9] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:60-165 -> DELETE get_tensor_core_efficiency and get_shape_dependent_op_intensity (proven dead for attention: get_dimensions() returns a 3-tuple of tensor shapes so both functions' len>=6/len>=4 guards fail and they return constant fallbacks 100 / 0.85) [10] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:397-434 -> DELETE get_tensor_core_efficiency_factor [11] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:445-470 -> in get_roofline: num_ops=self.get_effective_num_ops(system)*2; op_intensity = num_ops/num_data if num_data else 0 (single source of truth); drop tensor_core_eff multiply so effective_compute_efficiency = system.compute_efficiency only; drop 'TC Effcy' from ret (line 509)
Impact: COUPLED COMPUTATIONS THAT CHANGE: (1) Prefill attention compute_time drops by f (~0.5 at long context) for causal decoder LMs only. (2) op_intensity for these ops now equals scaled-ops/data and is internally consistent with compute_time; boundedness classification (operator_base.py:490) and the ridge crossover shift accordingly (this is the intended AI fix, audit line 51). (3) reported attention Throughput (operator_base.py:480) = scaled_ops/exec_time: rises by 1/f ONLY in the compute-bound regime; at short context where attention may be memory/launch-bound exec_time is set by memory_time so throughput does NOT scale reciprocally (correctly). (4) Deleting the TC-efficiency multiply changes ALL ops' compute_time by removing the flat 0.85 (and the broken-for-GEMM shape path) -> GEMMs get faster by 1/0.85; this is intentional and MUST be absorbed by re-fitting eta_compute against corrected structure (audit lines 100-106 plan-of-record: structure first, then differential_evolution fit). (5) get_model_characterstics 'Num ops' (operator_base.py:379) intentionally keeps get_num_ops() (RAW full count) as the canonical FLOP-count for memory/param/density accounting; the latency path uses get_effective_num_ops (scaled). This is the SAME raw-vs-effective split that already exists everywhere (get_compute_time uses effective, characteristics uses raw) -> documented as canonical: raw=hardware-agnostic workload size, effective=what the device executes. (6) MoE/FFN, embedding, norm ops: unchanged. (7) Encoder/bidirectional models: unchanged (keep plain Logit/Attend).
Regression risks: (R1) SP>1 SIGN BUG (the prior fatal flaw): mitigated by formula selection M(B) else (A) plus hard clamp [0,1]; square-prefill sp>1 has M=S>N=S/sp -> branch (A) -> (S+1)/(2S) in (0.5,1], never negative. (R2) If any future builder emits a causal op with M>N that is NOT the logical-square case, formula (A) would be used; clamp keeps it in range but the count could be wrong -> guarded by emitting Logit_PREFILL ONLY from the four audited causal builders. (R3) Bidirectional models wrongly halved: avoided because analyse_model has no causality info, so causality is decided at the builder (which knows is_decoder/attention type) NOT in analyse_model -> bidirectional keeps OpType.Logit/Attend, f=1. (R4) Sliding-window over-credit: for SWA prefill with sw<S, formula (A) over-credits early tokens (a query at pos i<sw still attends i+1 keys, but full count is min(sw, i+1)); the local builder already caps N=kv_len=min(sw,S) (attention.py:51,54), so applying (A) on M=S with capped N is an approximation. Mitigation: for _local emit Logit_PREFILL but use an exact windowed count f_sw computed in the builder where sw is known, threaded as the square-vs-window choice; documented as a follow-up refinement, and NOT worse than today (today counts full Skv_len, the fix counts <= that). (R5) Deleting TC/shape-intensity could change non-attention results that currently 'happen to' look calibrated against the 0.85 -> this is exactly why eta_compute must be re-fit after; pre-fit numbers for any device not yet recalibrated will shift and must be re-run through the calibration harness. (R6) get_model_characterstics consumers reading 'Num ops' see UNCHANGED full counts by design (canonical workload size); CSV exports/audit reproductions documented accordingly.
Verification: BEFORE applying: (V1) Unit test on the factor across regimes and clamp: for square (M==N, sp=1) assert eff/full == (M+1)/(2M) for M in {1,2,128,4096,32768}; for sp in {1,2,4,8} build the real row (M=S, N=S//sp) and assert 0<=f<=1 AND f==(S+1)/(2S) (shard-invariant); for chunked (M=chunk<N=past+chunk) assert f==1-(M-1)/(2N) and 0<=f<=1; explicit per-row triangle recount for S in {4,8} (enumerate sum_{i}(min(i+1,N))). (V2) Integration: run prefill_moddeling(model='llama2_7b', input_tokens=S) for S in {512,2048,8192} with tensor_parallel/sequence_parallel in {1, 4} (MUST exercise sp>1) and assert finite, positive compute_time and that attention compute_time(sp=4) ~ compute_time(sp=1)/4 within rounding (sharding) and that causal attention FLOPs == 0.5..~1.0x of pre-fix full-rectangle FLOPs (never >full, never <0). (V3) GB10 keystone: re-run the 1.498B Llama prefill grid from audit_microbench_data.md section 6; assert switching full->causal moves ctx=8192 prediction from -20% to about -31% vs measured BEFORE eta refit (the documented coupling magnitude, audit line 101) -> confirms the structural factor is ~0.5 at long context, isolating it from eta. REGRESSION CHECK (must hold): bidirectional/encoder model prefill attention FLOPs UNCHANGED (assert plain Logit/Attend, f=1); decode path FLOPs UNCHANGED (BM_PREFILL/Logit_suf keep their OpTypes); get_model_characterstics 'Num ops' UNCHANGED (raw count canonical); op_intensity for attention now equals effective_ops*2/data (assert == compute_time-consistent value, NOT the old dead constant 100); no op returns 'TC Effcy' key.

Unit: Parameter counter GLU/activation detection

Approach. The critique is correct on every substantive point and I adopt it. The original bug is real and narrowly scoped: in the verified regression panel only Gemma2-9B is wrong (7.08B vs published 9.24B), because (a) Gemma stores its activation under hidden_activation (not hidden_act/activation_function, the only keys _calculate_ffn_params reads) and (b) its value gelu_pytorch_tanh is not in glu_variants, so its gated geglu FFN is counted as 2-matrix instead of 3-matrix. The previously-proposed fix (hand-adding phi/starcoder2/nemotron to a gated allowlist and adding gelu_new/gelu_approx/gelu_pytorch_tanh to glu_variants) is rejected: I reproduced that it over-counts Phi-2 by +0.84B/+30% and StarCoder2 by one matrix/layer, and gelu_pytorch_tanh fires on StarCoder2 (non-gated) and Gemma (gated) identically so it cannot discriminate. The replacement is a single structural detector is_gated_ffn(config) keyed off (1) unambiguous gated activation spellings that only ever denote a gate*up MLP, then (2) a checkable architecture/family registry derived 1:1 from each model's HF modeling*.py MLP class (gate_proj present => gated), then (3) a conservative ambiguous-activation fallback (plain silu/swish on an unknown family => gated, else non-gated). The detector reads all activation-key variants itself so it works on multimodal/hybrid sub-configs that bypass normalize_config, and is shared by both _calculate_ffn_params and _calculate_moe_params. The hidden_activation->hidden_act alias is added to the normalizer as defense-in-depth. The out-of-scope M4 tie-embeddings edits are dropped (Gemma tie handling is already correct at 9.24B tied). Verified: full panel of 10 models (7 gated incl. Gemma fixed + Mixtral robustified, 3 non-gated anchors) all within 0.3%.

Net effect. Gemma-family dense models: parameter_count and weight-memory rise ~~+30% to their correct published values (Gemma2-9B 7.08B->9.24B); their prefill/decode FFN FLOPs rise correspondingly (~~+50% of FFN compute, since 2->3 matrices). Mixtral and other MoE: no change when activation present; becomes robust (no spurious -32% drop) when activation absent. All non-gated families (GPT2/Phi/StarCoder2/OPT/GPT-NeoX/Falcon/Nemotron) and all already-correct gated families (Llama/Mistral/Qwen): zero change (verified <=0.3%). Net: one corrected over/under-count on Gemma, removal of two fragile string proxies, no regressions.

Couplings. PC-GLU-1 and PC-GLU-2 are tightly coupled: the normalizer alias (GLU-1B) is only meaningful because the detector (GLU-2) reads hidden_activation directly; if you applied only the alias it would still miss multimodal/hybrid sub-configs (which bypass normalize_config), and if you applied only the detector the rest of the code reading hidden_act would be inconsistent for Gemma. Apply both. The MoE branch shares _is_gated_ffn, so GLU-2 simultaneously fixes the dense Gemma case and makes the Mixtral MoE expert count robust to a missing activation key (offsetting a latent 46.7B->31.67B failure that the activation-string heuristic could trigger). This unit is upstream of any FLOP/efficiency calibration in audit_microbench_data.md: parameter count feeds weight-memory and FFN FLOPs, so per the audit's stated rollout order, structural FLOP/param fixes (this unit) must land BEFORE eta_compute/eta_mem are re-fit by the differential_evolution harness, or the calibration absorbs a param-count error. No numeric efficiency constant is introduced here.

PC-GLU-1

Root cause: _calculate_ffn_params (parameter_counter.py:280) reads activation only from 'activation_function' or 'hidden_act'. Gemma/Gemma2/Gemma3/CodeGemma/PaliGemma store the activation under 'hidden_activation'. So the lookup returns the default 'gelu', missing the gated FFN. ConfigNormalizer.normalize_config has no hidden_activation alias either, and even if it did, multimodal/hybrid sub-configs (lines 434-468, 404-414) call _calculate_transformer_params directly and never re-normalize, so an outer-layer alias would not reach the Gemma3/PaliGemma text tower.
Solution: (A) Add a structural detector _is_gated_ffn(self, config) (see PC-GLU-2) that itself reads hidden_act, hidden_activation, and activation_function, so the activation is found regardless of which path constructed the config. (B) As defense-in-depth, add to ConfigNormalizer.normalize_config a non-destructive alias: if 'hidden_act' not in config and 'hidden_activation' in config, set normalized['hidden_act']=config['hidden_activation']. The detector (A) is the load-bearing fix; the alias (B) keeps the rest of the code that reads hidden_act consistent. No reliance on the alias reaching sub-configs because the detector reads hidden_activation directly.
Constants & sources: (1) activation key names hidden_act / hidden_activation / activation_function: SOURCE = HuggingFace PretrainedConfig field names (transformers config schema); hidden_activation is the documented Gemma key (transformers/models/gemma2/configuration_gemma2.py). Not fitted.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/config_normalizer.py:~117 (after the intermediate_size normalization block, before quantization parse) -> add: if 'hidden_act' not in normalized and 'hidden_activation' in config: normalized['hidden_act']=config['hidden_activation'] [2] llm-memory-calculator/src/llm_memory_calculator/parameter_counter.py:280 -> remove the local act_fn lookup; replace the GLU branch condition with self._is_gated_ffn(config)
Impact: Changes the FFN matrix multiplier (2 vs 3) for any config whose activation lived only under hidden_activation. In the verified panel this affects ONLY Gemma-family models (Gemma2-9B 7.08B->9.24B). Couples to: total parameter_count (the displayed field), weights-memory estimate, and any prefill/decode FLOP that scales with FFN matrix count. Does not touch attention, embeddings, norms, KV cache.
Regression risks: If the alias were destructive it could overwrite an existing hidden_act; guarded by 'if hidden_act not in config'. The detector subsumes the old string match, so models that were correct via hidden_act/activation_function are unaffected (verified: Llama-2/3.2, Mistral, Qwen2.5, Mixtral, Phi-2, StarCoder2, GPT2 all unchanged).
Verification: BEFORE: assert current Gemma2-9B config (model_type=gemma2, hidden_activation='gelu_pytorch_tanh') counts 7.08e9 (the bug). AFTER: assert it counts in [9.0e9,9.4e9] and equals the value when hidden_act is forced to 'geglu' (9.2417e9). REGRESSION: assert ConfigNormalizer.normalize_config does not mutate a config that already has hidden_act (alias only fills when absent).

PC-GLU-2

Root cause: Gated-vs-non-gated FFN was inferred from an activation-name substring match against glu_variants (parameter_counter.py:18,282,307). This is a fragile proxy in BOTH directions: it misses gated models whose activation spelling is plain (Gemma's gelu_pytorch_tanh) and would mis-fire on non-gated models if their plain activation (gelu_new, gelu_pytorch_tanh, silu) were added to the list. The previously-proposed remedy compounded this by hand-listing 'well-known gated families' that include factually non-gated members (phi=fc1/fc2, starcoder2=c_fc/c_proj, nemotron=squared-ReLU up/down) and by adding non-gated activations (gelu_new, gelu_approx) to glu_variants.
Solution: Replace the substring heuristic with a structural detector is_gated_ffn(self, config) using a priority cascade: (1) UNAMBIGUOUS_GATED_ACT = {swiglu, geglu, reglu, glu, silu_and_mul, geglu_and_mul, swish_glu} -> these activation names only ever denote a gate*up-then-down MLP in HF code; return True. (2) Architecture/family registry: build GATED_FFN_FAMILIES and NONGATED_FFN_FAMILIES, each entry being the structural fact 'this modeling class defines an MLP with (gated)/(no) gate_proj' read from its HF modeling*.py. Match against the architectures-class string FIRST (e.g. 'Gemma2ForCausalLM' vs 'Starcoder2ForCausalLM' -- this is what discriminates two models sharing gelu_pytorch_tanh) then model_type. (3) Fallback for unknown families: if activation in {silu, swish} return True (modern gated default), else return False (HF historical non-gated default). Crucially phi/starcoder2/nemotron/gpt2/gpt_neox/opt/falcon/bloom/mpt are in NONGATED_FFN_FAMILIES so they stay 2-matrix; gelu_new/gelu_pytorch_tanh are NOT treated as gating evidence on their own. Use this single method in both _calculate_ffn_params and _calculate_moe_params (delete the duplicated is_glu computations at :306-307). Remove the self.glu_variants attribute entirely (dead after this change).
Constants & sources: (1) UNAMBIGUOUS_GATED_ACT set: SOURCE = HF transformers ACT2FN registry + per-model modeling code; each listed name is only emitted by a gated MLP. Derived/structural, not fitted. (2) GATED_FFN_FAMILIES: SOURCE = each model's modeling_.py MLP class having gate_proj (verified for llama/mistral/mixtral/qwen2/gemma2/phi3/deepseek_v2_v3/cohere/olmo2/stablelm/granite etc.). Structural enumeration of a code fact, checkable per family. The 3 disputed members (phi, starcoder2, nemotron) are EXCLUDED and placed in NONGATED. (3) NONGATED_FFN_FAMILIES: SOURCE = modeling_.py MLP class WITHOUT gate_proj (gpt2 c_fc/c_proj, phi fc1/fc2, starcoder2 c_fc/c_proj, opt fc1/fc2, gpt_neox dense_h_to_4h/4h_to_h, falcon dense_h_to_4h, bloom, mpt up/down, nemotron squared-relu up/down). Structural, checkable. (4) fallback rule 'plain silu/swish on unknown family => gated': SOURCE = empirical prior over current HF causal-LM zoo (every silu-MLP decoder family ships a gate_proj); conservative default for UNKNOWN families only, overridden by any registry/activation evidence. Not a fitted weight.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/parameter_counter.py:14-24 -> in init replace self.glu_variants with self.UNAMBIGUOUS_GATED_ACT (set), self.GATED_FFN_FAMILIES (set), self.NONGATED_FFN_FAMILIES (set) [2] llm-memory-calculator/src/llm_memory_calculator/parameter_counter.py:264-287 (_calculate_ffn_params) -> compute mult = 3 if self._is_gated_ffn(config) else 2; return non_moe_layers*(multhidden_sizeintermediate_size) [3] llm-memory-calculator/src/llm_memory_calculator/parameter_counter.py:289-334 (_calculate_moe_params) -> replace the local act_fn lookup and is_glu at 306-307 with is_glu = self._is_gated_ffn(config) [4] llm-memory-calculator/src/llm_memory_calculator/parameter_counter.py: add new method _is_gated_ffn(self, config) implementing the cascade above
Impact: Affects the FFN matrix count (dense path) and expert FFN matrix count (MoE path) for every config. MoE coupling is now correct AND robust: previously _calculate_moe_params keyed off the same activation string, so Mixtral with hidden_act removed silently dropped from 46.7B to 31.67B; with the registry (model_type=mixtral in GATED) it stays 46.7B regardless of activation presence. Dense path: Gemma fixed; all other panel families unchanged. Couples to total param count, weight memory, FFN FLOPs in prefill/decode.
Regression risks: A new gated family not yet in GATED_FFN_FAMILIES but using plain silu still resolves correctly via the fallback (silu=>gated). A new gated family using a plain non-silu activation and not in the registry would be under-counted -- residual identical-direction risk to the old default, mitigated because new gated models overwhelmingly use silu. A non-gated family added to GATED by mistake would over-count -- mitigated by requiring each entry trace to a modeling_*.py gate_proj fact. Falcon/MPT/OPT/Phi/StarCoder2/GPT2/GPT-NeoX/Nemotron explicitly pinned non-gated, fixing the exact regressions the critique raised.
Verification: BEFORE applying, run the prototype harness (already executed) asserting is_gated() returns the expected boolean for 15 cases incl. Gemma2(T), Gemma3_text(T), Phi(F), StarCoder2(F), GPT2(F), GPT-NeoX(F), OPT(F), Falcon(F), Nemotron(F), Mixtral with and without act(T), UnknownGated-silu(T), UnknownPlain-gelu(F). AFTER wiring: assert full 10-model param panel within 3% (measured: all <=0.3%). REGRESSION assertions that must NOT increase: Phi-2 count stays 2.779e9 (NOT 3.62e9), StarCoder2-3B stays 3.030e9, GPT2 stays 0.1236e9; and Mixtral stays 46.70e9 whether or not hidden_act is present.

Unit: Collective / communication cost modeling

Approach. The collective model (genz/collective_times.py) is an alpha-beta (latency + data/bandwidth) ring/tree closed form (physically sound) wrapped in an UNSOURCED multiplicative overhead stack (compute_scale_aware_overhead: base_overhead 1.08-1.45, size_factor 1.0-1.15, congestion delta 0.07-0.18, straggler epsilon 0.03-0.08) plus a per-vendor magic table (COMM_EFFICIENCY_TABLES) that is never wired into the live alpha-beta path. The accepted critique is correct on every reproducible point: (1) compute_scale_aware_overhead applies 1.14-1.31x even at single-node N=8, so any byte-identical-single-node claim is false; (2) the three protocol-efficiency tier sets are mutually inconsistent (AR 0.3/0.6/0.85, AG 0.4/0.65/0.85, RS 0.35/0.6/0.8) and uncited; (3) floor vs ceil for num_nodes flips TP=12 from 1 to 2 nodes; (4) no collective calibration harness exists (inference_calibration.py fits only compute/memory eta + t_launch). The original H6 (add inter-node BW from hardcoded IB line-rate tiers and a 1-NIC-per-GPU DGX divisor) and its recycled 2.0us latency are themselves unsourced guesses. The FINAL design REVERSES course per HARD RULE 5: remove the unsourced overhead/efficiency machinery; keep one sourced alpha-beta model whose only tunable (bandwidth-efficiency beta_eff) is reconciled to ONE value set with a published-NCCL citation; route inter-node through a single explicit System field with documented precedence and a conservative pass-through default (no fabricated degradation). Every retained constant traces to (a) the closed-form alpha-beta collective coefficients, (b) published NCCL busBW/peak ratios with citation, or (c) device-sourced link BW/latency. Nothing is tuned to one model/GPU.

Net effect. Single-node inference (GB10 case and all TP<=8 configs): collective time DECREASES -- large-message AR/AG unchanged at ideal level, but the removed 1.14-1.31x M8 multiplier drops single-node TP AllReduce ~12-24%, and small-message beta 0.3->0.4 adds another ~25% on small AR. Because single-node TP comm is a minor fraction of compute/memory-bound prefill+decode, end-to-end TTFT/TPOT moves <1-3% for typical models. Multi-node training/EP: larger reductions -- inter-node term loses both the 0.4x BW penalty (H6, up to +2.5x BW) and the up-to-2.5x M8 scale multiplier, so multi-node AllReduce/All2All can drop ~2-4x vs current; those baselines change materially in a known (faster) direction. Memory: unchanged (comm model touches latency/throughput only). Net: removes unsourced inflation, makes collectives the conservative alpha-beta physical floor, confines remaining tunables to ONE sourced beta_eff set plus device-sourced link BW.

Couplings. M8 (remove overhead multiplier), H6 (inter-node BW), M9 (node count) ALL act on the same multi-node code path and MUST land together: M9 decides whether the inter-node branch runs, H6 sets that branch's bandwidth, M8 removes the multiplier previously stacked on the result. They are NOT independent; the joint multi-node verification test is mandatory. H5-H6-COMM-1 (beta_eff reconciliation) is independent on the single-node path but COMPOUNDS with M8 on every collective: removing the 1.14-1.31x multiplier AND raising small-message beta 0.3->0.4 both make single-node TP AllReduce cheaper, so the combined decode/prefill comm-time reduction exceeds either alone -- quantify on the SAME before/after run, not separately. Cross-unit: this unit feeds operator_base.get_communication_time which is divided by system.comm_efficiency (operator_base.py:474); if the calibration unit later fits comm_efficiency per device, that fit MUST happen AFTER these structural changes (same structure-before-efficiency ordering the audit established for compute/memory), else the fit absorbs the removed multiplier. The MoE-A2A coefficients (get_A2A_time_moe 2.01.15(1+0.05*EP^0.7); get_moe_a2a_time_with_alpha) are flagged unsourced but left UNCHANGED here (separate finding) to avoid coupling an unverified A2A change into the AR/AG/RS reconciliation; they need their own DeepSeek-V2 EP=8 calibration before any edit.

H5-H6-COMM-1

Root cause: COMM_EFFICIENCY_TABLES (collective_times.py:191-287) is a per-vendor magic table presented as real NCCL measurements but with no figure/version/citation, AND it is dead: get_comm_efficiency (290-339) is never called from get_AR_time/get_AG_time/get_reduce_scatter_time, which use their own inline bw_efficiency literals. So the table is simultaneously the only place per-vendor inter-node efficiency is distinguished (the H5 complaint that MI300X and H100 get the same efficiency) and completely unused. Patching it into the live path (original H6) would wire an unsourced table into production.
Solution: DELETE COMM_EFFICIENCY_TABLES and get_comm_efficiency (dead + unsourced). Replace the inline per-function bw_efficiency literals with a SINGLE module-level constant set derived from the NCCL protocol the formula already names, reconciled across AR/AG/RS: BETA_EFF={'small':0.40,'mid':0.65,'large':0.85} with cuts 64KB/1MB. Source: NCCL busBW/algBW efficiency ratios for LL/LL128/Simple protocols (busBW plateaus ~0.85-0.95 of peak for >1MB ring collectives; LL128 small-message ~0.4) per NVIDIA nccl-tests PERFORMANCE.md busBW definition and NCCL protocol thresholds. Hardware-agnostic by design: beta_eff is a PROTOCOL property (how well a ring/tree saturates ANY link) while absolute link BW (System.interchip_link_bw, device-sourced from ICN) carries all hardware difference. This is the generalizable split H5 actually wants: H100 vs MI300X differ via interchip_link_bw, not a hand-typed table. Keep beta_eff identical across AR/AG/RS (same protocol family) to remove the 0.3/0.4/0.35 inconsistency.
Constants & sources: (1) BETA_EFF 0.40/0.65/0.85 (small/mid/large) -- SOURCE: NVIDIA nccl-tests busBW/algBW for LL/LL128/Simple (published benchmark, nccl-tests PERFORMANCE.md). Reconciled to ONE set; previously inconsistent (AR 0.3/0.6/0.85, AG 0.4/0.65/0.85, RS 0.35/0.6/0.8). Chosen = the AG set (least optimistic large tier, documented LL128 small tier). (2) Message-size cuts 64KB,1MB -- SOURCE: NCCL protocol-selection thresholds (LL/LL128/Simple); the coarse bucket boundaries already in code. (3) ring AR coeff 2(N-1)/N, AG/RS coeff (N-1)/N -- SOURCE: closed-form alpha-beta derivation (Rabenseifner/Thakur); not fitted.
Code changes: [1] collective_times.py:191-339 -> DELETE COMM_EFFICIENCY_TABLES and get_comm_efficiency (dead, unsourced). [2] collective_times.py:390-401 (get_AR_time) -> replace inline 0.3/0.6/0.85 with shared BETA_EFF lookup. [3] collective_times.py:516-524 (get_AG_time) -> replace 0.4/0.65/0.85 with shared BETA_EFF. [4] collective_times.py:924-932 (get_reduce_scatter_time) -> replace 0.35/0.6/0.8 with shared BETA_EFF. [5] collective_times.py: add module-level BETA_EFF dict + _beta_eff(data) helper with cited comment.
Impact: Small-message AR beta 0.3->0.4 (~33% faster), mid 0.6->0.65 (~8%), large unchanged 0.85. RS small 0.35->0.40, mid 0.6->0.65, large 0.8->0.85. AG large unchanged. Live callers: operator_base.get_communication_time (TP AR prefill/decode L298/304), training_operators ZeRO AG/RS, training_modeling tp_comm (L3090), training_optimizer (L230/238). Dominant decode TP AR message is small, so decode TP comm ~25% cheaper, but TP comm is rarely the decode bottleneck (memory-bound), so end-to-end decode TPOT moves <1% for single-node TP<=8. Deleting the table changes nothing at runtime (never called).
Regression risks: Breaks any external importer of get_comm_efficiency/COMM_EFFICIENCY_TABLES -- grep confirms ZERO importers outside collective_times.py. Beta reconciliation shifts small/mid-message collective times; any golden baseline that locked old 0.3/0.35 small values moves in a KNOWN direction (collectives cheaper, never slower) and KNOWN magnitude (<=33% on the comm term, a fraction of total). No correctness inversion: beta in (0,1], formula monotone.
Verification: BEFORE: grep -rn 'get_comm_efficiency|COMM_EFFICIENCY_TABLES' src/ --include=*.py | grep -v collective_times.py -> assert EMPTY (dead). Unit assert: for data>1MB beta_eff==0.85 in all three funcs (reconciliation). Sanity: ring AR busBW for N=8/4MB/600GB/s eff=0.85 == 320.6 GB/s (reproduced), 53% of link peak, within published NCCL 8xGPU NVLink AR busBW band (~230-480); assert 200<=busBW<=500. REGRESSION: with use_realistic_overhead=False, >1MB AR/AG times byte-identical before/after (all used 0.85).

M8-OVERHEAD-STACK

Root cause: compute_scale_aware_overhead (collective_times.py:119-183) multiplies the alpha-beta ideal by base_overhead (1.08@N<=8 .. 1.45@4K) * size_factor (1.0-1.15) * half-weighted congestion (delta 0.07-0.18) * half-weighted straggler (epsilon 0.03-0.08). Reproduced: at N=8 the factor is 1.31 (32KB) to 1.14 (4MB), so EVERY single-node TP AllReduce in prefill AND decode silently carries 14-31% overhead. None of these coefficients is sourced (comments say NCCL measurements with no citation; audit_microbench_data.md has ZERO collective measurements). The internal 0.5 normalization (L181) and 2.5x cap are underived. This is the forbidden magic-multiplier.
Solution: REMOVE compute_scale_aware_overhead, compute_network_congestion_factor, compute_straggler_overhead, and the branches that call them. The alpha-beta formula ALREADY contains the only grounded overheads: the startup/latency alpha (base_latency_us) and the bandwidth beta (beta_eff); scale dependence is in the (N-1) factor and the hierarchical split. The deleted factors are a fudge with no measurement to validate. Any genuine per-device collective launch/overhead must later be ADDED via the EXISTING inference_calibration harness as a fit term, NOT a hardcoded multiplier. Keep use_realistic_overhead as an accepted-but-ignored deprecated kwarg (default True) returning the ideal time so all caller signatures are unchanged.
Constants & sources: (1) base_latency_us 20/10/5 (AR), 15/8/5 (AG), 18/10/5 (RS) -- the alpha terms; SOURCE: NCCL kernel+protocol launch latency, order-of-magnitude consistent with the measured 8.26us bare kernel-launch floor (audit_microbench_data.md sec 4). RETAINED (already in code); not newly invented. (2) REMOVED (all unsourced, no microbench, deleted): base_overhead 1.08/1.12/1.18/1.28/1.45, size_factor 1.0/1.05/1.15, congestion delta 0.07-0.18, straggler epsilon 0.03-0.08, 0.5 normalization weight, 2.5x cap.
Code changes: [1] collective_times.py:22-183 -> DELETE compute_network_congestion_factor, compute_straggler_overhead, compute_scale_aware_overhead. [2] collective_times.py:459-475 (get_AR_time) -> remove overhead multiply; return ideal_time. [3] collective_times.py:557-571 (get_AG_time) -> remove overhead*0.95; return ideal_time. [4] collective_times.py:966-978 (get_reduce_scatter_time) -> remove overhead; return ideal_time. [5] Keep use_realistic_overhead kwarg accepted-but-ignored (LLM_training/validation.py:944-945 passes it explicitly).
Impact: Largest behavioral change. Every collective gets faster: -14..31% single node, up to -2.5x at large scale. Coupled: operator_base.get_communication_time (prefill/decode TP AR, EP A2A, AG); training tp_comm, ZeRO-3/ZeRO++ AG+RS, gradient AR. Inference single-node TP=8 AR loses its 1.14-1.31x; prefill TTFT with TP>1 drops by comm_time*(factor-1)/total (small, prefill compute-bound per audit). Multi-node training AR drops substantially -- moves training baselines.
Regression risks: Training/multi-node golden baselines that encoded the inflated overhead now under-predict RELATIVE to those baselines -- but those baselines were never validated against measurement (no collective microbench exists), so they are not ground truth. Direction known (times decrease, monotone), magnitude bounded by the old factor (<=2.5x). Any genuine per-device overhead must come back only via measured calibration, not a guessed multiplier.
Verification: BEFORE: reproduce factor at N=8 (done: 1.31@32KB, 1.14@4MB) to document what is removed. Assert get_AR_time(use_realistic_overhead=False) UNCHANGED before/after (ideal path untouched). Assert get_AR_time(...,True)==get_AR_time(...,False) after. REGRESSION GUARD: pin ring AR busBW (N=8/4MB/600GB/s) to 320.6 GB/s +-1%. DROP the prior single-node-byte-identical guarantee; replace with: single-node large-message (>1MB) ideal time is byte-identical, the realistic-overhead multiplier is removed and that is the intended sourced behavior.

H6-INTERNODE-BW

Root cause: Original H6 added System.inter_node_link_bw from hardcoded IB line-rate tiers (NDR=50/HDR=25/XDR=100/EDR=12.5 GB/s) divided by a 1-NIC-per-GPU DGX assumption, and recycled the 2.0us inter_node_latency from the dead get_hierarchical_AR_time. Problems reproduced: (a) 1-NIC/GPU is DGX-specific, wrong for 4-NIC/8-GPU nodes and for TPU/Trainium/Gaudi (no IB); (b) 2.0us was admitted-unvalidated then re-promoted; (c) THREE inter-node BW sources of truth -- System default interchip_link_bw*0.4 (L384/511/919), the network_config override (L374-380), and the proposed field -- with no precedence; (d) GB10 is single-device so NO inter-node number is measurable, making the proposed verification circular.
Solution: Do NOT add an IB-tier table. Collapse to a SINGLE inter-node model with documented precedence: (1) network_config bandwidth[1]/latency[1]/npus_count[0] if present (caller-supplied real topology) -- existing branch kept, its hardcoded [0.25,2.0] default fallback fixed to read from system; (2) else System.inter_node_link_bw / inter_node_link_latency / gpus_per_node IF the device config populates them (per-device vendor scale-out spec: IB/RoCE/ICI GB/s and actual NIC count); (3) else FALL BACK to a documented physical default: inter_node_link_bw=interchip_link_bw (pass-through, NO fabricated 0.4 or 0.11), inter_node_link_latency=interchip_link_latency, gpus_per_node=num_collective_nodes (collapse hierarchy to the validated flat ring). When no topology is known, inventing a degradation is a guess; the conservative defensible default is the single-fabric flat model the alpha-beta formula already handles. Generalizes to TPU/Trainium/Gaudi (no IB assumption) and non-DGX NIC ratios.
Constants & sources: (1) inter_node default = interchip_link_bw (ratio 1.0 pass-through) -- SOURCE: conservative closed-form default; replaces unsourced 0.4 (L384/511/919) and rejected 0.11. (2) gpus_per_node default = num_collective_nodes when unpopulated -> flat ring -- SOURCE: derivation; avoids M9 fictional-NVLink-domain. (3) Per-device inter_node_link_bw/latency/gpus_per_node when populated -- SOURCE: that device's published scale-out fabric spec + node NIC count (vendor spec, per-entry, generalizable). (4) REJECTED: IB tiers NDR=50/HDR=25/XDR=100/EDR=12.5, 1-NIC/GPU divisor, recycled 2.0us -- all unsourced/topology-specific, removed from plan.
Code changes: [1] system.py:77 -> add params inter_node_link_bw=None, inter_node_link_latency=None, gpus_per_node=None (None=fallback); store BW via unit.unit_to_raw. [2] system.py:163-166 from_dict already filters by signature -> new fields auto-populate from hardware config when present. [3] collective_times.py:382-386,509-513,917-921 -> replace interchip_link_bw0.4 and interchip_link_latency10 with: inter_node_bw=getattr(system,'inter_node_link_bw',None) or system.interchip_link_bw; inter_node_latency=getattr(system,'inter_node_link_latency',None) or system.interchip_link_latency; gpus_per_node default=getattr(system,'gpus_per_node',None) or numNodes. [4] collective_times.py:377-378,505-506,913-914 -> network_config latency fallback [0.25,2.0] replaced with system-derived values so no second hardcoded default. [5] DELETE get_hierarchical_AR_time (collective_times.py:1063-1144) -- dead (no callers); source of recycled 2.0us/600/100.
Impact: Multi-node collectives only. 0.4x inter-node penalty removed: fallback callers get inter_node_bw=interchip_link_bw -> up to 2.5x faster inter-node term. Coupled: training_modeling tp_comm and ZeRO DP comm for nodes>1; inference EP A2A across nodes. Precedence fix removes the three-sources ambiguity (network_config > System fields > flat). All single-node inference/training (num_collective_nodes<=gpus_per_node, GB10 / typical TP<=8) does NOT enter the hierarchical branch -> NO change. Deleting get_hierarchical_AR_time changes nothing (unused).
Regression risks: Multi-node configs not populating new fields now use pass-through inter-node BW (faster) instead of 0.4x. Since 0.4x was itself unsourced and uniformly applied (the hardware-agnosticism H5 condemned), this removes a guess rather than introducing one, and makes a slow fabric EXPRESSIBLE (populate the field). Grep confirms get_hierarchical_AR_time has no callers.
Verification: BEFORE: grep -rn 'get_hierarchical_AR_time' src/ -> assert no callers. Assert single-node path (numNodes<=gpus_per_node) byte-identical before/after. Multi-node cannot be measured on GB10, so verification is a bound/precedence check not a measurement: assert fallback inter_node_bw==interchip_link_bw (no 0.4); unit test feeding network_config={'bandwidth':[600,50],'latency':[0.25,2.0],'npus_count':[8]} asserting the inter-node term uses 50e9. Explicitly do NOT claim a validated inter-node absolute number -- none is measurable here.

M9-NODE-DIVISION

Root cause: num_nodes = max(1, numNodes // gpus_per_node) (collective_times.py:404,526,935) floor-divides. Original M9 proposed ceil. Reproduced: TP=12, gpus_per_node=8 -> floor=1 (fictional 12-GPU NVLink domain), ceil=2 (activates inter-node). Both are wrong as a fixed rule with a hardcoded gpus_per_node=8: floor invents an oversized fabric, ceil invents node boundaries that may not exist. The bug is the FIXED gpus_per_node=8 default, not the rounding.
Solution: (1) Fix the divisor source (via H6: gpus_per_node defaults to num_collective_nodes when unpopulated, so floor(numNodes//numNodes)=1 -> flat ring, never fictional). (2) When gpus_per_node IS populated (real topology), num_nodes = ceil(numNodes/gpus_per_node) only when gpus_per_node<numNodes; else 1. For TP=12 with a real 8-GPU node this yields 2 nodes (8+4). Leave the MoE A2A path (get_A2A_time is_moe -> get_A2A_time_moe / get_moe_a2a_time_with_alpha) node math unchanged -- it is parameter-driven, not the floor bug.
Constants & sources: (1) num_nodes = ceil(numNodes/gpus_per_node) when gpus_per_node explicitly set and < numNodes; else 1 -- SOURCE: topology derivation (a collective spanning more chips than one node's fabric must cross >=2 nodes). No magic number. (2) gpus_in_last_node = numNodes - full_nodes*gpus_per_node -- exact derivation for the partial node.
Code changes: [1] collective_times.py:404,526,935 -> num_nodes = (math.ceil(numNodes/gpn) if (gpn and gpn<numNodes) else 1) where gpn is explicit gpus_per_node; with the H6 fallback gpn=numNodes this yields 1 (flat ring). [2] No change to get_A2A_time_moe node math (parameter-driven).
Impact: Only affects numNodes>gpus_per_node AND gpus_per_node populated. With the H6 fallback (gpus_per_node=numNodes when unpopulated) the common single-node case yields num_nodes=1 -- IDENTICAL to current floor for numNodes<=8 (GB10/typical inference). The TP=12 example changes only if a config explicitly sets 8 GPUs/node; then it correctly becomes 2 nodes and the inter-node term (pass-through BW from H6, NOT the deleted M8 multiplier) applies. The three fixes interact only on this explicit-multi-node path, now well-defined: ceil sets node count, H6 sets inter-node BW, M8 overhead is gone.
Regression risks: Configs relying on floor collapsing (TP=12 as one fast NVLink domain) become 2-node and slower on the inter-node term -- a correctness improvement (12 GPUs cannot share one 8-GPU NVLink fabric) but changes those baselines. Gated on gpus_per_node being explicitly populated, so NO currently-unpopulated config changes today -- forward-compatible, opt-in. NOT independent of H6/M8: verify them together.
Verification: BEFORE: assert all current configs leave gpus_per_node unpopulated -> num_nodes stays 1 for current usage -> byte-identical (note: does NOT exercise multi-node). JOINT multi-node unit test: System(gpus_per_node=8, inter_node_link_bw=50, numNodes=12, data=4MB) -> assert num_nodes==2 (M9), inter-node term uses 50e9 not 0.4*link (H6), no 1.14-2.5x multiplier present (M8); compare combined output to a hand-derived alpha-beta value (8-GPU intra RS + 2-node ring AR over 50GB/s + 8-GPU AG) within 1% to attribute correctness to the combined path (a single end-to-end delta cannot).

Unit: API correctness & honest failure

Approach. This unit makes three API surfaces fail honestly and report physically-grounded numbers instead of fabricated constants, while preserving byte-identical behavior on the success paths that are already correct. The critique was substantially right and I revised accordingly. H1: the system_name guard must route str CPU hardware NAMES to the CPU modeling path via the existing is_cpu_hardware(str) (already used by get_modeling_functions), never silently GPU-model them, and must raise a clear ValueError for truly-unknown str; the prior "guard chunked too" claim was wrong (estimate_chunked_performance and compare_performance_configurations never call system_name.get — verified) and is dropped. H2: the headline fix — the power endpoint's success path was feeding a hardcoded 0.5 (via getattr defaults for attributes that DO NOT EXIST on ModdelingOutput — verified) into PowerModel, so power was workload-insensitive. The correct generalizable fix derives real MFU/MBU per workload from the per-op roofline columns model_df['Compute time']/['Memory time'] that ModdelingOutput already carries (operator_base.py:516-517), making power shape/hardware/precision-sensitive with zero new constants. The 0.15 comm coefficient is replaced by the model_df Communication-time fraction. Unknown-model-vs-HF-failure is disambiguated by catching the loader's typed exceptions (RepositoryNotFoundError→404, GatedRepoError→403, timeout/network→502) at the loader surface, NOT get_configs (which collapses all to one ValueError — verified get_language_model.py:283), making H2 consistent with M7. H3: root cause is that usecases.py calls memory_calculator.calculate_parameters(), a method that DOES NOT EXIST on ModelMemoryCalculator (verified) — so the existing parameter fallback is dead and parameter_count silently stays 0; fix is to call the real memory_calculator.param_counter.count_parameters(config).

Net effect. Power endpoint (H2): estimated_power becomes workload/hardware/precision-sensitive instead of a flat-0.5 constant — prefill-dominated requests up to ~1.5-2x higher accelerator_w, decode-dominated higher dram_w; single-accelerator (tp=1) comm power drops to ~0 (was a small fabricated value). No change to memory/latency of the sim itself. Honest-failure (H1+H2): unknown/gated/unreachable hardware and models now return 404/403/502 instead of fabricated 200 numbers or generic 500; CPU hardware names produce correct CPU numbers instead of silently-wrong GPU numbers. Hardware recommendations + latency (H3): for >7B models, estimated TTFT/E2E increase by the size_multiplier band (1.5/2.0/3.0x) and CPUs are correctly excluded for >14B models; <7B models unchanged. Net: no change to any currently-correct success path (GPU dict modeling, <7B latency); all changes are corrections of previously-fabricated or dead-code outputs.

Couplings. H1 and H2 share the serving.py hardware-validation surface: H1's is_cpu_hardware-based routing must be in place BEFORE H2's util derivation, because H2 reads model_df from whichever modeling function H1 selects — a wrong (GPU) routing for a CPU would give a model_df with no CPU-specific timing and thus wrong utilization. H2's 404/403/502 disambiguation is the SAME exception surface as sibling findings M7/L4 — they must use one shared helper (catch RepositoryNotFoundError/GatedRepoError/_FutureTimeout at the loader, never get_configs's collapsed ValueError) so they cannot diverge; H2 was previously inconsistent with M7 and is now aligned. H2's switch from a hardcoded 0.5 to roofline-derived util is COUPLED to the broader calibration work in audit_microbench_data.md: the MFU/MBU proxies here are occupancy fractions (time-share), distinct from the eta_compute/eta_mem efficiency ratios fit by the differential_evolution harness; they are complementary (eta scales the per-op times that these fractions then aggregate), so applying device calibration later will automatically flow through into more accurate power without further code change. H3 is independent of H1/H2 (different router) but couples to the _estimate_performance latency heuristic and to any DB-persistence path — its parameter_count change propagates into both CPU-exclusion and latency size_multiplier simultaneously, so they must be regression-baselined together.

H1

Root cause: estimate_decode_performance/estimate_prefill_performance (performance_estimator.py) type-hinted system_name as Optional[Dict] but the serving/power paths pass a hardware NAME string. Inside get_inference_system (utils.py:126-135) a str is resolved via get_hardware_config; for a CPU hardware NAME passed as str the prior design's proposed dict-only guard would let it fall through to the GPU prefill_moddeling/decode_moddeling path, silently producing wrong GPU numbers for a CPU. The honest-failure thesis requires either correct CPU routing or a hard error, never a silently-wrong GPU number.
Solution: Do NOT add a dict-only guard. Instead, at every call site that selects modeling functions, use the existing is_cpu_hardware(name) (hardware/configs.py:1205, already accepts str and is the exact predicate get_modeling_functions uses) to route a str CPU NAME to cpu_aware_prefill/decode_moddeling, and a str GPU/ASIC name to the standard functions. For a str that is neither a known GPU config (get_hardware_config returns falsy) nor a CPU (is_cpu_hardware False), raise a clear ValueError('Unknown hardware: ') — a hard, honest failure — rather than letting get_inference_system raise its generic message or silently GPU-model. serving.py already does the correct routing via get_modeling_functions(req.hardware) (serving.py:146) and an up-front is_cpu_hardware/HARDWARE_CONFIGS membership check (serving.py:136-137); the fix is to widen the type hint to Optional[Union[str, Dict, System, CPUSystem]] on the two performance_estimator entry points so the str path is a documented, supported contract, and to ensure they delegate hardware-type routing to get_modeling_functions rather than assuming GPU. Drop the unverified claim that estimate_chunked_performance (performance_estimator.py:399) or compare_performance_configurations (:476) need a .get guard — verified neither calls system_name.get; chunked passes system_name straight to chunked_moddeling and falls back to a default dict when None (utils:441-448), and compare_performance_configurations iterates configuration dicts, so no regression there.
Constants & sources: (1) No new numeric constants. is_cpu_hardware predicate and get_hardware_config resolution are existing, data-driven (config 'type' field + CPU_PRESETS membership), source: hardware/configs.py:1205-1218.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/performance_estimator.py: widen system_name type hint on estimate_prefill_performance and estimate_decode_performance from Optional[Dict] to Optional[Union[str, Dict, 'System', 'CPUSystem']]; documentation only, no behavior change (get_inference_system already handles str/dict/System). [2] BudSimulator/apis/routers/serving.py:136-146: keep the existing up-front membership check (is_cpu_hardware OR in HARDWARE_CONFIGS else 404) and get_modeling_functions routing — this is already the honest path; no dict-only guard added. Add an explicit ValueError->404 mapping note so an unknown str cannot reach GPU modeling.
Impact: Coupled: every caller of get_modeling_functions (cluster.py:109/176, disaggregation.py:215, batch_scheduler.py:246/287, simulator.py) already routes by is_cpu_hardware, so the serving sim, disaggregation and batch scheduler all become consistent — a CPU hardware NAME now produces CPU numbers everywhere, not GPU. No change for GPU names. The type-hint widening is documentation; the runtime behavior of get_inference_system (utils.py) is unchanged for dict/System/CPUSystem inputs (verified byte-identical branches).
Regression risks: A previously-AttributeError-ing str-CPU call now SUCCEEDS with CPU numbers — this is the intended correction, not a regression, but any test that asserted that path raised must be updated. A truly-unknown str now raises a clear ValueError (was a generic ValueError from utils:135) — message text changes; assert on type not message.
Verification: Before: pytest -k 'serving' confirming a CPU hardware NAME str (e.g. 'intel_xeon_8592_plus' if in CPU_PRESETS) routed through estimate_decode_performance returns CPU-path numbers identical to cpu_aware_decode_moddeling(...) called directly (assert equality of Latency). Regression: assert estimate_decode_performance with a GPU dict system_name returns byte-identical Latency to current main (snapshot before/after). Honest-failure assertion: estimate_decode_performance(system_name='totally_unknown_xyz') raises ValueError (pytest.raises), and the serving /power endpoint returns 404 for that name (does NOT 200 with GPU numbers).

H2

Root cause: Two independent fabrication bugs in serving.py /power/estimate. (1) Success path utilization: serving.py:161-162 compute getattr(prefill_result,'compute_utilization',0.5) and getattr(decode_result,'memory_utilization',0.5). Verified ModdelingOutput (utils.py:39-79) and the prefill/decode returns (llm_prefill.py:124-132, llm_decode.py return) define NO such attributes; grep across genz/LLM_inference finds none. Therefore both getattrs ALWAYS return the constant 0.5, fed flat into PowerModel.estimate_from_simulation_result, making accel/dram power workload-, model- and hardware-insensitive — a HARD-RULE-1 violation. The 0.95 clamp never binds (0.5<0.95) and is dead. (2) comm_util = 0.15*log2(tp) (serving.py:172): 0.15 is an unsourced guess. (3) Model-existence oracle ambiguity: get_configs (get_language_model.py:269-283) wraps unknown-model AND HF rate-limit/auth/network in one except Exception -> one generic ValueError, so it cannot distinguish 404 from 502.
Solution: (1) Replace the getattr-0.5 with REAL roofline-derived utilization computed from data ModdelingOutput already carries. Each ModdelingOutput.model_df has per-op columns 'Compute time (...)','Memory time (...)','Communication time (...)' (operator_base.py:516-517) and Latency per op. Define, over repeat-aware summed ops (use count_repeat_aware logic / simplify_df which already multiplies by repeat factor, analyse_model.py:115-139): MFU_proxy = sum(Compute time)/sum(Latency) = fraction of wall time the device is compute-bound = achieved/ceiling compute occupancy; MBU_proxy = sum(Memory time)/sum(Latency) = memory-bound fraction. These are bounded [0,1] by construction (each op Latency=max(compute,memory,comm)+launch, so the part-times sum to <= total). Power then interpolates idle->active by the genuine bound fraction: compute-bound prefill -> high accel util, memory-bound decode -> high dram util — exactly the GB10 measurement (audit_microbench_data.md sec 7: P ~= P_idle + util*(P_max-P_idle), idle 7.2W, max ~70W, util 35%->44W). Use prefill's MFU_proxy for compute_util and decode's MBU_proxy for mem_util (weighted by their latency share of total). (2) comm_util = sum(Communication time)/sum(Latency) from model_df (the AllReduce/All2All time the roofline already computes for tensor/expert parallel) — replaces the 0.15*log2(tp) guess with the model's own communication fraction, zero new constants and generalizable to any TP/EP/PP. (3) Disambiguate existence at the LOADER surface, not get_configs: wrap the get_configs/loader call and catch huggingface_hub.utils.RepositoryNotFoundError -> 404, GatedRepoError -> 403, _FutureTimeout/requests ConnectionError/Timeout -> 502, all else (genuine config-parse / unknown-not-on-HF) -> 400. This matches M7/L4 and never maps a transient HF failure to 404. Keep the existing 0.95 clamp only as a defensive ceiling on the derived ratio (now it can legitimately bind for a pathologically compute-saturated op); document it as a numerical guard, not a physical fit.
Constants & sources: (1) MFU_proxy = sum(model_df['Compute time'])/sum(model_df['Latency]) — DERIVED ratio (closed-form roofline occupancy), source operator_base.py:516, no magic number. (2) MBU_proxy = sum(model_df['Memory time'])/sum(model_df['Latency']) — DERIVED ratio, same source. (3) comm_util = sum(model_df['Communication time'])/sum(model_df['Latency']) — DERIVED ratio (replaces unsourced 0.15). (4) Power interpolation P = P_idle + util*(P_max - P_idle): published/measured linear power model, source audit_microbench_data.md sec 7 (measured GB10 idle 7.2W, max ~70W) and PowerModel idle_power_w/active_power_w which come from per-device vendor TDP/idle (power_model.py:263-265,279-281). No invented constant. (5) 0.95 clamp: retained ONLY as a numerical safety ceiling on a ratio, not a fit; document as such. Source: defensive bound, not data.
Code changes: [1] BudSimulator/apis/routers/serving.py:161-162: replace getattr(...,0.5) defaults with a helper _roofline_util(result) that returns (compute_frac, mem_frac, comm_frac) from result.model_df by summing the per-op 'Compute time'/'Memory time'/'Communication time' over Latency (repeat-aware via simplify_df). Set compute_util from prefill result, mem_util from decode result, comm_util from the larger-comm phase. [2] BudSimulator/apis/routers/serving.py:172: delete comm_util = 0.15*math.log2(tp); replace with the model_df-derived communication fraction (remove the math import if now unused). [3] BudSimulator/apis/routers/serving.py:163-168: keep the except-branch fallback ONLY if model_df is unexpectedly None; if so, do NOT fabricate latency/util — re-raise as HTTPException(502, 'modeling unavailable') so the endpoint fails honestly rather than returning fabricated 0.01/0.5 numbers. Delete the 0.3/0.05/0.4/0.03/0.01/0.5 fabrication block. [4] BudSimulator/apis/routers/serving.py:58 and the analogous get_configs call: wrap in try/except importing RepositoryNotFoundError, GatedRepoError from huggingface_hub.utils and _FutureTimeout; map to 404/403/502/400 respectively.
Impact: Power numbers for ALL models/hardware/batch sizes change from a flat-0.5-derived value to a workload-sensitive value: prefill-heavy requests report higher accelerator_w, decode-heavy report higher dram_w, single-accelerator (tp=1) reports comm_util≈0 (was 0). energy_summary and avg_power_w shift accordingly. The MemoryTier endpoint and the rest of serving sim are unaffected (they do not read these utils). The 404/403/502 split changes status codes returned for bad/gated/unreachable models — coupled to any frontend that branched on 500; frontend must treat 502 as retryable. ServingSimulator.simulator.py:287 already calls estimate_from_simulation_result and would benefit from the same util derivation, but is out of scope for this endpoint-level fix (flag as follow-up, not changed here to avoid altering the heavier sim path).
Regression risks: Any snapshot test asserting estimated_power values will change — this is the intended correction (old values were fabricated). Risk: model_df column unit string includes the unit suffix; must select columns by prefix match ('Compute time') not exact name to be unit-agnostic. Risk: if a phase's total Latency is 0 (degenerate), guard division by zero -> util 0. The 404 split could break a test that asserted a gated model returns 500; update to 403.
Verification: BEFORE applying, run a workload-sensitivity experiment that the prior critique demanded: call estimate_power for the SAME model/hardware with (a) input_tokens=4096,output_tokens=8 (prefill-dominated) and (b) input_tokens=8,output_tokens=512 (decode-dominated). Assert estimated_power['accelerator_w'] for (a) > (b) and estimated_power['dram_w'] for (b) > (a) — under the OLD 0.5-flat code these are EQUAL (test fails on main, proving the bug); under the fix they differ in the known direction (test passes). Also assert estimated_power changes monotonically as batch_size grows for fixed shapes. Existence experiment: assert estimate_power(model='definitely-not-a-real-model-xyz') -> 404; assert a syntactically-valid private/gated id -> 403; simulate HF timeout (monkeypatch loader to raise _FutureTimeout) -> 502 (NOT 404). Regression: for tp=1, comm_util==0.0 (matches old) so single-accel power must remain within float tolerance of recomputing with the new util only where util genuinely was ~0; for the real-util case, snapshot the NEW values as the new baseline and assert they are reproducible run-to-run (deterministic).

H3

Root cause: usecases.py:711 and :821 call memory_calculator.calculate_parameters(config) inside try/except Exception. Verified ModelMemoryCalculator (llm_memory_calculator.calculator, re-exported by src/bud_models.py:20) has NO method named calculate_parameters (it has calculate_decoder_params, calculate_total_memory, and an internal self.param_counter.count_parameters at calculator.py:164). So both calls ALWAYS raise AttributeError, are swallowed by the bare except, and parameter_count silently stays 0. Consequently _get_hardware_recommendations treats every model as 0-param (CPU exclusion at usecases.py:716-719 never triggers for large models -> CPUs wrongly recommended for >14B models) and _estimate_performance size_multiplier (usecases.py:652-660) is always 1.0 (latency under-estimated for big models).
Solution: Call the parameter counter that actually exists and is the same one calculate_total_memory uses internally: memory_calculator.param_counter.count_parameters(config, respect_weight_tying=True) (parameter_counter.py:26, accessed via calculator.py:48 self.param_counter). This is the canonical, generalizable parameter count across all architectures (it is what calculate_total_memory relies on at calculator.py:164), so estimates become consistent with the memory calculator. Apply at both sites (usecases.py:711 and :821). Keep the try/except but narrow it to catch genuine parse failures only and log them, so a future signature change surfaces instead of silently zeroing.
Constants & sources: (1) No numeric constants. count_parameters is data-driven from the model config. The 14e9 CPU-exclusion threshold and the 7e9/30e9/70e9 size_multiplier bands (usecases.py:653-660,717) are PRE-EXISTING product heuristics — out of scope for this finding; flag them as unsourced product thresholds, not introduced here. respect_weight_tying=True matches calculator.py:164 default.
Code changes: [1] BudSimulator/apis/routers/usecases.py:711: replace memory_calculator.calculate_parameters(model_config) with memory_calculator.param_counter.count_parameters(model_config, respect_weight_tying=True). [2] BudSimulator/apis/routers/usecases.py:821: replace memory_calculator.calculate_parameters(config) with memory_calculator.param_counter.count_parameters(config, respect_weight_tying=True). [3] BudSimulator/apis/routers/usecases.py:712-713 and 823-824: keep fallback to 0 but logger.warning the exception so the dead-path regression cannot recur silently.
Impact: For every model whose DB parameter_count was 0 (the common case for HF-loaded models), parameter_count becomes nonzero. Coupled effects: (1) _get_hardware_recommendations CPU exclusion now correctly fires for >14B models (CPUs dropped from recommendations for large models — the intended fix). (2) _estimate_performance size_multiplier jumps from 1.0 to 1.5/2.0/3.0 for 7B/30B/70B models, increasing estimated_ttft and estimated_e2e for those models. (3) usecases.py:822 writes model_data['parameter_count'] into the in-memory dict returned by model_manager.get_model; if any caller persists that dict back to prepopulated.db, a previously-0 sentinel becomes nonzero — downstream consumers that treated 0 as 'unknown' would see a real value. Must verify model_manager.get_model returns a copy or that writes are not auto-persisted.
Regression risks: Latency/SLO estimates for >7B models change (increase) — any cached/stored category latency baselines shift. Hardware recommendation lists for >14B models change (CPUs removed). If model_data is persisted, stored parameter_count rows change from 0 to real values (prepopulated.db is already dirty in git — confirm whether this endpoint writes). Sentinel-comparison consumers (WHERE parameter_count = 0) would now miss these models.
Verification: BEFORE: unit-assert memory_calculator.param_counter.count_parameters(config) returns a known value for a fixed config (e.g. Llama-3-8B config -> ~8.03e9, within 2%) AND that the OLD memory_calculator.calculate_parameters(config) raises AttributeError (pytest.raises) — proving the existing path was dead. Functional: call _get_hardware_recommendations for a >14B model and assert no recommendation has type 'cpu' (was failing on main because params read as 0). Regression baseline for the latency heuristic: snapshot _estimate_performance(ttft,e2e) for a <7B model (size_multiplier stays 1.0 -> values UNCHANGED) and assert byte-identical; for a 8B model assert ttft/e2e increase by exactly the 1.5x size_multiplier factor (known-direction, known-magnitude). DB regression: confirm model_manager.get_model returns a fresh dict (not a live DB row) so writing parameter_count does not auto-persist; add a test asserting prepopulated.db model rows are unchanged after calling _get_models_by_category.

Unit: Frontend honesty & broken controls

Approach. This unit is about UI honesty: three surfaces present fabricated or mislabeled numbers as if they were real. The revised position, after re-verifying every claim in the critique against the code, is to STOP fabricating rather than to derive replacements that are not yet derivable. H7: the backend calculate_price_indicator (hardware.py:989) is an explicitly-documented relative index ("NOT actual pricing"), yet AIMemoryCalculator.tsx:1797,1914 literally render it as ~${price_approx} USD/hr. Fix = strip the "USD/hr" suffix and reuse the existing PriceDisclaimer component (the HardwareList.tsx:202-203 pattern) so the same number is presented honestly everywhere; do NOT invent a dollar figure. H8: the HardwareDetail "Model Compatibility" tab (HardwareDetail.tsx:66-92,448-529) is 100% hardcoded mock data (memory_gb literals, "batch size: 10", 2k/4k/8k/16k columns) with a // Mock ... in real implementation comment. Per HARD RULE 5, remove the dead tab + nav entry now rather than wire a live calculation that the critique correctly shows cannot be made correct yet (Gemma counter H0 is ~23% low, batch_size and precision are unsourced policy choices, and a model allowlist would special-case the catalog). H9: the optimizer's utilization=0.7 (hardware_optimizer.py:345,459,573,581) and self.hardware_costs dict (74-85, $3.0 default) are both fabricated. The critique proves a derived MFU/MBU cannot be honest today (compute time is already de-rated at operator_base.py:468-469; eff=1.0 on 56/57 devices yields MBU>100%; the denominator is in flux pending Unit B/C). So: drop the fabricated 0.7 from any user-facing "utilization %" — show the engine's own binding-resource latency fraction ONLY on devices that carry an inference_calibration block (GB10 today), otherwise omit the field. For cost ranking, replace the static dict with get_hardware_config(name) + the SAME single sourced cost policy as H7 (one helper, one allow_spot flag, displayed WITH its tier/provider label), and render N/A / drop cost from the ranking key for devices with no sourced cost row instead of substituting $3.0.

Net effect. No change to any prefill/decode latency, memory, or throughput numbers — every fix is presentational or ranking-only. Honesty improves materially: one fabricated dollar unit removed (H7), one fully-fabricated data tab removed (H8), and two fabricated optimizer signals (0.7 utilization, $3.0 cost default) removed (H9). Recommendation ORDERING in the usecase optimizer will shift: devices that gain a real on-demand list price (via alias resolution) reorder by genuine $/request, and devices with no sourced price move to a 'cost N/A' bucket instead of being ranked with a fake $3.0 — direction is toward correctness. User-facing utilization disappears for 56/57 devices (shown only on the calibrated GB10) until per-device calibration lands, which is the honest state. Net: zero metric regression, strictly fewer fabricated numbers, ranking changes are bounded and explainable.

Couplings. H7 and H9 are now deliberately coupled to a SINGLE cost source: both call get_best_rate(cost_data, allow_spot=False) and display the (provider,tier) label, eliminating the on_demand-vs-spot contradiction the critique found. H9's utilization fix is coupled to Unit B/C (audit_microbench_data.md section 6 plan-of-record): it reads the engine's de-rated binding-resource fraction rather than recomputing against peak, so when Unit B removes tensor_core_eff (operator_base.py:469) and Unit C fixes dense-FLOPS/causal-mask and refits eta via differential_evolution, the displayed utilization recomputes automatically and stays consistent with latency — it does NOT need to be re-derived. H8 (tab removal) is independent but coupled to H0: a real compatibility matrix can only return after the Gemma param counter (H0) is fixed and batch_size/precision become sourced user axes. The alias additions in configs.py (H9) couple to the hardware-manager alias map (manager.py:72-89) and to any other caller of get_hardware_config for those names.

H7

Root cause: BudHardware.calculate_price_indicator (BudSimulator/src/hardware.py:989-1033) is a log-regression RELATIVE index whose own docstring says 'This is NOT actual pricing, only for relative comparison.' The frontend at BudSimulator/frontend/src/AIMemoryCalculator.tsx:1797 and :1914 renders that index as ~${hw.price_approx.toFixed(2)} USD/hr, fabricating a dollar/hour unit. Separately, the design doc claimed a 'single source of truth' across UI and optimizer, but H7 (this index) and H9 (cost_utils.get_best_rate) are DIFFERENT sources returning DIFFERENT numbers with DIFFERENT meaning, so the claim was false.
Solution: Two-part, no fabricated constants. (a) Honesty fix for the relative index: at AIMemoryCalculator.tsx:1797 and :1914 remove the literal ' USD/hr' and the '~$' currency framing; render it as a unitless relative indicator (e.g. Relative price index: {hw.price_approx.toFixed(2)}) and mount the existing component (already used at HardwareList.tsx:202-203) guarded by hw.price_approx != null so the existing sourced disclaimer text ('approximate values intended for relative comparison only ... do not represent actual costs') is shown. No new numeric constant introduced. (b) Single-source policy: when a device DOES carry a real cost block in configs.py, prefer showing the real sourced rate. Use ONE shared helper with an explicit policy. Define the policy as on-demand (allow_spot=False) because that is the only tier present for ALL 15 cost-bearing configs (verified: every cost block has aws_on_demand/gcp_on_demand; spot is sparse), and label the displayed number with its provider+tier ('$2.21/hr AWS on-demand'). Devices with no cost block fall back to the relative index + PriceDisclaimer. This makes H7 and H9 call the identical helper with the identical allow_spot=False, removing the inconsistency the critique flagged.
Constants & sources: (1) price_approx values: SOURCE = derived by calculate_price_indicator log-regression (hardware.py:1019); it is explicitly a RELATIVE index, NOT a sourced dollar value -> therefore must NOT be displayed as USD. No new constant added by this fix. (2) Real per-device hourly rates ($2.21 aws_on_demand A100 etc.): SOURCE = cloud provider list prices already in configs.py cost blocks (e.g. configs.py:34 aws_on_demand=2.21). These are vendor-published list prices; cite the configs.py provenance. Only used when present (15/57 devices). (3) allow_spot=False policy: SOURCE = derived/closed-form policy choice justified by data availability — on_demand is the only tier present in 100% of the 15 cost blocks (verified via grep of configs.py), so it is the only tier that yields a consistent cross-device number; spot is region/availability-dependent and not generalizable. Documented procedure, not a magic number.
Code changes: [1] BudSimulator/frontend/src/AIMemoryCalculator.tsx:1797 -> change ~${hw.price_approx.toFixed(2)} USD/hr to a unitless Relative price index: {hw.price_approx.toFixed(2)} (remove USD/hr) [2] BudSimulator/frontend/src/AIMemoryCalculator.tsx:1914 -> same change as :1797 [3] BudSimulator/frontend/src/AIMemoryCalculator.tsx (import + render) -> import PriceDisclaimer from '../components/Common/PriceDisclaimer' (path relative to AIMemoryCalculator location) and render it once in the hardware section, guarded by any hw.price_approx != null, mirroring HardwareList.tsx:201-203 [4] BudSimulator/src/optimization/hardware_optimizer.py (cost path, see H9) and any UI consuming get_best_rate -> call get_best_rate(cost_data, allow_spot=False) and surface (rate, provider, tier) so the displayed dollar is labeled; this is the SAME helper H9 uses
Impact: Purely presentational for the relative-index path: price_approx values are unchanged; only the label/unit changes, so no downstream computation moves. The PriceDisclaimer mount is additive UI. The allow_spot=False unification changes which dollar number the optimizer's cost ranking sees (couples to H9): with spot excluded, cost_per_hour rises for any device that had a spot rate, which can reorder recommendations — this is the intended, honest direction (on-demand is the conservative, always-available price). No memory/latency math is touched.
Regression risks: If any other component reads price_approx expecting a currency-formatted string, the label change could look inconsistent — grep shows only HardwareList (already uses PriceDisclaimer), HardwareDetail (separate vendor/cloud real-price path, untouched), and AIMemoryCalculator consume it, so risk is contained. Switching get_best_rate to allow_spot=False raises displayed/used cost for spot-bearing devices; the optimizer's sort key changes accordingly (covered under H9 regression test).
Verification: BEFORE: render AIMemoryCalculator with a device that has price_approx -> assert the DOM no longer contains the substring 'USD/hr' and that the PriceDisclaimer text ('relative comparison only') is present. Unit-assert get_best_rate(cost_data, allow_spot=False) for configs.py A100 cost block (cost at :33) returns (2.06,'GCP','on_demand') i.e. min of the on_demand rates and NEVER a spot tier. REGRESSION: assert calculate_price_indicator(1000,80,2000) still returns its prior value (45.67 per docstring) — the index math is unchanged; only its presentation changed.

H8

Root cause: The HardwareDetail 'Model Compatibility' tab is entirely fabricated: HardwareDetail.tsx:66-92 defines mockCompatibility with literal memory_gb values (14.2, 16.8, ...) and an inline comment 'Mock model compatibility data - in real implementation, this would come from API'; the tab body (448-529) renders only that mock; the subtitle hardcodes 'batch size: 10' (line 453) and the columns hardcode 2K/4K/8K/16K (462-465). batch_size=10 and the seq grid have NO source and materially flip the green/yellow/red verdict (bs swings KV-cache ~linearly). The original design tried to keep the tab and wire /api/models/calculate; the critique correctly shows that path is not yet honest because (i) H0 Gemma param counter is ~23% low, (ii) batch_size and precision are unsourced policy choices, (iii) restricting to a Llama/Mistral/Qwen allowlist special-cases the catalog (violates HARD RULE 2), and (iv) it adds an unbounded N-model x 4-seq fan-out of POST /api/models/calculate on tab open, where calculate may itself hit the HF-config fetch the repo just rate-limit-fixed.
Solution: Take the interim REMOVAL path per HARD RULE 5: delete the fabricated tab and its nav entry rather than patch fabricated numbers or ship a non-generalizable allowlisted live calc. Concretely: remove the 'compatibility' nav item from the tabs array (HardwareDetail.tsx:190), delete the entire activeTab === 'compatibility' block (448-529), delete the mockCompatibility constant (66-92), and remove 'compatibility' from the activeTab union type (line 16) plus the now-unused getCompatibilityStatus helper (59-64) and ModelCompatibility import if unused. A compatibility matrix can be reintroduced later ONLY after H0 lands and after batch_size/precision are exposed as sourced, user-controlled axes feeding the real /api/models/calculate endpoint with bounded, cached fan-out — out of scope for an honesty fix.
Constants & sources: (1) NONE. This finding only DELETES hardcoded constants (mock memory_gb literals at HardwareDetail.tsx:73-90, batch size: 10 at :453, seq columns 2K-16K at :462-465). Removing fabricated constants requires no new sourced constant.
Code changes: [1] BudSimulator/frontend/src/components/Hardware/HardwareDetail.tsx:16 -> change activeTab union 'overview' | 'pricing' | 'compatibility' to 'overview' | 'pricing' [2] BudSimulator/frontend/src/components/Hardware/HardwareDetail.tsx:59-64 -> delete getCompatibilityStatus (only used by the removed tab) [3] BudSimulator/frontend/src/components/Hardware/HardwareDetail.tsx:66-92 -> delete mockCompatibility constant [4] BudSimulator/frontend/src/components/Hardware/HardwareDetail.tsx:190 -> remove the { id: 'compatibility', ... } entry from the tabs array [5] BudSimulator/frontend/src/components/Hardware/HardwareDetail.tsx:448-529 -> delete the entire activeTab === 'compatibility' render block [6] BudSimulator/frontend/src/components/Hardware/HardwareDetail.tsx:3 -> drop the now-unused ModelCompatibility import and unused icons (XCircle if only used there)
Impact: Removes a user-facing feature surface (a tab). No backend or calculation is affected — the tab consumed no live data. Reduces one navigation option in HardwareDetail; overview + pricing tabs unchanged. Eliminates the never-implemented on-tab-open fan-out risk entirely (it was never wired, but removal forecloses anyone wiring the fabricated version).
Regression risks: TypeScript: leaving the 'compatibility' string in the union or a dangling reference will break the build — must remove all four references (type, tabs entry, render block, helper). If any test or route deep-links to the compatibility tab, that link 404s within the component (defaults to overview); grep for 'compatibility' usage across frontend before merge. Visual snapshot tests of HardwareDetail will change (expected).
Verification: BEFORE: npm run build (tsc) must pass with the union narrowed — proves no dangling 'compatibility' references remain. Render HardwareDetail and assert the tab nav contains exactly ['Overview','Pricing'] and that the strings '14.2', 'batch size: 10' no longer appear in the rendered DOM. REGRESSION: Overview and Pricing tabs render identical content to before (snapshot of those two tabs unchanged).

H9

Root cause: Two fabricated signals in BudSimulator/src/optimization/hardware_optimizer.py. (1) utilization=0.7 is hardcoded at four sites (345,459,573,581) with the comment 'GenZ doesn't provide this directly, so use a reasonable default' and surfaced to users (OptimizedHardwareRecommendations.tsx:422-423 renders it as 'Utilization {x100}%'). (2) self.hardware_costs (74-85) is a fabricated dict with a $3.0 default (used at _calculate_cost:425) feeding cost_per_hour and cost_per_request (rank_by_cost_effectiveness:367-369) and the ranking. The original design proposed deriving MFU=2MACs/(peaklat) and MBU=bytes/(BWlat). The critique is correct that this is NOT honestly derivable today: (a) compute_time is ALREADY divided by system.compute_efficiencytensor_core_eff at operator_base.py:468-469, so a nominal-peak MFU diverges from the engine's de-rated latency; (b) system.py:100 op_per_sec=flops/2 and summary_table reports MACs not FLOPs, a factor-of-2 convention trap; (c) eff=1.0 on 56/57 devices (only GB10 has inference_calibration, configs.py) makes MBU=bytes/(BWlat) exceed 1.0 — the audit measured eff=1.0 overpredicts decode by 30% / prefill ~3x; (d) the denominator (dense vs sparse peak, the 0.85 tensor_core factor) is being REMOVED/refit by Unit B/C, so any number wired now is invalidated. And the cost replacement the design proposed (get_hardware_config(name)['cost']) yields None for most optimizer devices: keys A100_40GB/A100_80GB/H100_80GB/GH200/TPUv5 are NOT config keys (verified), only 15/57 configs carry cost, and get_best_rate returns SPOT (cheapest) not on-demand.
Solution: Do not replace one fabricated number with a less-wrong fabricated number. (1) Utilization: remove the user-facing fabricated 0.7. Surface a utilization ONLY when it is the engine's own honest quantity on a calibrated device: read the engine's per-op binding-resource fraction (sum of max(compute_time,memory_time) decomposed into compute-bound vs memory-bound time) which the engine ALREADY computes post-de-rating, and expose it ONLY for devices whose config carries an inference_calibration block (GB10 today). For all other devices, OMIT the utilization field (and hide the row in OptimizedHardwareRecommendations.tsx:422-423 when null) rather than print 0.7. This is generalizable: it self-enables per device as each device is calibrated by the differential_evolution harness (the audit's plan-of-record), and it uses the SAME de-rated latency the engine produced, so it cannot disagree with the displayed TTFT/E2E. It also defers cleanly to Unit B/C: once the denominator is refit, this fraction recomputes automatically because it is read FROM the engine, not computed against a frozen peak. (2) Cost: delete self.hardware_costs (74-85) and the $3.0 default; resolve cost via get_hardware_config(hardware_type) and the SAME helper as H7: get_best_rate(config.get('cost'), allow_spot=False). When it returns a rate, set cost_per_hour=rate*num_nodes and label it with (provider,tier). When it returns None (device has no sourced cost — most of the optimizer's current keys), set cost_per_hour=None, cost_per_request=None, and DROP cost from the efficiency ranking key for those entries (sort survivors by cost, push None-cost entries to a 'cost N/A' bucket sorted by SLO headroom only) — never substitute a fabricated $3.0. To make this generalizable rather than silently-None, ALSO add the missing aliases so the optimizer's device names resolve: the real fix is to align optimizer device keys to config canonical names / add aliases in configs.py (e.g. A100_80GB -> alias of the A100 PCIe/SXM config that has cost at :33), so cost data actually reaches the devices the optimizer iterates. Devices genuinely lacking list prices (consumer/edge) legitimately render N/A.
Constants & sources: (1) utilization fraction: SOURCE = engine-computed compute_time/memory_time per operator (operator_base.py:468-471), aggregated; it is a derived ratio of the engine's own de-rated times, not a constant. Gated on presence of inference_calibration (measured, configs.py GB10 block). No magic number. (2) cost_per_hour: SOURCE = configs.py cost blocks = cloud provider list prices (vendor-published), e.g. aws_on_demand=2.21 at configs.py:34. Used only where present. (3) allow_spot=False: SOURCE = same derived policy as H7 (on_demand is the only tier present in 100% of cost blocks; spot is non-generalizable). Documented. (4) REMOVED constants (no source, deleted): self.hardware_costs values 2.5/3.0/4.0/5.0/6.0/7.0/3.5/5.0/3.5/2.8 (hardware_optimizer.py:75-84) and the $3.0 default (425) and utilization=0.7 (345,459,573,581) — all fabricated, eliminated.
Code changes: [1] BudSimulator/src/optimization/hardware_optimizer.py:74-85 -> delete self.hardware_costs dict [2] BudSimulator/src/optimization/hardware_optimizer.py:422-426 (_calculate_cost) -> reimplement using get_hardware_config(hardware_type) + get_best_rate(cfg.get('cost'), allow_spot=False); return None (not 3.0) when no rate; multiply by num_nodes only when a rate exists [3] BudSimulator/src/optimization/hardware_optimizer.py:249 -> 'cost_per_hour': _calculate_cost(...) may now be None; propagate None [4] BudSimulator/src/optimization/hardware_optimizer.py:345,459,573,581 -> remove the hardcoded 0.7; set utilization only from engine binding-resource fraction when the system carries inference_calibration, else None [5] BudSimulator/src/optimization/hardware_optimizer.py:365-386 (rank_by_cost_effectiveness) -> handle cost_per_hour None: skip cost_per_request (None), and sort None-cost entries into a separate bucket ranked by headroom; do not let None feed efficiency_score [6] llm-memory-calculator/src/llm_memory_calculator/hardware/configs.py -> add aliases ('A100_80GB','A100_40GB','H100_80GB','GH200','TPUv5' as appropriate) to the canonical configs that carry cost, so optimizer device keys resolve to a cost-bearing config instead of None [7] BudSimulator/frontend/src/components/Usecases/OptimizedHardwareRecommendations.tsx:363-366,422-423 -> render 'N/A' for null cost_per_hour/cost_per_request and hide/label the Utilization row when utilization is null (not 0.7)
Impact: Ranking changes: removing the fabricated $3.0 default means devices without sourced cost no longer get a fake-cheap ranking; they move to a 'cost unknown' bucket. Adding correct aliases means A100/H100/etc. now get their real on-demand list price, which CHANGES cost_per_request and therefore the recommendation order — intended and honest. utilization is removed from output for 56/57 devices; any consumer of OptimizationResult.utilization must tolerate None. cost_per_hour/cost_per_request can now be None; OptimizedHardwareRecommendations.tsx and summary.min_cost_per_request (already None-guarded at :257) must handle None (formatCost already guards). Couples to H7 (shared get_best_rate/allow_spot=False) and to Unit B/C (the utilization fraction recomputes when the de-rated denominator is refit).
Regression risks: (1) Any test asserting utilization==0.7 or cost_per_hour==3.0*nodes will break — those assert fabricated values and SHOULD be updated. (2) If rank_by_cost_effectiveness is not hardened against None cost, JSON serialization or sort will throw — must bucket None explicitly. (3) Alias additions must not collide with existing aliases or shadow a different real device (verify each alias maps to a config whose specs match the optimizer's intent, e.g. A100_80GB must alias the 80GB A100 not the 40GB). (4) Showing utilization only on GB10 means the optimizer's internal cost_per_request still divides by throughput (unaffected) — only the DISPLAY field changes; do not let removing the field break throughput math (throughput is computed from e2e at :250, independent of the 0.7).
Verification: BEFORE: (a) enumerate the optimizer device set _get_hardware_types() (hardware_optimizer.py:394-396) and assert, for each, get_hardware_config(name) resolves and report which have get_best_rate(...,allow_spot=False) -> rate vs None; this directly tests the critique's 'which keys resolve to cost' gap. (b) Assert get_best_rate for an A100 cost block with allow_spot=False returns an on_demand tier (never 'spot'/'preemptible'), matching H7. (c) On GB10 (has inference_calibration), assert the surfaced utilization equals compute_bound_time/(compute_bound_time+memory_bound_time) from the SAME model_df the engine used for latency, and that 0<=utilization<=1 (no >100% bug). On a non-calibrated device assert utilization is None (NOT 0.7, NOT a clamped fabricated value). REGRESSION: before/after recommendation ORDER for one fixed usecase must change ONLY in the documented direction (devices gaining real on-demand cost reorder by real $/req; None-cost devices drop to the unknown bucket); assert TTFT/E2E/throughput numbers are byte-identical (this change touches only cost and the display utilization, not latency). Add a regression test that no OptimizationResult.utilization is the literal 0.7 unless it genuinely equals the engine fraction on a calibrated device.

Unit: Engine modeling nuances

Approach. This revision abandons every global magic constant and every "fix one device" tuning the critique flagged, and instead routes all device-specific physics through fields the hardware config ALREADY carries (interconnect, interconnect_bandwidth_gbps, memory_type, architecture, compute_capability) plus the existing measurement-fit inference_calibration block + differential_evolution harness. Concretely: (L2) replace the single OFFLOAD_BW global with a per-device effective-offload-BW derived from the config's interconnect spec (with a sourced direction/overhead derivation), not 64 and not 128; (L3/M3/C4) the 0.85 tensor_core_eff and get_shape_dependent_op_intensity are confirmed dead/redundant against the measured GB10 roofline and removed — but ONLY after a mandatory re-fit of the GB10 inference_calibration block (the keystone device's eta was fit with 0.85 live, so deletion without refit breaks it); (L1) gate the FP8 warning on a derived supports_fp8 capability threaded from the config, not on a string nor at a site that lacks the field; (M1) fix the PP latency/throughput math after correctly identifying that the PP graph already serializes ALL stages (so decode_latency is full-model, not one stage); (M2) the output_tokens>10 cache claim is withdrawn and re-scoped; (L5) weight/KV precision separation is fully specified with sourced int4 dequant term or explicitly deferred with rationale. Every numeric constant below traces to a vendor spec, a measured GB10 microbench (audit_microbench_data.md), a published number with citation, or a closed-form derivation, and every device-specific value is fit per-device by the documented harness — none is hand-typed to make one case look right.

Net effect. Prefill/compute-bound latency on the ~91 UNCALIBRATED devices drops ~15% (removal of the flat 0.85 de-rate) — direction: faster, and more physically defensible since the de-rate now lives only in measured per-device eta_compute. GB10 (the one calibrated device) net ~0% after the mandatory refit (eta_compute absorbs the 1.176x). Decode latency essentially unchanged except the small ot<=10 KV-growth correction (sub-1% for weight-dominated small models, larger for long-context). Offloaded-model latency INCREASES for PCIe-class devices (effective BW 128->~54 pcie5 / ~27 pcie4) and stays high for NVLink/C2C — more realistic, only affects offload path. PP>1 single-request Latency drops ~2x (removes the erroneous (2-1/PP) inflation); PP>1 throughput shifts to the GPipe steady-state. Memory estimates unchanged except optional mixed-precision KV (L5) when explicitly enabled. FP8 changes are warning-only (zero numeric impact). Net: more accurate, removes ~3 magic globals (0.85, OFFLOAD_BW, 0.1 KV factor), adds zero new hand-typed constants, and every device-specific value remains a measured/derived ratio.

Couplings. CRITICAL ORDERING (from audit_microbench_data.md section 6 plan-of-record): L3/M3/C4 (remove 0.85 + dead shape code) MUST land BEFORE the GB10 inference_calibration block is refit, and the refit MUST happen in the SAME change-set — otherwise the one calibrated device (GB10) is left wrong by 1/0.85=1.176x on the compute-bound prefill path. The refit (re-running inference_calibration.py) is the load-bearing verification for this whole unit. L2 (offload BW) and L1 (FP8 gate) are independent of the roofline and of each other. M1 (PP) is independent but its claimed parallelism_optimizer coupling is WITHDRAWN (grep shows no such consumer). M2 (KV-growth) only touches 1<ot<=10 and does NOT touch the GB10 calibration grid (ot=160), so it neither helps nor breaks the refit but is also therefore unvalidated by the existing grid — flagged. L5 default path is byte-identical and couples to nothing unless kv_bits is explicitly passed. Compounding effect: L3 removal (compute +17% for uncalibrated devices) and M1 PP-latency correction (PP>1 latency drops ~2x) BOTH make latencies smaller; they do not offset each other (different code paths) but a reviewer must not attribute one's change to the other when diffing.

L2-offload-bw-per-device

Root cause: utils.py:10 defines a single global OFFLOAD_BW (currently 128) used at utils.py:96 for EVERY device whose system.external_mem_bw==0, and at utils.py:168 as the External-mem BW for every System built from a dict. Inference NEVER populates external_mem_bw from the config (grep confirms interconnect_bandwidth_gbps is read only in training/time_estimator.py:368), so the global is ALWAYS used. Offload weight-streaming BW is physically the host<->device interconnect bandwidth, which is device/topology-specific (PCIe4 x16 ~32 GB/s/dir, PCIe5 x16 ~64 GB/s/dir, NVLink-C2C hundreds of GB/s, CXL different again). Both 128 and the proposed 64 are single global magic numbers and violate generalizability identically.
Solution: Delete the global constant as the source of truth. Derive offload BW per-device from the config field the hardware library ALREADY carries: configs.py stores interconnect_bandwidth_gbps for every entry (e.g. A100 nvlink=600, H100_PCIe pcie5=128, RTX pcie4=64, GB10 NVLink-C2C). Thread this into get_inference_system so external_mem_bw is set from interconnect_bandwidth_gbps when present. The offload path streams weights uni-directionally (read host->device), so the EFFECTIVE achievable BW = interconnect_bandwidth_gbps is ALREADY the per-direction figure for PCIe entries in this table (pcie4=64? -> see note) — to avoid the direction ambiguity the critique raised, define effective_offload_bw = interconnect_unidirectional_GBps * EFFICIENCY where EFFICIENCY=0.85 (measured PCIe DMA streaming efficiency, see constants) and interconnect_unidirectional is computed from the config's interconnect class via a small sourced table (PCIe figures in configs are already per-direction x16 theoretical; NVLink/C2C figures are bidirectional aggregate and must be halved for a one-way stream). Keep a single conservative FALLBACK only when neither external_mem_bw nor interconnect_bandwidth_gbps is available, and document it as PCIe4 x16 (the most common host link) = 32 GB/s effective, not 64/128. Implementation: in utils.py get_offload_system, offload_bw = system.get_external_mem_bw() if system.external_mem_bw>0 else FALLBACK; and in get_inference_system set external_mem_bw = _effective_offload_bw(system_name dict) instead of OFFLOAD_BW.
Constants & sources: (1) PCIe5 x16 unidirectional theoretical = 63.0 GB/s (= 32 GT/s x 16 lanes x 128b/130b PCIe5 encoding / 8). SOURCE: PCI-SIG PCIe 5.0 base spec, derived. The configs.py value 128 for pcie5 is the BIDIRECTIONAL aggregate; one-way stream uses ~63. (2) PCIe4 x16 unidirectional theoretical = 31.5 GB/s (16 GT/s x 16 x 128/130 / 8). SOURCE: PCI-SIG PCIe 4.0 spec, derived. configs.py pcie4=64 is bidirectional aggregate. (3) PCIe DMA streaming achievable efficiency = 0.85 of theoretical. SOURCE: NVIDIA bandwidthTest / published H2D pinned-memory measurements (typically 0.80-0.90 of theoretical on PCIe4/5). Cited as a published-benchmark range; using 0.85 midpoint. This is a per-link-class efficiency, applied uniformly to the interconnect class, NOT a per-device tune. (4) NVLink/NVLink-C2C bidirectional figures in configs (600/900/100) must be halved for unidirectional offload read. SOURCE: NVIDIA NVLink spec (quoted GB/s are aggregate bidirectional). Derived. (5) FALLBACK offload BW = 32 GB/s effective (PCIe4 x16 most-common host link, * already-effective). SOURCE: PCI-SIG derived; chosen as conservative lower bound, documented as fallback-only and only reachable when a config carries neither external_mem_bw nor interconnect_bandwidth_gbps.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py:10 -> remove OFFLOAD_BW = 128 as a physics constant; replace with OFFLOAD_BW_FALLBACK_GBPS = 32 # PCIe4 x16 effective, used ONLY when config lacks interconnect data and a module-level _PCIE_UNIDIR_EFFICIENCY = 0.85. [2] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py:96 -> offload_bw = system.get_external_mem_bw() if system.external_mem_bw != 0 else OFFLOAD_BW_FALLBACK_GBPS. [3] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py: add helper _effective_offload_bw(cfg: dict) -> float that reads cfg['interconnect_bandwidth_gbps'] and cfg['interconnect'], converts NVLink-class aggregate->unidirectional (halve) and applies _PCIE_UNIDIR_EFFICIENCY for pcie classes, returning effective GB/s; returns OFFLOAD_BW_FALLBACK_GBPS if fields absent. [4] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py:168 -> replace external_mem_bw=OFFLOAD_BW with external_mem_bw=_effective_offload_bw(system_name) (system_name is the resolved dict at that point).
Impact: Only affects the OFFLOAD path (get_offload_system), reached when per_chip_memory < total_memory_req/memory_parallelism AND model_offload=True (llm_decode.py:78-80, llm_prefill.py:61-63). For all in-memory cases NOTHING changes (offload_bw is never read). When offloading, the recomputed effective offchip BW at utils.py:105 will DROP for PCIe-class devices (was 128, now ~54 for pcie5, ~27 for pcie4) -> offloaded-model latency INCREASES (slower, more realistic) and stays HIGH-BW for NVLink/C2C devices. Coupled outputs: ModdelingOutput.Latency/Throughput for offloaded configs; the offload warning message BW value. No coupling to non-offload results, to the calibration block, or to the roofline.
Regression risks: Any currently-'correct' offloaded result that was implicitly tuned to 128 GB/s will change (get slower for PCIe). This is a correction, not a regression, but downstream snapshot tests that assert exact offloaded latency must be re-baselined. Risk if a config has interconnect_bandwidth_gbps in a non-GB/s unit — verified all configs.py entries are GB/s. Risk that GB10 (unified memory) should NOT offload over the 100 GB/s C2C as if it were discrete — GB10 has unified_memory=True and Memory_size=128GB, so it rarely triggers offload; when it does, NVLink-C2C unidirectional is the correct link. Guard: if unified_memory==True, offload BW = offchip_mem_bw (the LPDDR5X is the host memory) — add this branch.
Verification: BEFORE: unit test asserting _effective_offload_bw({'interconnect':'pcie5','interconnect_bandwidth_gbps':128}) == round(63.0*0.85,1) ~= 53.6, ({'interconnect':'pcie4','interconnect_bandwidth_gbps':64}) ~= 26.8, NVLink ({'interconnect':'nvlink','interconnect_bandwidth_gbps':600}) == 300 (halved, no pcie efficiency penalty), and ({}) == 32.0. Run decode_moddeling with a model larger than device memory + model_offload=True on H100_PCIe and assert effective offchip BW print < pre-fix value. REGRESSION: assert that for every config where memory fits (no offload), decode_moddeling/prefill_moddeling Latency is byte-identical to pre-fix (diff the model_df Latency sum on A100/H100/GB10 across the calibration grid).

L3-M3-C4-remove-dead-shape-code-with-mandatory-refit

Root cause: operator_base.py:465-469 multiplies compute_time by get_tensor_core_efficiency_factor() (returns 0.85 default, else get_tensor_core_efficiency 0.4-0.95), and operator_base.py:454 calls get_shape_dependent_op_intensity for the reported 'Op Intensity' column. audit_microbench_data.md section 3 proves the MEASURED GB10 GEMM curve is already a clean roofline (min(AI*eff_BW, eff_peak)); shape/M dependence emerges automatically from per-op arithmetic intensity via the existing max(compute_time, memory_time). So the 0.85 tensor_core_eff is a redundant flat de-rate that double-counts what eta_compute already captures, and get_shape_dependent_op_intensity feeds ONLY the reported 'Op Intensity' column (operator_base.py:505) — it NEVER feeds exec_time (line 479 uses get_compute_time/get_memory_time directly). The critique is correct on both: C4 has ZERO latency impact (cosmetic column only), and 0.85 DOES affect latency.
Solution: Two coupled edits with a HARD ordering gate. (1) Remove the tensor_core_eff multiply: operator_base.py:465-470 collapse to effective_compute_efficiency = system.compute_efficiency; compute_time /= effective_compute_efficiency. The shape de-rate now lives entirely in the measured eta_compute. (2) Remove get_shape_dependent_op_intensity usage at operator_base.py:454-460, reverting 'Op Intensity' to num_ops/num_data (matching get_model_characterstics at line 381 — making the two code paths consistent). KEYSTONE REQUIREMENT (was missing): because GB10's prefill eta_compute=0.096 and decode eta_mem=0.659 in configs.py:349-352 were FIT by inference_calibration.py while the 0.85 multiplier was LIVE in get_roofline (the harness calls the real estimate_prefill_performance->prefill_moddeling->get_model_df->get_roofline path), deleting 0.85 raises raw compute throughput by 1/0.85=1.176x and INVALIDATES the fitted prefill eta_compute. Per audit_microbench_data.md section 6 plan-of-record, structural/dead-code fixes MUST land first, THEN re-run python -m llm_memory_calculator.validation.inference_calibration GB10 and paste the refit block. The decode eta_mem is memory-bound (compute_time path not on critical path for decode) so is largely unaffected, but it is refit in the same harness run for consistency. For the 'TC Effcy' reported column: remove it rather than substitute a placeholder 1.0 (the critique correctly rejects 1.0 as a new magic number); if a consumer needs it, it must read 'C Effcy' (the real system efficiency). grep confirms no production consumer reads 'TC Effcy' (only operator_base writes it).
Constants & sources: (1) 0.85 tensor_core_eff default: REMOVED (was an unsourced flat placeholder; operator_base.py:434). (2) GB10 prefill eta_compute, eta_mem, t_launch_ms and decode eta_mem, t_launch_ms, c_stream: to be REFIT by differential_evolution / lstsq in inference_calibration.py against the measured 14-point GB10 grid (audit_microbench_data.md + INFERENCE_BENCHMARKS). SOURCE: measured microbenchmark, fit by documented harness. Current values (prefill eta_compute=0.096 etc.) are STALE post-deletion and MUST be regenerated; expected new prefill eta_compute ~= 0.096*0.85 ~= 0.082 (since raw compute got 1.176x faster, the fit compensates), but the exact value comes from the re-run, not hand-typed. (3) All other devices: have NO inference_calibration block, so ceff/meff default to 1.0 and the 0.85 removal changes their reported compute_time by exactly 1/0.85. This is the correct direction (the roofline should not carry a hidden flat de-rate); these devices were never validated against measurement so there is no calibrated value to break.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:464-470 -> delete tensor_core_eff lines; effective_compute_efficiency = system.compute_efficiency; compute_time /= effective_compute_efficiency; compute_efficiency = effective_compute_efficiency. [2] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:452-460 -> delete the get_shape_dependent_op_intensity branch; op_intensity = num_ops / num_data if num_data else 0. [3] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:509 -> delete the 'TC Effcy': tensor_core_eff dict entry. [4] llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py:397-434 -> delete get_tensor_core_efficiency_factor; operator_base.py:110-165 -> delete get_shape_dependent_op_intensity (dead after the call site removal). Also delete get_tensor_core_efficiency (lines ~80-107) if it has no other caller (grep: only get_tensor_core_efficiency_factor calls it). [5] llm-memory-calculator/src/llm_memory_calculator/hardware/configs.py:346-353 -> REPLACE the GB10 inference_calibration block with the freshly refit values (run the harness AFTER the code deletions above; do not hand-edit numbers).
Impact: Latency-affecting for ALL devices via compute-bound ops (prefill, large-M GEMM): removing 0.85 makes compute_time 1.176x smaller -> compute-bound latency drops 15% for every UNCALIBRATED device. For GB10 (the one calibrated device), net effect ~= 0 after refit because eta_compute absorbs the change (this is exactly the section-6 coupling proof). 'Op Intensity' reported column values change (revert to num_ops/num_data) — affects budevolve/roofline_analyzer.py:65 and plot_rooflines.py which READ 'Op Intensity'; this is a reported-metric change with no latency effect. 'Bound' classification at operator_base.py:483-490 is unchanged (uses com_to_mem_ratio from compute_time/memory_time, and both scale, ratio unaffected by removing a constant factor on compute_time... NOTE: removing 0.85 from compute_time DOES change com_to_mem_ratio by 1.176x, which can flip a borderline op from Compute to Memory — see regression).
Regression risks: (1) GB10 calibration invalidation — MITIGATED by the mandatory refit gate; CI must fail if configs.py GB10 block predates the operator_base edit (add a checksum/marker). (2) 'Bound' flips: an op with com_to_mem_ratio in (0.85, 1.0] was 'Memory', becomes 'Compute' after the 1.176x — affects the boundedness label and runtime_breakdown bucketing, not total latency. Acceptable and more correct (0.85 was hiding compute-bound ops). (3) Any consumer reading 'TC Effcy' breaks — grep shows none in production (only the writer). (4) Reported 'Op Intensity' changes for GEMM/Logit/Attend — consumers in budevolve/plot get different intensity values; these are diagnostic plots, no correctness contract.
Verification: BEFORE applying: grep -rn "'TC Effcy'|"TC Effcy"|TC Effcy" across BOTH repos to confirm zero readers (only operator_base.py:509 writer). Then ORDERED experiment: (a) apply the operator_base deletions; (b) run python -m llm_memory_calculator.validation.inference_calibration GB10 and capture decode_rms_pct/prefill_rms_pct; ASSERT decode RMS <= 7% and prefill RMS <= 29% (the measured envelope from audit_microbench_data.md sections 6: decode +-7%, prefill -20..-29%); (c) paste the printed block into configs.py. REGRESSION: assert that GB10 across the 14-point grid stays within +-7% decode / +-29% prefill of measured AFTER refit (re-run estimate_*_performance and compare to INFERENCE_BENCHMARKS.measured_ms). Assert uncalibrated A100 prefill latency drops by exactly factor 0.85 on a pure-GEMM micro-model (sanity that only the flat de-rate was removed).

L1-fp8-warning-gate-supports-fp8

Root cause: system.py:130-136 unconditionally warns whenever bits in fp8 family, regardless of hardware. The critique is correct on two counts: (a) System.init receives only bits/flops/frequency and has NO architecture/capability field, so the warning site cannot currently see whether the device supports FP8; configs.py carries architecture+compute_capability but they are not threaded into System via get_inference_system (utils.py:142-170 reads Flops/Memory_BW/ICN from the dict but drops architecture). (b) Gating on a coarse architecture string is not robust: Ada Lovelace (sm_89) supports FP8 but is not Hopper; some Ampere does not; AMD CDNA3 supports fp8 but not all variants. A name check misclassifies the 92 DB entries.
Solution: Introduce an explicit, derivable capability rather than a string match. (1) Add a supports_fp8 boolean to System.init (default None=unknown -> preserve current warn-always behavior so nothing regresses for callers that don't pass it). (2) Derive supports_fp8 at the config layer from the fields already present: a device supports FP8 if its NVIDIA compute_capability >= 8.9 (Ada/Hopper/Blackwell) OR architecture in the FP8-capable set {ADA_LOVELACE, HOPPER, BLACKWELL, CDNA3, GAUDI3} OR tensor_cores gen >= gen4. This mirrors the EXISTING training catalog which already encodes supports_fp8 per SKU (training/hardware_catalog.py:99,384...). Reuse that table as the source of truth instead of inventing a parallel one. (3) Thread it through get_inference_system: when system_name is a dict, compute supports_fp8 = _derive_fp8_support(dict) and pass to System(); when system_name is already a System, leave as-is. (4) In system.py, only warn when self.supports_fp8 is explicitly False.
Constants & sources: (1) FP8 capability threshold: NVIDIA compute_capability >= 8.9. SOURCE: NVIDIA — FP8 Tensor Cores introduced in Ada Lovelace (sm_89) and Hopper (sm_90); Ampere sm_8.0/8.6 has no FP8. Derived from CUDA architecture docs. (2) FP8-capable architecture set {ADA_LOVELACE, HOPPER, BLACKWELL, CDNA3, GAUDI3}. SOURCE: vendor: NVIDIA Ada/Hopper/Blackwell Transformer Engine; AMD CDNA3 (MI300) FP8; Intel Gaudi3 FP8. Matches training/hardware_catalog.py supports_fp8=True entries. (3) No new tuned numeric — supports_fp8 is a boolean derived from existing sourced config fields.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/system.py:70-92 -> add supports_fp8=None to init signature; store self.supports_fp8 = supports_fp8. [2] llm-memory-calculator/src/llm_memory_calculator/genz/system.py:130 -> if self.bits in (fp8 family) and self.supports_fp8 is False: then warn (only on KNOWN-unsupported). [3] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py: add _derive_fp8_support(cfg) reading cfg['compute_capability']/cfg['architecture']/cfg['tensor_cores']; in get_inference_system pass supports_fp8=_derive_fp8_support(system_name) into the System(...) constructor at utils.py:167-170. [4] Optionally import the canonical set from training/hardware_catalog to avoid duplication.
Impact: Affects ONLY the warning emission (no numeric/latency change anywhere). Callers passing a System object directly (utils.py:156-163) keep supports_fp8=None -> warn-always (unchanged). Dict-config callers (the normal serving path) now suppress the warning on Ada/Hopper/Blackwell/CDNA3/Gaudi3 and still warn on Ampere/Volta/Turing/older. No coupling to roofline, calibration, or offload.
Regression risks: Low. Risk if a config lacks compute_capability AND architecture -> _derive_fp8_support returns None -> warn-always (safe fallback, current behavior). Risk of false-negative for a future SKU not in the set -> warns spuriously (conservative). No risk of false-positive silencing on truly-unsupported hardware because Ampere/Volta are excluded by the >=8.9 threshold.
Verification: BEFORE: unit tests: _derive_fp8_support(A100 cfg cc=8.0) is False; (RTX4090 ADA cc=8.9) is True; (H100 cc=9.0) True; (GB10 BLACKWELL cc=12.1) True; (MI300 CDNA3) True; ({} ) is None. Assert System(bits='fp8', supports_fp8=False) emits the warning and System(bits='fp8', supports_fp8=True) emits NONE (pytest.warns / recwarn). REGRESSION: assert no latency/throughput field of any calculate path changes (warning-only); assert legacy System(bits='fp8') with no supports_fp8 still warns.

M1-pp-latency-throughput

Root cause: The critique is correct and the original design misidentified what model_df latency represents. get_language_model.py:471-484 (and the prefill builder 460-491) emits repeat(PP-1)layers_per_stage + a Message-Pass op + layers_last_stage, i.e. ALL num_decoder_layers are counted exactly once. Therefore decode_latency/prefill_latency from model_df.sum() is the FULL-MODEL single-forward latency, NOT one pipeline stage. The current code at llm_decode.py:177-179 / llm_prefill.py:108-110 computes stage_latency=full/PP and total_latency=full+(PP-1)stage = full(2-1/PP), which is ~2x the true full-model latency, and thrpt=1000B/total uses that inflated denominator. Also ceil(L/PP) makes stages unbalanced when L%PP!=0, so latency/PP is not the true max-stage time, and the graph's Message Pass op (inter-stage activation transfer) is already inside model_df, so dividing by PP wrongly amortizes the comm op into a stage.
Solution: Reframe using the GPipe steady-state with the CORRECT identification that model_df = full-model forward across all stages, and that the per-stage compute is full_compute/PP only in the balanced case. Define per-stage latency from the ACTUAL graph, not full/PP: tag each stage's layer-sum during graph construction is invasive; the minimal correct closed form uses balanced-stage approximation with the true max-stage layer count. (1) Single-request LATENCY (TTFT/TPOT for one sequence traversing the pipeline) = full-model forward latency = decode_latency (the graph already serializes all stages once, including the Message Pass). So total Latency for PP>1 = decode_latency (NOT *(2-1/PP)). The current *(2-1/PP) is wrong and must be removed. (2) THROUGHPUT under microbatching (GPipe steady state) = num_microbatches tokens drained at the bottleneck-stage rate: thrpt = 1000 * batch_size / (full_model_latency * (1 + (PP-1)/num_microbatches)) where the bubble fraction is (PP-1)/(num_microbatches+PP-1). With the unbalanced split, the bottleneck stage time = full_model_latency * (max_layers_per_stage / total_layers); compute max_layers_per_stage = ceil(L/PP). So stage_latency = full_model_latency * ceil(L/PP)/L. Throughput = 1000 * batch_size / (PP * stage_latency + (num_microbatches-1) * stage_latency) for the GPipe fill+drain. This removes both the 2x latency inflation and the balanced-split assumption, and does NOT double-count Message Pass (it stays in full_model_latency which equals the sum of all stages including comm).
Constants & sources: (1) Bubble fraction (PP-1)/(num_microbatches+PP-1): SOURCE: Huang et al. GPipe (2019), Narayanan et al. Megatron-LM PP — closed-form derivation, not a tuned constant. (2) max_layers_per_stage = ceil(num_decoder_layers/PP): SOURCE: the graph's own split at get_language_model.py:472, derived (not invented). (3) No fitted constants. There is NO measured PP>1 anchor in the grid (critique correct: INFERENCE_BENCHMARKS has only PP=1), so this fix is validated by INTERNAL CONSISTENCY (PP=1 reduces to existing result) and by the closed-form GPipe derivation, with a comment marking that a measured PP>1 anchor is needed before claiming sub-10% PP accuracy.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/llm_decode.py:175-181 -> replace the stage_latency=decode_latency/PP + total_latency=decode_latency*(2-1/PP) block: set total Latency returned = decode_latency (full-model, unchanged by PP for a single request); compute stage_latency = decode_latency * ceil(num_decoder_layers/PP)/num_decoder_layers and num_microbatches as already defined (ub-based); thrpt = 1000batch_size / (stage_latency(PP-1+num_microbatches)). Requires plumbing num_decoder_layers (available via model_df Repeat dims = max repeat * layers, already extracted as n_layers at llm_decode.py:170) and num_micro_batches (computed at top, currently only when PP>1 at line 25-26 — expose it). [2] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/llm_prefill.py:105-113 -> same correction; prefill num_microbatches from ub; Latency returned = prefill_latency (full model). [3] Add an inline comment at both sites: 'decode_latency is the FULL-MODEL forward (graph serializes all PP stages once incl Message Pass); single-request Latency is unchanged by PP. Throughput uses GPipe steady-state with ceil(L/PP) bottleneck-stage. No measured PP>1 calibration anchor yet — see audit_microbench_data.md.'
Impact: Affects ONLY pipeline_parallel>1 results. Returned Latency for PP>1 DROPS by factor (2-1/PP)/1 (e.g. PP=2: was 1.5x, now 1.0x) -> reported single-request latency becomes the correct full-model value. Throughput changes from 1000B/(full(2-1/PP)) to the GPipe steady-state; for large num_microbatches throughput RISES toward 1000*B/stage_latency. parallelism_optimizer.py impact: the original design CLAIMED the optimizer re-ranks PP via these Throughput fields — grep shows parallelism_optimizer.py does NOT call decode_moddeling/prefill_moddeling nor read Throughput; that coupling claim is WITHDRAWN as unsubstantiated. Real coupled consumers are any API/serving caller that reads ModdelingOutput.Latency/Throughput for PP>1 configs.
Regression risks: PP=1 path MUST be byte-identical: with PP=1, ceil(L/1)/L=1, stage_latency=decode_latency, num_microbatches=1, thrpt=1000B/(decode_latency(1-1+1))=1000*B/decode_latency — matches existing exactly. Verify this reduction algebraically and in test. Risk: any snapshot test asserting the OLD inflated PP>1 latency must be re-baselined (it was wrong). Risk: heavy cross-stage activation models — Message Pass is in full_model_latency; multiplying by ceil(L/PP)/L slightly under-counts comm in the bottleneck stage, documented as a known approximation pending a measured anchor.
Verification: BEFORE: algebraic assertion test: for PP=1 the new formulas equal the pre-fix scalar outputs (Latency and Throughput) exactly. For PP=2,L=16: stage_latency=full8/16=full/2, thrpt=1000B/(full/2*(2-1+nmb)). Assert returned Latency==full (not 1.5*full). REGRESSION: run decode_moddeling/prefill_moddeling with pipeline_parallel=1 on GB10/A100 over the grid and assert IDENTICAL Latency+Throughput to pre-fix. Add a skipped/xfail test documenting that PP>1 magnitude is unvalidated until a measured multi-GPU anchor is added to INFERENCE_BENCHMARKS.

M2-decode-kv-growth-cache

Root cause: The original design claimed removing the output_tokens>10 gate (llm_decode.py:116) would be 'absorbed by the commit-99784b8 decode cache'. The critique is correct that this is false: the final-step model is built at context input_tokens+output_tokens, a DIFFERENT context bucket than the initial model at input_tokens, so it is a genuine cache miss every call and would DOUBLE the GenZ build cost for the dominant 2<ot<=10 interactive regime. The current code (llm_decode.py:116-133) keeps the gate and uses a linear 1+0.1*ot/10 growth approximation for ot<=10.
Solution: WITHDRAW the gate-removal. The two-point (initial+final)/2 average is only needed when KV growth materially changes per-token latency, which for ot<=10 is negligible (KV grows by <=10 tokens on top of input_tokens, typically <1% memory-time change). Keep the gate. The correct fix is NOT to always build the final model, but to make the ot<=10 approximation physically grounded instead of the magic 0.1 factor: replace decode_latency = initial_latency*(1+0.1*ot/10) with an analytic KV-growth scaling derived from the actual KV memory-time fraction. Decode is memory-bound; the per-token memory time scales with (weights_bytes + kv_bytes(ctx))/BW. The midpoint context over the ot output tokens is input_tokens+ot/2, so the expected per-token latency = initial_latency scaled by (mem_bytes(input+ot/2)/mem_bytes(input)). This reuses already-computed quantities (KV cache bytes vs weight bytes from summary_table) — no extra get_model_df build, no cache miss, and replaces the unsourced 0.1 constant with a derived ratio. For ot>10 keep the explicit two-point build (one extra cached build, acceptable since the absolute token count is large and amortized).
Constants & sources: (1) KV-growth scale = mem_bytes(input+ot/2)/mem_bytes(input) where mem_bytes = model_weights + kv_cache(ctx). SOURCE: closed-form roofline (decode is memory-bound; latency proportional to bytes moved/BW), derived. Replaces the unsourced 0.1 magic factor at llm_decode.py:133. (2) kv_cache(ctx) is linear in ctx (KV grows 1 token/step): SOURCE: standard transformer KV-cache derivation, already computed in summary_table['KV Cache'].
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/llm_decode.py:131-133 -> replace decode_latency = initial_latency * (1.0 + 0.1 * output_tokens / 10) with a derived scale: compute kv_per_token = kv_cache / input_tokens (from initial_summary), weights = model_weights; scale = (weights + kv_cache + kv_per_token*output_tokens/2) / (weights + kv_cache); decode_latency = initial_latency * scale. (memory-bound midpoint). [2] Keep the output_tokens>10 gate and the two-point average at llm_decode.py:116-130 unchanged (do NOT remove the gate).
Impact: Affects decode TPOT for 1<output_tokens<=10 only. Replaces a flat +0..+0.1 bump with a context/KV-aware scale that for small models (KV<<weights) is ~1.0 (correct: weight-dominated decode barely changes) and for long-context (KV comparable to weights) scales up correctly. No extra GenZ build -> no cache regression (critique's perf concern is fully avoided). ot>10 path unchanged. ot==1 path unchanged.
Regression risks: Numeric change for 1<ot<=10 decode latencies (was tied to 0.1 magic). For weight-dominated small-batch decode the new value is very close to initial_latency (slightly lower than the old +0.1 bump) — more correct. Risk if input_tokens==0 (division) -> guard kv_per_token with input_tokens>0 else 0. Risk: the GB10 calibration grid uses ot=160 (>10 path), so this ot<=10 change does NOT touch the calibrated points — safe for calibration but ALSO means it is not validated by the existing grid (see verification).
Verification: BEFORE: unit test that for a small model (kv_cache << weights) scale ~= 1.0 +- 1%; for a long-context config (kv_cache ~ weights) at ot=10 the scale increases proportionally to kv growth. ASSERT no extra get_model_df call for the 2<ot<=10 path (mock/spy get_model_df call count == initial build only). REGRESSION: assert ot==1 and ot>10 decode latencies byte-identical to pre-fix. Because no measured ot<=10 grid exists, ADD a measured short-generation point (ot=8) to INFERENCE_BENCHMARKS for GB10 in a follow-up and assert within +-7%; document this gap explicitly rather than claim validation.

L5-weight-vs-kv-precision-separation

Root cause: The summary referenced separating weight precision from KV-cache precision but provided no concrete solution. Currently System.bits (system.py:202-218) applies one self.bits to BOTH weights and KV via get_bit_multiplier; quantized-weight + higher-precision-KV (common: int4/fp8 weights, fp16 KV) cannot be expressed except via the gear path. The critique correctly notes any int4-dequant-overhead claim needs a sourced term across MQA/GQA/MLA layouts.
Solution: Two options, choose the conservative one. OPTION A (full, deferred): add a separate kv_bits parameter to System and route get_operators_size (operator_base.py:255-271) to use kv_bits for the 'k'/'v' data paths (Logit K, Attend V) while weights use bits. This is GENERALIZABLE across MQA/GQA/MLA because it keys off the existing data='k'/'v' tagging, not the attention layout. OPTION B (the int4 dequant overhead) is DEFERRED: there is no sourced, generalizable constant for int4 dequant overhead on hardware without a native int4 path — it depends on whether dequant is fused into the GEMM epilogue (negligible) or a separate pass (memory-bound add). Rather than invent a constant, DEFER the dequant term and document that it must be MEASURED per-arch (e.g. a marlin/AWQ-kernel microbench) before inclusion. Ship only OPTION A (precision routing), which adds no magic constant — it reuses the existing mem_multiplier table (system.py:39-69) keyed by the kv_bits string.
Constants & sources: (1) No new numeric constant. kv_bits reuses System.mem_multiplier (already-sourced bytes-per-element per dtype, system.py:39-69). (2) int4 dequant-overhead term: EXPLICITLY DEFERRED — no sourced/derived value exists; must be measured (marlin/AWQ kernel microbench) per-arch before adding. Stated as a known gap, not guessed.
Code changes: [1] llm-memory-calculator/src/llm_memory_calculator/genz/system.py:70-92 -> add kv_bits=None to init; self.kv_bits = kv_bits or bits. [2] llm-memory-calculator/src/llm_memory_calculator/genz/system.py:202-218 get_bit_multiplier -> when type=='M' and data in ('k','v'), use self.mem_multiplier[self.kv_bits] instead of self.bits (preserve gear branch). [3] llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py / decode/prefill signatures -> thread an optional kv_bits param (default None == bits, byte-identical).
Impact: When kv_bits is unset (default), self.kv_bits=bits -> get_bit_multiplier returns identical values -> byte-identical to today for ALL existing calls. When set (e.g. weights int4, kv fp16), KV-cache memory-time in Logit/Attend ops increases relative to int4, correctly modeling mixed precision. Couples to KV-cache memory reporting (summary_table KV Cache bytes) and decode memory-bound latency.
Regression risks: Zero for default path (kv_bits=None). Risk only for explicit mixed-precision callers, which is new functionality. The deferred int4-dequant term means int4-weight prefill on non-native-int4 hardware may be slightly optimistic — documented as a known gap, not silently wrong (no fabricated constant).
Verification: BEFORE: assert get_bit_multiplier(type='M', data='k') with kv_bits=None equals the bits-based value (byte-identical). With bits='int4', kv_bits='fp16': assert K/V multiplier == mem_multiplier['fp16']=2 while weight multiplier == 0.5. REGRESSION: full grid decode/prefill with no kv_bits arg is byte-identical to pre-fix on GB10/A100.

3.9 Remaining findings (gap-fill — the revise stage consolidated to top-severity per unit; these were verified covered or are authored here with the same rigor)

Coverage audit (done after the workflow): of 39 findings, the 8 units produced dedicated entries for 23 and folded C5/C7 into F1/F2. The items below were either dropped during consolidation or need explicit homing. M14/L0 were correctly dropped from the efficiency unit (wrong unit, no anchor in that scope) and are re-homed here with their measured/datasheet anchors.

⚠ Correction (verified against code/DB, per §4 synthesis): the DB has no flops=0 placeholders — the schema enforces flops REAL NOT NULL CHECK(flops > 0) (verified), and real_values splits 41 (=0, "not vendor-reconciled") vs 51 (=1) (verified). So C5/C6/C7 must flag real_values=0 devices as estimated metadata and let them simulate (surfaced as uncalibrated/optimistic) — NOT hard-gate them (which would break 41 devices' endpoints for no physics reason). The text below is corrected to that conclusion.

C5 (18 devices unsimulatable) — covered by F1+F2, made explicit. Root issue: the engine reads the static configs (set_hardware_db_path is exported but never called in the inference path — verified), which know ~74 devices, while the catalog serves 92. F2 (engine resolves from the same DB) makes every catalog device simulatable; F1 marks real_values=0 ones estimated so their numbers are surfaced as uncalibrated rather than silently trusted. Verification: assert for every catalog device: get_inference_system(name) succeeds (no KeyError/404), and real_values=0 results carry estimated:true. Regression: the 51 verified devices resolve with identical specs post-F2 (full pre/post spec snapshot).

C6 (Intel Max/PonteVecchio use FP32 peak as bf16) — folded into F1. Same mechanism as the sparse fix: set dense bf16 XMX peak from datasheet — Max 1550 ≈ 839 TFLOPS, Max 1100 ≈ 362 TFLOPS (Intel Data Center GPU Max datasheet; 45/52 were the FP32/FP64 peaks). Until hand-checked against the datasheet they remain real_values=0 → surfaced as estimated, still simulatable. No invented value.

C7 (duplicate/conflicting rows) — folded into F1. Dedupe to one canonical row per accelerator (dense bf16, per-chip). Where two rows disagree (MI300X 1307 vs MI300X_GPU 1600; V100 125 vs 157), keep the datasheet-correct one as real_values=1 and mark the other real_values=0 (estimated, still usable). Verification: a test asserting no two real_values=1 rows map to the same physical accelerator with different Flops/BW.

H4 (usecase create 500 on null SLOs). RCA: a min≤max validation compares Optional[float] SLO bounds without a None guard → None > None. Solution: guard each min/max pair (if lo is not None and hi is not None and lo > hi: raise 422); a missing bound is valid (unconstrained). Constants: none. Impact: only usecase create/update validation. Regression: usecases with both bounds set must still validate identically; add a test for all-null and one-sided SLOs. Verify: POST /api/usecases with null ttft/e2e returns 200; with lo>hi returns 422 (not 500).

M5 (/models/popular?limit unbounded/negative). Solution: limit: int = Query(10, ge=1, le=100) (bound matches the other list endpoints — derived from existing convention, not arbitrary). Impact: caps the HF-metadata fan-out. Regression: default limit=10 unchanged. Verify: ?limit=-5→422, ?limit=100000→422, ?limit=10→10 items.

M6 (serving length clamp min>max collapse). RCA: serving.py:282-291 hardcodes min:32/8 while max=mean*4, so max<min for small means → np.clip collapses all samples to max. Solution: make the floor consistent: lo=min(32, mean); hi=max(mean*4, lo) (or validate mean≥floor). The clamp must satisfy lo≤hi by construction — a derived invariant, no new constant. Impact: only the synthetic workload generator. Regression: for mean≥8 the bounds are unchanged. Verify: with input_length_mean=4, sampled lengths span a real distribution around 4 (not all-identical), and lo≤hi asserted.

M7 (training can't resolve GenZ model names; HF 401→404). Solution: in _get_model_config, resolve via the GenZ registry first (as serving/get_configs does), then fall back to HF; map HF auth/network failures to 502/400, not 404 (404 means "model unknown", which is false for a gated-but- real model). Constants: none. Impact: training endpoints accept the same identifiers as serving. Regression: HF-fetchable ids (gpt2) still work. Verify: check-fit model=llama2_7b→200 (not 404); a gated model w/o token→502 with a clear message.

M12 (check-fit returns memory_per_gpu_gb=0.0 when not-fit). RCA: cluster_selector._find_best_parallelism returns the default 0.0 when no parallelism fits. Solution: populate the field with the minimal achievable per-GPU memory (total / max_shardable) even on the not-fit path, so the typed field matches the prose reason. Impact: response field only. Verify: num_gpus=1 full-FT 7B → memory_per_gpu_gb≈152.9 (matches reason), fits=false.

M13 (three hardware sources). Solution: subsumed by F2 — training's cluster_selector and its GPU_COST_PER_HOUR must read the same single DB source (cost from the per-cloud cost.* block already in configs, not a parallel dict). Verify: a test asserting training, inference, and catalog return identical Flops/BW/cost for every device.

L4 (nonexistent model → "MODEL_GATED"). Solution: distinguish 404 (HF returns not-found) from 403 (HF returns gated/401) in /models/calculate error mapping. Verify: a clearly-nonexistent id → 404 "not found"; a real gated id → 403 "gated".

M10 (catalog Sort/Order/FLOPS filters silently ignored). RCA: hardwareAPI.ts:94-101 sends sort_by/sort_order/min_flops/max_flops to GET /api/hardware, which only accepts type/manufacturer/min_memory/max_memory/limit/offset (hardware.py:203-211); FastAPI drops unknowns. Solution (lowest-risk): repoint the catalog UI to the existing /api/hardware/filter endpoint, which already supports these params (no backend change); OR add the four params to the list endpoint mirroring /filter. Constants: none. Impact: UI only. Regression: default (no filter) list unchanged. Verify: set Sort=Performance desc → list reorders by Flops; set min FLOPS → list shrinks.

M11 (fabricated pagination count). RCA: HardwareList.tsx:61-66 invents a total because the list endpoint returns none. Solution: return total_count from GET /api/hardware (the DB query already computes it; /filter and other list endpoints already return counts), and compute totalPages from it. Constants: none. Impact: pagination display only. Regression: single-page results unchanged. Verify: with >12 devices, the pager shows the true page count and the last page is non-empty.

L6 (UI parameter_count NaN). Solution: guard AIMemoryCalculator.tsx:1667,2092: results.parameter_count != null ? (results.parameter_count/1e9).toFixed(1)+'B' : '—'. Verify: a model with null param count renders "—", not "0.0B"/"NaNB".

M14 (power workload-insensitive) — re-homed with measured anchor. RCA: accelerator power is a fixed 0.4125·TDP step, not utilization-driven (verified: bs=1 and bs=32 both 288.75W). Solution: P_accel = P_idle + utilization·(TDP − P_idle) where utilization is the roofline-derived max(compute_util, memory_util) already computed per op in get_roofline (operator_base.py:495), and P_idle/TDP come from the device's vendor spec. Measured anchor (this box): GB10 idle 7.2 W, 96%-util load 73 W → the linear model fits (7 + 0.96·(TDP_gpu−7)); refute the constant model. Constants: P_idle, TDP per device — vendor-sourced (GB10 idle measured 7.2 W). No magic fraction. Impact: energy/token and cost now track batch/context. Regression: must reproduce the GB10 measured idle→load curve; a "power monotonic in utilization" invariant test. Verify: bs=1 vs bs=32 power must differ in the modeled direction; cross-check a measured nvidia-smi power.draw sweep vs predicted.

L0 (tf32 multiplier=1) — re-homed with datasheet anchor. Solution: set System.compute_multiplier['tf32']=2 (system.py:24). Constant source: NVIDIA A100 datasheet — TF32 tensor = 156 TFLOPS vs BF16 = 312 TFLOPS, exactly 2× (the multiplier is time, so 2× slower). Generalizes: TF32 is half-rate vs bf16 on every NVIDIA tensor-core arch (Ampere→Blackwell). Impact: only bits='tf32' paths (rare for inference). Regression: bf16/fp16/fp32 multipliers unchanged; add a test tf32_prefill ≈ 2× bf16_prefill on a compute-bound case. Verify: prefill bits=tf32 is 2× bits=bf16 (currently equal).

4. Cross-coupling, rollout order, and master regression suite

Independently re-verified (this session) — the synthesis's 3 load-bearing claims all hold: (1) DB schema = flops REAL NOT NULL CHECK(flops > 0), real_values = 41 (0) / 51 (1) ✓; (2) RTX4090 DB=661 (sparse, real_values=0) vs static=330 (dense) → the F1×F2 inverted regression is real ✓; (3) set_hardware_db_path is exported but never invoked in the inference path (engine reads static configs) ✓. The plan below is therefore trustworthy; proceed in the stated order.

I have all the ground truth I need, verified against the actual code and DB. Here is the plan of record.

BudSimulator Accuracy-Fix — Cross-Coupling Synthesis (Plan of Record)

All numerics below are either (a) traced to a vendor datasheet, (b) the MEASURED GB10 grid in audit_microbench_data.md, (c) a published MFU/MBU range with citation, or (d) closed-form. Every code site is file:line verified against the working tree this session.

0. Verified ground-truth corrections to the unit summaries (read this first)

Three unit claims are WRONG or imprecise against the actual code/DB. The plan is built on the verified facts, not the summaries:

The DB has NO flops=0 placeholders. BudSimulator/data/prepopulated.db hardware table has a hard CHECK(flops > 0) constraint; min(flops)=17.2, max=62500 (Cerebras_WSE3, a real spec). The split is real_values=1 (51 devices) vs real_values=0 (41 devices). So unit-1's "19 placeholder devices with garbage zero flops, gate them out" is mis-stated: real_values=0 means not vendor-reconciled / heuristic, not zero. The F1 action is therefore "audit + reconcile to dense bf16; flag real_values=0 as estimated in API metadata", NOT "gate out of simulatability." Gating them out would break 41 devices' endpoints for no physics reason.
The C1×B regression is real but device-asymmetric and INVERTED between H100 and RTX4090 — verified:
- Static configs.py: H100_GPU.Flops=1979 (sparse), RTX4090.Flops=330 (dense FP16, correct).
- DB: H100_GPU=989 (dense, correct), RTX4090_GPU=661 (sparse, WRONG).
- The engine today reads static configs (set_hardware_db_path is never called at model-build time — grep-confirmed; it exists in hardware/manager.py:537 and systems/system_configs.py:18 but is not invoked in the inference path).
- So wiring the DB (F2) without F1 fixes H100 (1979→989) but breaks RTX4090 (330→661). The two move in opposite directions. This is the single most dangerous coupling and is why F1 must precede F2.
The "dead code" is only HALF dead — verified by tracing get_dimensions():
- GEMM.get_dimensions() returns [self.get_tensors()] — a list (operators.py:103). At operator_base.py:416 isinstance(dims, tuple) is False ⇒ get_tensor_core_efficiency never runs ⇒ returns flat 0.85 (line 434). At line 453 isinstance(dimensions, tuple) is False ⇒ get_shape_dependent_op_intensity never runs ⇒ op_intensity = num_ops/num_data. Fully dead for GEMM/FC.
- Logit/Attend inherit base get_dimensions() → returns get_tensors() = a 3-tuple (input_a, input_w, output) (operator_base.py:212). At line 427 len(dims)>=4 is False (len 3) ⇒ flat 0.85. BUT at line 453 len>=3 is True, so get_shape_dependent_op_intensity IS called — then its own len(dimensions)>=6 guard (line 144) is False (len 3) ⇒ returns the constant 100 into the Op Intensity column. So for attention ops the shape-intensity path is NOT dead: it writes a hardcoded 100 that budevolve's roofline_analyzer.py consumes. Deleting it changes Op Intensity for attention from 100 → true num_ops/num_data. This is a behavior change for budevolve and must be regression-pinned, not assumed inert.
Net: the flat 0.85 (tc_eff) is applied to every compute op (GEMM, FC, Logit, Attend) — confirmed via line 465–469 in the live get_roofline. Removing it multiplies every compute-bound op's compute_time by 1/0.85 = 1.176×.
The compute path (verified): op_per_sec = flops/2 (system.py:100), num_ops = 2·MACs (operator_base.py:445), GEMM MACs = B·M·N·K (operators.py:123-125). So compute_time = 2·MACs / flops_raw, then /= (compute_efficiency · tc_eff) (operator_base.py:468-469). Effective achievable TFLOPS = flops_raw · eta_compute · tc_eff. This closed form is the backbone of every interaction below.

1. INTERACTION MATRIX

Sign convention: + = increases latency (slower prediction), − = decreases latency (faster). Magnitudes are on compute-bound prefill unless noted; decode (memory-bound, bs=1) is byte-identical under every FLOP/tc_eff/causal fix because the compute term loses max(compute,memory).

Pair	Sign	Magnitude (prefill, compute-bound)	Interaction type	Resolution
F1 dense-FLOPS × F2 DB-wiring	device-dependent	H100 `+2.0×` (1979→989); RTX4090 would be `−2.0×` (330→661) if F2 ships first	Inverted regression	F1 lands first; F2 only after a full pre/post spec snapshot asserts no device's `flops` moves in the wrong direction.
F1 dense-FLOPS × C4/L3 tc_eff removal	`+ × +` compound	F1 `2.0×` (H100) AND tc_eff `1.176×` ⇒ `~2.35×` combined on H100 prefill	Compound (both correct)	Both measured in the post snapshot; attribute `2.0×` to F1 and `1.176×` to C4 separately, never lump.
C1/F1 real-MFU × C causal-mask	`+ × −`	F1+C4 `+2.35×`; causal halves attention FLOPs ⇒ attention `compute_time ×0.5`; attention ≈56% of prefill compute at 8k ⇒ total `~0.7×`	OFFSETTING — the headline coupling	Co-calibrate (see §1a). NET pre-refit prefill at 8k ≈ audit's −31% gap; eta_compute refit closes it.
C4 tc_eff removal × B eta_compute	`− × (refit)`	tc_eff removal makes uncalibrated prefill `1.176×` faster; GB10's fitted `0.096` was fit on top of 0.85	Same-change requirement	C4 and GB10 eta refit MUST land in ONE commit. `eta_compute' = 0.096 × 0.85 = 0.0816` to hold `250·eta·tc = 20.4 TF` invariant (see §1b).
C causal-mask × B eta_compute	`− × (refit)`	causal −50% attention FLOPs; eta_compute absorbs the FLOP-counting convention (audit L100-106: full→−20%, causal→−31% at fixed eta=0.34)	Ordering	Structural causal fix lands BEFORE eta refit; refit against corrected (causal) structure.
C4 attention `Op Intensity` 100→true × budevolve	n/a (different consumer)	`Op Intensity` for Logit/Attend changes from constant `100` to `num_ops/num_data` (decode ~1–8, prefill ~hundreds)	Downstream semantic	Pin `roofline_analyzer.py` `arithmetic_intensity` outputs in a golden test; this is the ONLY non-dead piece of the "dead code."
F1 placeholder/`real_values=0` flag × power/cost endpoints	metadata only	41 devices flip to `estimated` metadata; numbers unchanged	Surfaced behavior change	API marks `estimated:true`; no numeric regression.
F1+C4 × power model (compute_utilization)	`+`	higher `compute_time` on prefill raises `compute_util` ⇒ energy/token rises	Coupled, re-snapshot	Power endpoint re-snapshotted; not assumed unchanged. M14 0.4125·TDP is DROPPED (see §5).
C causal-mask × sequence-parallel sp>1	correctness bug	naive `f=1−(M−1)/(2N)` goes NEGATIVE (S=4096, sp=4 ⇒ f=−1.0)	Hard correctness	Derive `f` from LOGICAL causal geometry per builder, clamp `[0,1]`; never from confounded sharded M,N.
C causal-mask × SWA/sliding-window	`−` (deferred)	windowed causal count deferred; `*_local` uses capped-N (upper-bound-safe under-count)	Deferred dependency	Until SWA unit lands, `*_local` over-counts safely (conservative).
F2/F4 parallelism_optimizer × F1 DB	inherit	F4 repoints `HARDWARE_CONFIGS` at same DB	Delegation	F4 after F1+F2; verify optimizer picks identical strategies pre/post for already-correct devices.
t_launch (calibration) × decode/prefill	`+` small-op regime	decode adds `t_launch·n_ops`; prefill adds `t_launch·count_repeat_aware_ops`	Additive, guarded	Default 0.0 ⇒ byte-identical for uncalibrated. Verified llm_decode.py:166-172, llm_prefill.py:101-102.

1a. The offsetting coupling resolved precisely (C real-MFU × C causal — "the whole game")

The danger: causal-mask (faster) and real-MFU (slower) can cancel, letting a wrong eta_compute hide a wrong FLOP count — "gaming." The proof they can't both be free is in the audit (L100-106): at fixed eta=0.34, the gap is −20% with full-S² and −31% with causal-S²/2. That −11pp swing is the pure structural signal. Therefore:

Land causal-mask FIRST so the FLOP count is physically correct (lower triangle). The exact factor is closed-form: for a logical causal square of query length Sq over key budget Sk with Sk≥Sq, computed-fraction f = (Sq·Sk − Sq(Sq−1)/2) / (Sq·Sk); for the self-prefill square Sq=Sk=S ⇒ f = (S+1)/(2S) → 0.5. Derive from LOGICAL S (input_sequence_length), not the sharded N=S//sp, then clamp [0,1].
Then fit eta_compute by differential_evolution against measured TTFT. Because the structure is now correct, the fitted eta is the true achievable-MFU-vs-peak ratio and is device-portable. If someone later "fixes" the FLOP count again, eta no longer matches measurement → the regression test fails. The grid (multiple contexts 0.5k/2k/8k) over-determines eta (1 free param, ≥3 measured points), so eta cannot silently absorb a per-context structural error — a structural error shows as a context-dependent residual the fit cannot remove. That is the anti-gaming guarantee.

NET prefill TTFT target after both: GB10 within ~10% RMS of measured (26.1/100.6/490.7 ms at 0.5k/2k/8k), vs −68% to −72% at today's 250·0.85 baseline.

1b. The double-correction arithmetic resolved precisely (C1/F1 dense-FLOPS × B eta_compute)

The invariant the user demands: peak · eta_compute · (tc factor) must equal MEASURED achievable.

GB10 measured large-GEMM achievable = 85 TFLOPS (audit §2). Configured peak = 250.
TODAY (small-prefill block): effective = 250 · eta_compute(0.096) · tc_eff(0.85) = 20.4 TF. This matched small-context prefill (the launch-heavy ~27% MFU regime), with the 0.85 already baked in.
AFTER C4 removes 0.85: to hold the same physical achievable, refit gives eta_compute' = 0.096·0.85 = 0.0816. One commit.
For large-GEMM steady state, the same harness against the M=4096 point (83.8 TF) yields eta_compute_large = 83.8/250 = 0.335. The prefill/decode split + t_launch let one device carry both regimes; differential_evolution fits them jointly — no hand-typing.
For non-GB10 HBM datacenter GPUs after F1 sets the dense peak: published cuBLAS/CUTLASS large-GEMM MFU is 0.7–0.85 of dense peak [NVIDIA CUTLASS profiling; cited in get_tensor_core_efficiency docstring lines 72] ⇒ eta_compute ≈ 0.75–0.85. The roofline's max(compute,memory) with arithmetic intensity handles the small-M falloff automatically (audit §3), so a single steady-state eta per device is correct — no shape table.

Guard against double-correction: assert flops_raw · compute_efficiency (post-C4, no tc_eff) at the M=4096 GEMM reproduces the measured 83.8 TF within tolerance, AND that no tc_eff multiply remains anywhere (grep assert TC Effcy removed / pinned to 1.0).

1c. Dead-code removal safety (C4/L3 × B eta) — roofline alone reproduces the shape curve

Audit §3 is the proof: measured throughput(AI) = min(AI·205 GB/s, 85 TFLOPS), ridge AI≈419, and GenZ's per-op max(compute_time, memory_time) (operator_base.py:479) IS that roofline. Verification before removal: run the operator on the audit's M-sweep (M=1,128,512,4096 at N=K=8192) with tc_eff forced to 1.0 and eta=1.0; assert predicted TFLOPS tracks min(AI·BW, peak) (which equals the measured curve once eta_mem=0.73, eta_compute=0.335 scale BW and peak). If it tracks, the flat 0.85 and the get_shape_dependent_op_intensity add nothing for GEMM ⇒ safe to delete. For attention, separately pin the new Op Intensity (true ratio, not 100).

2. UNIFIED CALIBRATION PROTOCOL

One generalizable procedure, run by validation/inference_calibration.py (fit_inference_calibration, line 113), AFTER all structural fixes land.

Free parameters per device, per phase: {eta_mem, eta_compute, t_launch_ms, c_stream_ms_per_layer} (decode carries c_stream; prefill omits it — single pass, per llm_prefill.py:99). Fitter: scipy.optimize.differential_evolution (already wired).

Data that feeds it (in priority order, each tagged with its source class a/b/c/d):

(b) Measured grid if the device is in-house: real decode TPOT + prefill TTFT over batch ∈ {1,2,4,8} × ctx ∈ {0.5k,4k,12k} (the GB10 14-point grid). This is the keystone and the only fully-measured device today.
(c) Published MBU/MFU for devices with literature: vLLM/TensorRT-LLM/llama.cpp benchmark tables → derive achievable TPOT/TTFT. Cite the benchmark in the config block comment.
(a)+(d) Anchored-by-class for un-measured devices: do NOT hand-type a derate. Instead set the bounds/prior of differential_evolution from the device's memory technology and architecture, then leave eta=1.0 (honest over-prediction) if no measured/published point exists:
- eta_mem prior by memory tech (vendor STREAM-style efficiency): HBM2e/HBM3 0.80–0.90, LPDDR5X ~0.75 (measured here), GDDR6/6X 0.70–0.80. Source per device from vendor/STREAM number.
- eta_compute prior by tensor-core gen + the published cuBLAS/CUTLASS 0.70–0.85 large-GEMM MFU band.
- t_launch floor: kernel-launch is host/driver-bound, ~5–10 µs/op (GB10 measured 8.26 µs; decode block 4.56 µs). Use 8 µs as the cross-device prior, refine if measured.

Hard rule compliance: every shipped number is either a measured ratio, a fitted ratio against measured/published data, or eta=1.0 (explicit, surfaced "uncalibrated/optimistic" metadata). NO per-device magic constant is hand-typed. The config block carries a calibration_source field: one of measured | published:<cite> | uncalibrated.

Procedure:

Strip any existing inference_calibration (line 94-99 already does this) → fit against raw roofline.
Fit decode {eta_mem, t_launch, c_stream} against the memory-bound grid; fit prefill {eta_compute, eta_mem, t_launch} against the compute-bound grid.
Assert physical invariants on the fit: eta_mem ≤ 1, eta_compute ≤ 1 (MFU/MBU ≤ 1 — see §3), residual RMS reported.
Emit the ready-to-paste block (line 178-182). Config edits stay manual (no auto-write to configs.py).

3. MASTER REGRESSION SUITE (golden tests — all must pass before merge)

Tuples are (model, hardware, precision, parallelism, phase).

A. Physical-limit invariants (every device, every model):

MBU = achieved_BW / peak_BW ≤ 1.0 and MFU = achieved_FLOPS / dense_peak ≤ 1.0. Compute from the engine's own Throughput column. Today H100-static gives implied MFU ~155% (physically impossible) → after F1+C4 must be ≤1.0 and ~0.665 on the Llama-2-7B prefill anchor.
eta_mem ≤ 1, eta_compute ≤ 1 for every calibration block.

B. Decode byte-identical guard (the strongest regression):

For ALL 92 devices, bs=1 decode latency must be byte-identical pre/post the FLOPS/tc_eff/causal changes (these touch only the compute term that loses max() in memory-bound decode). Snapshot decode latency for {Llama-3-8B, Qwen2.5-7B, Mixtral-8x7B} × {H100,A100,RTX4090,GB10,MI300X} × bf16 × TP1 and diff to zero. Exception: GB10 decode changes by the documented eta_mem refit only (assert it matches the new fitted value, ~±7% vs measured 15.72/15.97 ms).

C. Prefill known-direction/magnitude:

H100 prefill: must increase ~2.35× (F1 2.0× × C4 1.176×) for short ctx where attention share is low; smaller net at long ctx where causal −50% offsets. Assert implied MFU drops from ~155% to ~0.66 vs 989.5 dense peak.
GB10 prefill (KEYSTONE): within ~10% RMS of measured TTFT 26.1/100.6/490.7 ms at 0.5k/2k/8k. Today's baseline is −68% to −72%.
RTX4090 prefill: must stay at dense 330 at the engine (must NOT jump to 661). This is the F2-without-F1 regression sentinel.

D. The 7 parameter families unchanged (memory math has no FLOPS dependence):

weights, KV-cache, activations, embeddings, attention-param, FFN-param, total-footprint: byte-identical for all snapshot tuples. Memory time = bytes/BW, untouched by any compute fix.

E. Single-node TP≤8 communication byte-identical:

For TP ∈ {1,2,4,8} on one node, AllReduce/AllGather/All2All comm bytes and comm_time must be byte-identical (no fix touches collectives). Assert on {Llama-3-70B} × {H100×8} × {TP2,TP4,TP8}.

F. budevolve Op Intensity pin:

roofline_analyzer.py:65 OperatorInsight.arithmetic_intensity for attention ops changes from constant 100 to true num_ops/num_data. Pin the NEW values in a golden test; assert GEMM/FC Op Intensity unchanged (was already num_ops/num_data).

G. Causal-mask correctness:

Assert causal factor f ∈ [0,1] for all (S, sp) including S=4096, sp=4 (must be 0.5, not −1.0). Assert bidirectional/encoder/vision prefill keeps f=1 (plain Logit/Attend, no _PREFILL optype). Assert chunked builder uses its own valid 1−(M−1)/(2N) with M≤N.

Keystone: the GB10 end-to-end decode AND prefill validation (B-exception + C) is the single test that proves the whole stack — measured hardware, real model, full pipeline.

4. SAFE ROLLOUT ORDER (dependency-correct, each step independently verifiable)

Step 0 — Lock invariant + byte-identical tests (no production code change). Land suite A, B, D, E as the safety net FIRST. They must pass on main (capturing today's behavior, including the wrong H100 MFU as a recorded baseline, not an assertion). Byte-identical guard armed.

Step 1 — F1: Audit & reconcile DB FLOPS to dense bf16; flag real_values=0 as estimated metadata. Verify: pre/post full spec snapshot over all 92 devices; H100_GPU stays 989, RTX4090_GPU 661→330 (or DB corrected to dense), Cerebras stays its real spec. NOT wired to engine yet → engine still reads static configs → decode/prefill byte-identical (B guard). This step is pure data + API metadata.

Step 2 — C (causal-mask) structural fix. Add Logit_PREFILL/Attend_PREFILL optypes from causal decoder builders; derive f from logical geometry, clamp [0,1]; scale num_ops AND op_intensity consistently in the operator. Verify suite G. Decode byte-identical (B). Prefill moves in known direction (C, attention −50%).

Step 3 — C4/L3 dead-code removal + GB10 eta refit (ONE commit — same-change requirement). Delete get_tensor_core_efficiency, get_shape_dependent_op_intensity, get_tensor_core_efficiency_factor, and the tc_eff multiply (operator_base.py:60-165, 397-434, 465-470, 509). Refit GB10 (eta_compute 0.096→~0.082). Verify §1c roofline reproduces measured shape curve; suite F pins new Op Intensity; GB10 keystone within RMS. Decode byte-identical except GB10's documented eta_mem.

Step 4 — F2: wire DB as single source (set_hardware_db_path at startup). ONLY now, after F1 corrected the DB. Verify: full spec snapshot — every device's effective flops identical to the Step-1 audited value; RTX4090 sentinel (C) holds at dense; no device regresses. Re-run A (MFU≤1 everywhere).

Step 5 — F4: repoint parallelism_optimizer.HARDWARE_CONFIGS at the same DB. Verify optimizer picks identical strategies pre/post for already-correct devices; new strategies only where flops were corrected.

Step 6 — Calibration rollout (§2) for published/measured devices. Per-device blocks, each guarded; uncalibrated stays eta=1.0 with surfaced metadata.

Step 7 — Power re-snapshot + UI metadata. Re-snapshot energy/token on compute-bound prefill (now higher compute_util). Surface calibration_source and estimated flags in API/UI. (M14 0.4125·TDP and L0 tf32-flip are DROPPED — see §5.)

Byte-identical-guarded at every step: decode (B) at Steps 1–2,4–5; the only intended decode change is GB10 at Step 3.

5. RESIDUAL RISKS / OPEN QUESTIONS (need real measurement)

GB10 is the ONLY measured device. Every other eta_compute/eta_mem is a published-band prior or 1.0. The plan is honest about this (surfaced metadata), but cross-device portability of the prefill/decode eta split is unproven off-GB10. Need ≥1 HBM datacenter measurement (H100 or A100) to validate the 0.75–0.85 large-GEMM band before claiming calibrated accuracy there.
eta_compute regime ambiguity. GB10 small-prefill fits ~0.082 (post-C4) but large-GEMM steady-state is 0.335. A single per-phase eta + t_launch must span both via the roofline + launch floor; if real models live between regimes, residuals may exceed 10%. Needs the expanded multi-model GB10 grid (more sizes) the audit calls for.
DROPPED units M14 (power 0.4125·TDP) and L0 (tf32 multiplier flip) are un-anchored and out of scope. Power is replaced by the measured-grounded P ≈ P_idle + util·(P_max−P_idle) (audit §7: idle 7.2 W, 96%-util 67–73 W, 35%-util 44 W — linear), but P_idle/P_max per device need vendor idle+TDP, not yet sourced for most devices. tf32 paths left unchanged until measured.
real_values=0 (41 devices) reconciliation. Flagging as estimated is safe, but whether their flops are sparse vs dense is unknown per device. The F1 audit must hand-check each against a datasheet; un-auditable ones stay estimated and uncalibrated.
SWA/sliding-window exact causal count is deferred; *_local over-counts (conservative, safe) until the SWA unit lands. Quantify the over-count for Mistral/Gemma-class windowed models.
budevolve Op Intensity consumers beyond roofline_analyzer.py:65 — confirmed one consumer; a broader grep for arithmetic_intensity/Op Intensity readers across budevolve/ is needed before Step 3 to ensure the 100→true change has no other dependents.

Key files for implementation (all absolute):

/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/genz/operator_base.py (lines 60-165 dead funcs, 397-434 tc_eff, 453-470 roofline apply, 509 column) — C4/L3
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/genz/operators.py (94-191 GEMM/Logit/Attend) — causal num_ops scaling
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/genz/system.py (100, 111-116) — flops/eta application
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/utils.py (149-170) — calibration block read
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/genz/LLM_inference/llm_prefill.py (101-102), llm_decode.py (166-172) — t_launch/c_stream
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/hardware/configs.py (149,184,210,236,317-353,412) — F1 dense FLOPS, GB10 block
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/hardware/manager.py (537 set_hardware_db_path) — F2 wiring
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/parallelism_optimizer.py — F4
/home/bud/ditto/simulator/llm-memory-calculator/src/llm_memory_calculator/validation/inference_calibration.py (113-182) — calibration harness
/home/bud/ditto/simulator/BudSimulator/data/prepopulated.db — F1 data audit (CHECK flops>0; 51 real_values=1 / 41 =0)
/home/bud/ditto/simulator/audit_microbench_data.md — measured anchors

FilesExpand file tree

solutions.md

Latest commit

History