perf: avoid redundant operator cache lookups

[baominghelly]

Summary

• Add a two-entry thread-local hot path in Operator::Call so repeated calls can match cached tensor metadata and scalar arguments without rebuilding a full CacheKey or probing the hash table.
• Define a distinct cache.match host-profiling range for the hot-cache probes. cache.lookup now covers only the hash-table try_emplace; cache.key and cache.construct continue to cover full-key materialization and operator construction respectively.
• Preserve the existing hash-table fallback, custom CacheKeyBuilder compatibility, cache-generation invalidation, and exception behavior; extend the focused C++ and PR feat: add host range profiling #817 profiling tests for the new path and range boundaries.

Motivation

Eager operator calls repeatedly rebuild the same cache key and perform an unordered_map lookup even when consecutive calls use identical metadata. For small operators this host-side work is measurable relative to kernel launch latency.

The hot path compares arguments directly against the two most recently used cache entries. It avoids full key construction and map lookup on common repeated/alternating call patterns while retaining the existing path for misses and legacy custom key builders.

Type of Change

• feat — new feature / new operator / new platform
• fix — bug fix
• perf — performance improvement (no behavioral change)
• refactor — code restructuring without behavior change
• test — adding or fixing tests only
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change (requires a ! in the Conventional Commits prefix or a BREAKING CHANGE: footer)

Platforms Affected

• CPU (WITH_CPU)
• NVIDIA (WITH_NVIDIA)
• Iluvatar (WITH_ILUVATAR)
• MetaX (WITH_METAX)
• Cambricon (WITH_CAMBRICON)
• Moore (WITH_MOORE)
• Ascend (WITH_ASCEND)
• PyTorch C++ bindings (WITH_TORCH)
• Build system / CMake / CI
• Python bindings / user-facing API

Smoke Test Result

A100-SXM4-80GB; isolated profiling ON/OFF builds used CMAKE_BUILD_PARALLEL_LEVEL=$(nproc), CPU, NVIDIA, and PyTorch support.

# Profiling OFF Release smoke wheel
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1 CUDA_VISIBLE_DEVICES=0 \
  python -m pytest tests -m smoke -q
107 passed, 23 skipped, 13306 deselected

# Focused cache behavior
python -m pytest tests/test_cpp_api.py::test_cpp_operator_cache_fast_path -q
1 passed

Test Results on Supported Platforms

Platform	Affected	Build / Smoke Result	Full Result / Notes
CPU	Yes	profiling ON and OFF builds passed; shared smoke suite passed	Full suite not run
NVIDIA	Yes	profiling ON and OFF builds passed; smoke suite passed on A100	Full suite not run
Iluvatar	Yes	Not tested	Hardware unavailable
MetaX	Yes	Not tested	Hardware unavailable
Cambricon	Yes	Not tested	Hardware unavailable
Moore	Yes	Not tested	Hardware unavailable
Ascend	Yes	Not tested	Hardware unavailable
PyTorch C++ bindings	Yes	WITH_TORCH=ON builds passed	Full suite not run
Python bindings	Yes	smoke and focused profiling suites passed	Full suite not run

PR #817 host-range profiling regression coverage:

# RelWithDebInfo, INFINI_OPS_ENABLE_HOST_RANGE_PROFILING=ON
ops._host_range_profile_compiled() == True
45 passed

# Release, INFINI_OPS_ENABLE_HOST_RANGE_PROFILING=OFF
ops._host_range_profile_compiled() == False
41 passed, 4 skipped

# OFF shared object, demangled symbol scan
HostRangeScope constructor/destructor symbols: 0

The expected range counts are asserted directly: a cold call emits one each of cache.match, cache.key, cache.lookup, and cache.construct; a hot hit emits only one cache.match. The explicit implementation_index=0 binding case also asserts that no binding.device_conversion range is emitted.

Benchmark / Performance Impact

Incremental A/B benchmark on one NVIDIA A100-SXM4-80GB:

• Baseline: master + PR perf: reuse ATen tensors and optimize PyTorch fallback wrappers #798 + PR perf: optimize generic tensor conversion #832.
• Candidate: the same stack plus this cache-key change.
• PyTorch 2.6.0+cu124, CUDA 12.4, FP16, standard shapes, ATen implementation index 8.
• 99 configurations with comparable torch out= references.
• Three paired alternating rounds on the same physical GPU using P-C-C-P-P-C; no external process appeared on the selected card.

Metric	Cache-key incremental result
Paired median InfiniOps latency improvement	0.328 us (4.17%)
Median InfiniOps-to-Torch gap reduction	0.353 us (14.46%)
Configurations with lower three-round median raw latency	99 / 99
Configurations with a smaller Torch gap	97 / 99

Representative reference-adjusted improvements: maximum 0.768 us, div 0.593 us, bitwise_and 0.557 us, sub 0.504 us, exp 0.358 us, and sign 0.190 us. mm (0.085 us) and addmm (0.023 us) are compute-dominated and close to measurement noise.

These numbers are the incremental cache-key effect only; they do not attribute the larger combined-stack improvement to this PR. A narrow post-profiling-boundary FP16 add index-0 check showed no median regression, but was phase-noisy and is not used as the primary performance claim.

Notes for Reviewers

• CacheKey::Matches compares tensor metadata directly and hashes only scalar/config arguments; vector lengths are included so grouped tensor arguments remain unambiguous.
• One cache.match scope wraps both the most-recent and previous-entry probes, so each Operator::Call emits at most one match range. It is logically separate from cache.lookup, which is reserved for the hash-table operation.
• The two hot entries are thread-local pointers into the existing thread-local unordered_map. Cache generation changes clear both the map and hot pointers.
• Custom CacheKeyBuilder specializations that do not implement Matches automatically fall back to the existing full-key/hash-table path.
• try_emplace avoids a second lookup on misses; a failed operator construction erases the placeholder before rethrowing.
• operator.h uses unconditional named [[maybe_unused]] HostRangeScope objects. With profiling OFF, PR feat: add host range profiling #817's empty scope type optimizes away; the OFF binary symbol scan above confirms that no scope constructor/destructor symbols remain.
• The final diff is limited to src/operator.h, src/host_range_profiler.h, src/host_range_profiler.cc, tests/test_cpp_api.py, and tests/test_host_range_profile.py. It does not include PR perf: reuse ATen tensors and optimize PyTorch fallback wrappers #798, PR perf: optimize generic tensor conversion #832, PR perf: avoid redundant tensor conversion in eager bindings #815, benchmark artifacts, or *infinilm operator-specific changes.
• This PR remains draft because affected non-NVIDIA accelerator backends could not be validated locally.