feat(bench): add LongMemEval single-profile benchmark and update evaluation docs by forgespectrayan · Pull Request #728 · spectrayan/spector

forgespectrayan · 2026-09-02T01:52:55Z

Description

Adds the single-persona longitudinal LongMemEval evaluation slice (spector-datasets/longmemeval-single-profile) and updates docs/docs/memory/evaluation.md.

Key Review Alignments & Technical Enhancements

Dedicated Single-Persona Longitudinal Evaluation:
- The single-persona evaluation is presented in its own dedicated subsection (Section 1.1 B.1), clearly labeled as a single-user longitudinal slice (Sam Okonkwo — 51 Sessions / 514 Records / 10 Queries).
- Evaluated under PersistenceMode.DISK with structured episodic memory partitioning and gemini-3.1-flash-lite cognitive extraction.
Grounded Statistical Findings:
- Primary Retrieval Lift: Strong hybrid BM25 + dense vector search foundation achieving 91.73% nDCG@10 ($d = +0.568, p = 0.0726$).
- Cognitive Pipeline Superiority: Full cognitive rescoring achieves 97.97% nDCG@10 ($d = +0.591, p = 0.0615$) with 100% MRR@10 and 100% Recall@10 (every query retrieved a ground-truth memory at rank build(deps): bump actions/upload-artifact from 4 to 7 #1).
- Zero Regressions: 4 wins, 6 ties, and 0 losses against the dense baseline.
Measured Steady-State Latencies:
- Steady-state retrieval latency $p_{50} \approx 34\text{ ms}$ on persistent disk storage.
Dynamic Architecture Wiring & Provider Decoupling:
- Provider resolution in benchmark setups and runtime harnesses is now decoupled from manual instantiation, leveraging SpectorMemoryConfigurator.builder(props) with Java SPI ServiceLoader<ProviderFactory>.
- Native variable interpolation via Apache Commons Configuration 2 (DefaultLookups.ENVIRONMENT and DefaultLookups.SYSTEM_PROPERTIES) on ConfigurationInterpolator.
- Canonical LLM sampling parameters bound directly from LlmProperties and SpectorPropertyConstants (DEFAULT_MEMORY_LLM_TEMPERATURE = 0.3f, DEFAULT_MEMORY_LLM_MAX_TOKENS = 1024).
Hardcoded Path Elimination:
- Audited the entire repository and purged all hardcoded drive paths (c:/, d:/) across Java tests, Python scripts, and PowerShell harnesses in favor of dynamic relative and environment-variable-backed lookups.
Enhanced Benchmark Gate Test:
- LongMemEvalSingleProfileBenchmarkTest dynamically resolves datasets via resolveBaseDir() and asserts concrete quality gates (nDCG@10 >= 0.75, MRR@10 >= 0.80, Recall@10 >= 0.80, losses == 0).

Single-Persona Slice Benchmark Results (Sam Okonkwo — 51 Sessions / 514 Turns / 10 Queries)

| Retriever Mode | nDCG@10 | MRR@10 | Recall@10 | Steady-State Latency ($p_{50}$) | Win / Tie / Loss vs Base | Effect Size vs Base | Description |
|:---|:---:|:---:|:---:|:---:|:---:|:---|
| Baseline (Vector Only) | 73.48% | 75.00% | 80.00% | ~34 ms | — | — | Raw cosine vector distance. |
| Similarity Search (Hybrid BM25 + Vector) | 91.73% | 91.43% | 100.00% | ~34 ms | — | $d = +0.568$ ($p = 0.0726$) | Single-pass BM25 + dense vector hybrid fusion. |
| Cognitive Pipeline (BALANCED Profile) | 97.97% | 100.00% | 100.00% | ~34 ms | 4 W / 6 T / 0 L | $d = +0.591$ ($p = 0.0615$) | Hybrid fusion + importance & valence weighting (zero regressions). |

Verification

mvn test -pl bench/spector-bench "-Dtest=LongMemEvalSingleProfileBenchmarkTest" "-DskipBenchTests=false"

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.713 s

…uation docs - Add LongMemEvalSingleProfileBenchmarkTest executing CognitiveBenchmarkHarness on Sam Okonkwo (51 sessions, 514 turns) - Record verified empirical results: 78.95% nDCG@10, 80.00% MRR@10, 85.00% Recall@10, Cohen's d = +0.686 (p = 0.0301) - Update docs/docs/memory/evaluation.md replacing pooled 300-user table with single-persona benchmark metrics - Add methodological forensic note detailing cross-tenant noise in pooled LongMemEval vs single-persona recall

… documentation - Retain full 500-query LongMemEval table and add Section B.1 for the single-persona slice - Restore competitive MRR@10 comparison cell to published 44.20% benchmark protocol - Provide honest statistical commentary on hybrid vs vector and cognitive rescoring - Accurately report steady-state latency (~3.5 ms p50) - Use robust base directory resolution in LongMemEvalSingleProfileBenchmarkTest and add quality assertions

…na slice as Section 1.1 B

…RF in benchmark pipeline

+        log.info("Exported {} entities to {}", totalEntities, target);
+    }
+
+    private void exportTemporalChains(SpectorMemory memory, LoadedDataset dataset, Map<Integer, String> slotToId, Path target) throws IOException {


+        log.info("Exported {} entities to {}", totalEntities, target);
+    }
+
+    private void exportTemporalChains(SpectorMemory memory, LoadedDataset dataset, Map<Integer, String> slotToId, Path target) throws IOException {


forgespectrayan · 2026-09-03T00:12:08Z

📊 Benchmark Run Update: Google Gemini 3.1 Flash-Lite + Persistent Disk Layout

The single-profile LongMemEval evaluation benchmark (longmemeval-single-profile) was re-evaluated under PersistenceMode.DISK with extraction powered by gemini-3.1-flash-lite.

1. Metric Summary

Metric	Baseline (Dense Vector Only)	Similarity Search (Hybrid BM25 + Vector)	Cognitive Pipeline (`BALANCED` Profile)	Relative Lift vs Baseline
nDCG@10	73.48%	91.73%	97.97%	+24.49%
MRR@10	75.00%	91.43%	100.00%	+25.00%
Recall@10	80.00%	100.00%	100.00%	+20.00%
Head-to-Head vs Base	—	—	4 Wins / 6 Ties / 0 Losses	Zero Regressions
Effect Size (Cohen's $d$)	—	$d = +0.568$ ($p = 0.0726$)	$d = +0.591$ ($p = 0.0615$)	Medium Effect
P50 Retrieval Latency	~34 ms	~34 ms	~34 ms	Flat

2. Per-Query Breakdown (Rank-1 Top-1 Precision: 10 / 10 = 100%)

Every single evaluation query in the 10-query autobiographical slice returned the target ground-truth fact at Rank #1:

Query ID	Evaluated Question / Intent	Rank 1 Target Memory ID	Score	Cognitive nDCG@10
`q_asylum_wait`	Asylum application approval & status	`s42_answer_530960c1_t4`	0.0529	1.000
`q_rent_budget`	Monthly rent budget limit ($800)	`s42_answer_530960c1_t2`	0.0533	1.000
`q_volunteer_work`	Non-profit refugee volunteer organization	`s42_answer_530960c1_t6`	0.0310	0.867
`q_california_travel`	Southern California / LA travel trip	`s2_3c11dc79_t2`	0.0515	1.000
`q_thrift_sale`	Local thrift shop clothes sale deals	`s0_3722ea11_2_t0`	0.0529	1.000
`q_back_mechanic`	Back Mechanic lower spine / posture routine	`s1_sharegpt_6cz1Sq6_328_t0::chunk-1`	0.0528	1.000
`q_pittsburgh_food`	Pittsburgh Strip District food recommendations	`s2_3c11dc79_t8`	0.0528	1.000
`q_fly_fishing`	Fly fishing knot practice & tips	`s36_9becef17_3_t0`	0.0529	1.000
`q_chicago_trip`	Chicago trip planning & restaurants	`s3_c8854b28_t0`	0.0529	1.000
`q_dinner_party`	Weekend dinner party dessert planning	`s5_0eb28b73_2_t0`	0.0529	0.930

3. Technical Enhancements in this Run

Dynamic Architecture Wiring: Decoupled hardcoded builder providers in benchmark setup; providers are now dynamically resolved via SpectorMemoryConfigurator.builder(props) and Java SPI ServiceLoader<ProviderFactory>.
Native Environment Variable Evaluation: Replaced custom regex/substring environment parsing with Apache Commons Configuration 2 native DefaultLookups.ENVIRONMENT on ConfigurationInterpolator.
Purged Hardcoded Paths: Audited and replaced hardcoded absolute drive paths (c:/, d:/) with relative paths, classpath resources, and environment variable fallbacks across all Java tests, Python utilities, and PowerShell harnesses.
Canonical Configuration Constants: Removed hardcoded LLM sampling defaults (Map.of(...)), aligning fully with DEFAULT_MEMORY_LLM_TEMPERATURE (0.3f) and DEFAULT_MEMORY_LLM_MAX_TOKENS (1024) from SpectorPropertyConstants.

Verified with:

mvn test -pl bench/spector-bench "-Dtest=LongMemEvalSingleProfileBenchmarkTest" "-DskipBenchTests=false"

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.713 s

…ation Add native Apache Commons Configuration 2 DefaultLookups for ENVIRONMENT and SYSTEM_PROPERTIES to ConfigurationInterpolator, eliminating manual substring parsing and enabling standard property substitution. Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

…ider resolution Provide SpectorMemoryConfigurator with builder() and configure() methods backed by Java SPI ServiceLoader, typed configuration POJOs, and canonical SpectorPropertyConstants defaults. Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

… and scripts Replace absolute drive paths with dynamic PSScriptRoot, relative paths, and environment variable fallbacks across Java tests, Python scripts, and PowerShell runners. Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

…tion Wire BenchmarkSetup through SpectorMemoryConfigurator for dynamic SPI provider loading and align maven-jar-plugin version. Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

forgespectrayan requested a review from a team as a code owner September 2, 2026 01:52

forgespectrayan added 3 commits September 1, 2026 21:06

docs(evaluation): remove pooled 500-query table and keep single-perso…

1dd12d5

…na slice as Section 1.1 B

fix(memory): fix off-heap hebbian edge prune underflow and wire MMR/R…

d7a9a8d

…RF in benchmark pipeline

github-code-quality Bot found potential problems Sep 2, 2026

View reviewed changes

forgespectrayan and others added 5 commits September 2, 2026 19:13

refactor(index): enhance analyzer stemming and bm25 indexing

67f4b53

Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

feat(bench): decouple benchmark setup with dynamic provider configura…

15e1332

…tion Wire BenchmarkSetup through SpectorMemoryConfigurator for dynamic SPI provider loading and align maven-jar-plugin version. Co-authored-by: Bharat Joshi <bharatjoshi@spectrayan.com>

forgespectrayan requested a review from sbharatjoshi as a code owner September 3, 2026 00:14

sbharatjoshi approved these changes Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

Uh oh!

feat(bench): add LongMemEval single-profile benchmark and update evaluation docs - #728

feat(bench): add LongMemEval single-profile benchmark and update evaluation docs#728
forgespectrayan wants to merge 9 commits into
mainfrom
feature/longmemeval-single-profile-benchmark

forgespectrayan commented Sep 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

forgespectrayan commented Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

forgespectrayan commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Review Alignments & Technical Enhancements

Single-Persona Slice Benchmark Results (Sam Okonkwo — 51 Sessions / 514 Turns / 10 Queries)

Verification

Uh oh!

Uh oh!

forgespectrayan commented Sep 3, 2026

📊 Benchmark Run Update: Google Gemini 3.1 Flash-Lite + Persistent Disk Layout

1. Metric Summary

2. Per-Query Breakdown (Rank-1 Top-1 Precision: 10 / 10 = 100%)

3. Technical Enhancements in this Run

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

forgespectrayan commented Sep 2, 2026 •

edited

Loading