SNAP dataset · Quickstart · Testing · License
Python 3.13.1 · Apache Spark 4.0.0 · GraphFrames 0.12.1 · MIT
This project turns the SNAP amazon-meta.txt file into a controlled recommendation experiment. Apache Spark parses it into typed Bronze, Silver and Gold tables, five recommenders are trained and compared under one leakage-safe temporal protocol, and each stage must pass a fingerprinted gate before the next one can run.
The source is a 977.5 MB text file with blank-line delimited records, and some titles span physical lines. It combines product metadata, directed similar-product links, category hierarchies and historical ratings. Records are nested, discontinued products use a shorter record shape, and category labels carry bracketed ids. Malformed blocks must be kept and explained rather than silently discarded.
These are the counts that the G4 and G5 gates assert: expected values in the code, not outputs of a stored run, because run artifacts are git-ignored.
- Source-faithful ingestion - Hadoop
TextInputFormatsplits the file on its blank-line record delimiter (CRLF CRLF), and malformed blocks go to quarantine with the error code, raw block, source offset, path and error detail. - Five complementary recommenders - popularity, ALS, FP-Growth, graph and category models are trained from training data only, under one temporal split.
- Deterministic hybrid ranking - weighted reciprocal rank fusion with
c = 60combines two weight sets, H-A and H-B. The winner is chosen on validation data only: higher NDCG@10 when the gap is at least 0.001, otherwise higher user coverage, otherwise H-A. - Thirteen gates with manifests - each passed gate writes a JSON manifest with SHA-256 fingerprints, and a later gate is blocked by missing, failed or mismatched evidence.
- Spark-free serving - Streamlit reads run exports through DuckDB with 2 threads and a 384 MB memory limit, accepts read-only queries (
SELECT,WITHorDESCRIBE SELECT) with bound parameters, and never imports PySpark. - Audited data quality - a fixed taxonomy of 18 event types. 146,745 duplicate review occurrences are removed and reconciled, and 500 invalid sales ranks are counted as 459 plus 41.
Spark runs the expensive work before the application starts. Streamlit reads completed, run-scoped Parquet exports through DuckDB, so a page load, search or filter never starts a distributed job. The application serves only run directories that have a Spark _SUCCESS marker, and it never imports PySpark. NetworkX draws only deterministic ego layouts of at most 50 nodes; full graph analytics stay in GraphFrames.
The design keeps raw evidence next to typed tables, compares structurally different recommenders under one leakage-safe protocol, and makes every result explorable without starting Spark from the user interface.
The five models are independent candidate generators. Each produces its own ranked list, so raw scores on different scales never have to be compared directly.
| Model | Signal and candidates per user |
|---|---|
| Bayesian popularity | Weighted rating over the unique raters of each item. 100 candidates |
| ALS | Explicit-feedback latent factors: rank 20, regularization 0.10, 10 iterations, cold start dropped. 100 candidates (raw depth 200) |
| FP-Growth | Association rules from baskets of ratings of at least 4.0. 50 candidates |
| Graph | One-hop, reciprocal and two-hop paths in the similar-product graph. 50 candidates |
| Category | Blend of category-vector cosine (0.80), user group affinity (0.10) and popularity percentile (0.10). 50 candidates |
The Bayesian popularity score is (n * item mean + 20 * global mean) / (n + 20), where n counts the unique raters of the item.
ALS and FP-Growth run on Spark MLlib, and graph analytics run on GraphFrames. Every model is fed from training data only. Candidates exclude items the user has already seen at that stage, and discontinued or out-of-catalog products.
Records are separated by a blank line (CRLF CRLF). The primary path reads them with Hadoop TextInputFormat and a 128 MiB split maximum, and an integration test covers split boundaries at that size. The fallback framer writes 64 integrity-checked JSONL shards, and each framed block stores its SHA-256. Blocks that fail parsing go to quarantine with the error code, raw block, source offset, source path and error detail. Quality events follow a fixed taxonomy of 18 types, including PARSE_ERROR and MULTILINE_TITLE. Valid but unusual values stay in the data and appear as quality events, separate from structural failures.
Product headers declare 7.781.990 reviews, while the downloaded source contains 7.593.244 physical review rows. Models start from the physical rows, and the declared total remains a data-quality signal.
Reconciliation counts
| Count | Value | Asserted by |
|---|---|---|
| Products | 548,552 = 542,684 active + 5,868 discontinued | G4, G5, G12 |
| Physical review rows | 7,593,244 | G4, G5 |
| Duplicate review occurrences removed | 146,745 | G5 |
| Deduplicated reviews | 7,446,499 = 7,593,244 - 146,745 | G5 |
| Products whose declared review total differs from the downloaded count | 8,615 above, 131 below | G5 |
| Products with zero declared review total | 139,949 | G5 |
| Invalid sales ranks | 500 = 459 at -1 plus 41 at 0 | G5 |
| Orphan graph targets | 172,790 ASINs referenced 557,286 times | G5 |
| Set | Rule |
|---|---|
| Train | All earlier interactions |
| Validation | Penultimate interaction, position 2 from the end |
| Test | Latest interaction, position 1 from the end |
Interactions are ordered by interaction date and then product id. A user is evaluation-eligible with at least 5 distinct items. The test seen-set includes the validation item, while the model profile does not. ALS trains on a k-core universe of users with at least 3 items and items with at least 5 users.
Ranking uses NDCG@10, HitRate@10 and MRR@10. Coverage is reported as user coverage, fill rate at 10 and catalog coverage at 10. Users with empty recommendation lists stay in the denominator, so coverage loss cannot inflate ranking success. RMSE and MAE use raw, unclipped ALS predictions. Slices are overall, Book and non-Book, and no model is rerun per slice.
Each model adds its normalized weight divided by 60 + rank, where rank is one-based and c = 60. Weights are renormalized over the models that produced candidates for each user, and contributions are folded in sorted model order. The fused list is ordered by hybrid score, contributing-model count, Bayesian score and ascending product id, and the top 100 are kept.
The winner is chosen from validation data only, on the common_warm cohort. The higher NDCG@10 wins when the gap is at least 0.001. Otherwise the higher user coverage wins, and otherwise H-A. The config loader rejects any weight set that does not sum to 1.
Exact weights
| Model | H-A (balanced) | H-B (behavior-weighted) |
|---|---|---|
| ALS | 0.35 | 0.50 |
| Graph | 0.20 | 0.20 |
| Category | 0.20 | 0.10 |
| FP-Growth | 0.15 | 0.15 |
| Popularity | 0.10 | 0.05 |
| Sum | 1.00 | 1.00 |
The board above groups the thirteen ordered gates, G0 to G12, into five phases. G0 is a script (scripts/g0_smoke.py), G1 is built into the CLI, and phase handlers implement G2 to G12.
Gates G1 to G12 write a manifest to artifacts/runs/$RUN_ID/manifests/G<n>.json with the run id, the config and source SHA-256, prerequisite evidence digests and timing. G0 writes environment evidence (versions, hardware, GraphFrames JAR hashes and runtime tests) to its manifest at the same path, and the fingerprint check does not apply to G0. A gate is blocked when an earlier passed manifest, other than G0, has a different config or source fingerprint. A passed gate is reused unless --force is given.
configs/project.yaml fixes the source identity, seed, Spark settings, split rules and model parameters, and the configuration loader rejects changes to these binding values. Each published table records its row count, schema, file count, size and SHA-256.
Tables are written to a temporary directory, read back to count their rows, and renamed into place. A table path without _SUCCESS is treated as an interrupted workspace and rebuilt.
G12 is the final acceptance audit. It recomputes prior manifest and artifact fingerprints, reconstructs the frozen hybrid selection, verifies the official comparison matrix and this README, then publishes the delivery atomically and writes its success marker last.
The repository holds no measured results, so this section describes the performance protocol only. G11 compares local[1] with local[min(4, logical cores)], runs 1 warm-up and 3 measured runs per condition, and disables dataset caching. Its output is artifacts/runs/$RUN_ID/performance/summary.json.
Gate reference
| Gate | Responsibility | Primary entry |
|---|---|---|
| G0 | Runtime proof: Parquet round trip, GraphFrames PageRank and WCC, durable checkpoint | scripts/g0_smoke.py |
| G1 | Passing JUnit evidence and the project command contract | gate G1 |
| G2 | Parser and record-boundary contract | gate G2 |
| G3 | Deterministic Bronze-to-Gold smoke pipeline | gate G3 |
| G4 | Full-source ETL and canonical source counts | gate G4 |
| G5 | Deduplication, interaction aggregation and quality profile | gate G5 |
| G6 | Temporal split, cohorts, k-core and leakage checks | gate G6 |
| G7 | Five independent model outputs | gate G7 |
| G8 | Shared hybrid evidence and H-A and H-B rankings | gate G8 |
| G9 | Validation freeze and official evaluation | gate G9 |
| G10 | Spark-free dashboard exports and application contracts | gate G10 |
| G11 | Controlled local[1] versus bounded multi-core experiment |
gate G11 |
| G12 | Independent acceptance audit and atomic delivery | gate G12 |
The application is titled Catalog Observatory and has four pages under app/. Its UI copy is Turkish, while this README is English.
- Home (
Home.py): scale, distributions, sparsity and quality events. - Product and Graph Explorer (
2_Product_and_Graph_Explorer.py): product metadata, taxonomy, PageRank, degree, weak components and a bounded first-degree graph. - Recommendation Lab (
3_Recommendation_Lab.py): customer, seed-product and category-onboarding recommendations with their evidence. - Model and Experiment Comparison (
4_Model_and_Experiment_Comparison.py): ranking and coverage metrics, ALS prediction metrics, model runtimes, local Spark performance and reproducibility metadata.
Serving bounds
- DuckDB runs with 2 threads and a 384 MB memory limit.
- Queries must start with
SELECT,WITHorDESCRIBE SELECT, and user input is passed as bound parameters. - Detail reads return at most 500 rows, and recommendation lists at most 100 rows.
- Bounded reads are cached for 600 seconds, in at most 512 entries.
- Only run directories with a Spark
_SUCCESSmarker are served, and only allowlisted logical tables. - Recommendation Lab searches are limited to 20 customers and 30 categories, and its top-K slider runs from 1 to 50 in the customer section and from 1 to 30 in the seed-product and category sections, with a default of 10.
- The dashboard needs gates G0 to G10 to have passed; G11 and G12 are not needed to serve it.
| Layer | Technologies and role |
|---|---|
| Language and runtime | Python 3.13.1, Java 21.0.11 and Scala 2.13.16. Application logic in Python, on the JVM and Scala runtime that Spark needs |
| Distributed processing | Apache Spark 4.0.0, PySpark and Spark SQL. Parsing, transformation, feature engineering, candidate generation and evaluation |
| Machine learning | Spark MLlib. Explicit-feedback ALS and FP-Growth training |
| Graph analytics | GraphFrames 0.12.1. PageRank, degrees, reciprocity and weakly connected components |
| Analytical storage | Parquet with Snappy and PyArrow 25.0.0. Typed, compressed tables with fingerprinted artifacts |
| Query layer | DuckDB 1.5.4. Bounded, read-only queries over completed Parquet exports |
| Application UI | Streamlit 1.59.1 and Plotly 6.9.0. Four-page interactive analytics application |
| Local helpers | pandas 2.3.0 and NetworkX 3.5. Bounded result shaping and small deterministic ego-graph layouts |
| Testing | pytest 9.1.1 and pytest-cov 7.1.0. Unit, Spark integration and contract tests |
| Configuration and automation | YAML, JSON, Make and Bash. Configuration, evidence manifests, repeatable commands and environment enforcement |
Prerequisites:
-
Linux with
pyenvand thepyenv-virtualenvplugin. -
Python 3.13.1 in an environment named
bil401_env_1. The launcher refuses any other interpreter. -
OpenJDK 21.0.11 at
/usr/lib/jvm/java-21-openjdk-amd64. G0 records this Java version. -
At least 15 GiB of total memory, the baseline that G12 checks in the G0 evidence.
-
At least 20 GiB of free disk for source data, Parquet outputs, models and publication workspaces.
-
The SNAP source at
Dataset/amazon-meta.txt, with this SHA-256 digest:600135116a05b7ce2dcb7e842e892d663c6190a0567d00373e0c5c4f3c908f02
The dataset and generated run artifacts stay out of version control.
Run the pipeline in this order:
-
Create the locked environment once and install the project.
make bootstraprunspip install -r requirements.lock -e .; it does not fetch the GraphFrames JARs.pyenv install -s 3.13.1 pyenv prefix bil401_env_1 >/dev/null 2>&1 \ || pyenv virtualenv 3.13.1 bil401_env_1 export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 make bootstrap
-
Cache the GraphFrames runtime JARs in
.cache/ivy/jars, which is not committed. The copy loop also copies the datasketches JARs, although no code references them.PY="$(PYENV_VERSION=bil401_env_1 pyenv which python)" mkdir -p .cache/ivy/jars PYSPARK_PYTHON="$PY" PYSPARK_DRIVER_PYTHON="$PY" \ PYSPARK_SUBMIT_ARGS='--packages io.graphframes:graphframes-spark4_2.13:0.12.1 pyspark-shell' \ "$PY" -c 'from pyspark.sql import SparkSession; SparkSession.builder.getOrCreate().stop()' find "$HOME/.ivy2/jars" -maxdepth 1 -type f \ \( -name 'io.graphframes_*.jar' -o -name 'org.apache.datasketches_*.jar' \) \ -exec cp -n {} .cache/ivy/jars/ \;
-
Create the test evidence and the run id. This step runs only with the
bil401_env_1interpreter thatscripts/run_tests_sharded.pypins, so it does not run on other machines as written.export SOURCE8="$(sha256sum Dataset/amazon-meta.txt | cut -c1-8)" export RUN_ID="run-$(date -u +%Y%m%dT%H%M%SZ)-$SOURCE8" export JUNIT="artifacts/runs/$RUN_ID/test-results/preflight.xml" make test TEST_JUNIT="$JUNIT" \ TEST_SHARD_DIR="artifacts/runs/$RUN_ID/test-results/shards"
-
Record the G0 runtime proof with the Spark settings that
bin/amazon-recpasses to Spark, using its default values. G12 requires the recorded driver memory to be8g.export JARS="$(find .cache/ivy/jars -type f -name '*.jar' | sort | paste -sd, -)" export SPARK_LOCAL_IP=127.0.0.1 export PYSPARK_SUBMIT_ARGS="--master local[*] --driver-memory 8g \ --driver-java-options=-Duser.timezone=UTC \ --conf spark.driver.host=127.0.0.1 \ --conf spark.driver.bindAddress=127.0.0.1 \ --conf spark.driver.maxResultSize=1g \ --conf spark.ui.enabled=false \ --conf spark.ui.showConsoleProgress=false \ --conf spark.task.cpus=4 \ --conf spark.memory.fraction=0.35 \ --conf spark.memory.storageFraction=0.30 \ --conf spark.sql.shuffle.partitions=64 \ --conf spark.sql.adaptive.enabled=true \ --conf spark.sql.files.maxPartitionBytes=134217728 \ --conf spark.sql.parquet.compression.codec=snappy \ --conf spark.sql.session.timeZone=UTC \ --jars $JARS pyspark-shell" JAVA_HOME="$JAVA_HOME" PYSPARK_PYTHON="$PY" PYSPARK_DRIVER_PYTHON="$PY" \ "$PY" scripts/g0_smoke.py --output "artifacts/runs/$RUN_ID/manifests/G0.json"
-
Run gates G1 to G12 in order, then print the chain status. Each gate receives the JUnit evidence file, and the loop stops at the first gate that does not pass.
for gate_number in {1..12}; do ./bin/amazon-rec --run-id "$RUN_ID" gate "G${gate_number}" \ --evidence-file "$JUNIT" || break done ./bin/amazon-rec --run-id "$RUN_ID" status
-
Open the dashboard once G0 to G10 have passed. Streamlit uses its default port, 8501.
./bin/amazon-rec --run-id "$RUN_ID" dashboard
Pin the dashboard address
With these variables set, the application is served at http://127.0.0.1:8501.
STREAMLIT_SERVER_HEADLESS=true \
STREAMLIT_SERVER_ADDRESS=127.0.0.1 \
STREAMLIT_SERVER_PORT=8501 \
STREAMLIT_BROWSER_GATHER_USAGE_STATS=false \
./bin/amazon-rec --run-id "$RUN_ID" dashboardNotes:
- Run ids must match
^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$. Always pass--run-id; the CLI's built-in default is a dated run id that is not a current run. - Exit code 3 means a gate is blocked by missing or incompatible prerequisites, and exit code 1 means it failed.
- A passed gate is reused unless
--forceis given before the subcommand. - The
makeshortcuts do not pass--evidence-file, so this guide calls the CLI directly.
Project layout
.
├── .github/readme/ # README visual spec and generated SVG assets
├── .gitignore # Ignored data, run artifacts and local state
├── .streamlit/ # Streamlit theme configuration
├── app/ # Streamlit pages and bounded DuckDB access
├── bin/amazon-rec # Environment-enforcing command launcher
├── configs/project.yaml # Source, Spark, model and experiment contract
├── LICENSE # MIT License
├── Makefile # Bootstrap, test and shortcut targets
├── pyproject.toml # Package metadata and pinned versions
├── requirements.lock # Locked runtime and test environment
├── scripts/ # Dataset analysis, G0 proof and sharded tests
├── src/amazon_recommender/
│ ├── cli.py # Command interface for gates, aliases and status
│ ├── gate_handlers.py # Registry of G2 to G12 phase handlers
│ ├── core/ # Configuration, gates, manifests and paths
│ ├── ingestion/ # Delimiter framing, schemas and strict parser
│ ├── pipelines/ # Bronze/Silver/Gold transforms and atomic storage
│ ├── quality/ # Data profiling and quality-event taxonomy
│ ├── features/ # Temporal split, cohorts, baskets and profiles
│ ├── models/ # Five recommenders and hybrid rank fusion
│ ├── evaluation/ # Ranking, coverage and ALS prediction metrics
│ ├── performance/ # Fixed workload and experiment protocol
│ └── phases/ # G2-G12 gate implementations
└── tests/ # Unit and local Spark integration contracts
The suite contains 184 test cases in 31 test modules, across unit and local Spark integration tests. The sharded runner executes 12 sequential shards, 10 of them Spark, and merges their JUnit fragments with scripts/merge_junit.py. It refuses to start while another project Spark process is active.
Official outputs are written under artifacts/runs/$RUN_ID/:
- Gate manifests G0 to G12:
manifests/G0.jsonthroughmanifests/G12.json - Model comparison:
data/g9/official_test_comparison/ - Selection evidence:
data/g9/selected_hybrid/ - Performance evidence:
performance/summary.json - Accepted delivery:
delivery/
Test coverage areas
- CRLF and LF record framing, and mixed line endings fail closed.
- Hadoop split boundaries at 128 MiB.
- Hand-calculated association rules, category math and ranking metrics.
- Traversal-safe run ids: unsafe values such as
../escape, absolute paths, spaces and empty ids are rejected. - Parameterized, bounded DuckDB search.
- NetworkX layouts that are deterministic and never exceed 50 nodes.
- The Streamlit application never imports PySpark.
- An interrupted G11 run reuses only atomically completed trials.
- Canonical source counts: G5 accepts only the exact canonical counts, and the quality-event taxonomy contains every binding event once.
- Temporal split and leakage checks: G6 invariants reject held-out leakage and missing held-out users.
- Stable, seeded sampling of the smoke set.
- ALS recommendations are raw depth 200, then active and unseen items only, without fallback.
- Interruption-safe table publication: incomplete temporary publications are removed.
- Stable hybrid tie-breaking: all four deterministic ranking keys are applied.
- Validation-only selection: strict NDCG@10 threshold, then coverage, then H-A.
- Validation-only G9 selection, recomputed from frozen Parquet artifacts.
- G8 experiment budget: no refit, and a third variant is rejected.
- G12 recomputes every manifest digest and rejects tampered manifests and README placeholders.
- Binding configuration rejects parameter changes.
- Registered pytest markers:
unit,integrationandslow.e2eandfull_dataare registered but not yet applied.
- The dataset and run artifacts are not in the repository, so the README shows configuration values and the counts that the gates assert, and no measured ranking results.
- The test runner and the configuration pin one interpreter path, so step 3 of the Quickstart runs only where that interpreter exists.
The source code is available under the MIT License. Copyright (c) 2026 Arda Tuz. The SNAP dataset retains its own source and citation terms; see the SNAP dataset page.