sift-api

Python FastAPI + LangGraph backend for Sift — the AI-curated news reader.

Handles the background content pipeline (RSS feeds → Claude Haiku summaries → Voyage AI embeddings → Neon Postgres) and the multi-source comparison workflow (LangGraph fan-out web search → claim extraction → comparison).

Architecture

Railway asyncio scheduler (every 30 min)
  → LangGraph pipeline: fetch_rss → deduplicate → summarize (Claude) → context/primer (Claude, batch) → embed (Voyage) → link_entities → store (Postgres)
      → nested LangGraph story-threading (per category): fetch_articles → extract_entities → cluster (Claude) → synthesize_and_store

User compare request (via Vercel proxy)
  → LangGraph compare: search_sources (parallel, Claude web_search) → extract_and_compare → format_response

This service compiles three LangGraph StateGraphs: the ingestion pipeline, a story-threading graph the pipeline invokes per category, and the on-demand compare graph. (Two are user-facing workflows; story-threading is nested inside ingestion.)

User-facing reads happen in the Next.js frontend — this service handles background AI processing and on-demand comparison.

Content quality gate (why_it_matters / context_primer)

The per-article why_it_matters line and context_primer.background paragraph are AI-generated (Claude Haiku, via the Message Batches API). A 500-article audit (sift#150) found why_it_matters failed two ways — it either restated the summary or hand-waved with vague-significance clichés. The gate (sift-api#90) is semantic, at generation time, in three layers:

1. Generation rubric (services/context_generator.py, services/primer_generator.py) — the prompt demands a concrete, verifiable stake not already in the title/summary, strictly neutral, no restatement, no clichés, and returns an empty line when there's no real stake. An absent line renders nothing — null-over-filler is the intended outcome, not a failure. why_it_matters is therefore NULL for a meaningful share of articles by design (it has always been nullable; the UI hides it).
2. Deterministic gate (services/quality_gate.py) — a pure, free cliché/restatement check applied in the same poller callback that stores generated copy, as a backstop to the prompt. Clichés (not lexical overlap) are the workhorse.
3. LLM judge (services/judge.py) — a Sonnet judge scoring each line on the issue's three axes (restates? adds verifiable significance? neutral + cliché-free?). Runs offline by default (the eval's measurement tool). Optionally runs in the batch-result path behind WHY_IT_MATTERS_JUDGE_ENABLED (default off) to drop the paraphrase/editorial residual the deterministic gate can't catch — dropping on restatement or non-neutrality only, respecting the cost guard. Background paragraphs are cliché-trimmed only when short, so an informative paragraph with a stray clause is kept.

# Eval: baseline existing copy vs. the new rubric, before/after rates (judge = API spend)
python scripts/eval_why_it_matters.py                                   # offline, fixture corpus, gate only
python scripts/eval_why_it_matters.py --from-db --limit 500 --judge --mode compare

# Re-gate already-stored copy (ON CONFLICT does not regenerate, so audit-era
# bad lines persist until reprocessed). Dry run by default.
python scripts/regate_existing.py            # preview drops
python scripts/regate_existing.py --apply    # NULL failing lines / blank cliché backgrounds (terms kept)

Setup

Prerequisites

• Python 3.12+
• Docker (for local Postgres + pgvector)

Local development

# Start Postgres
docker compose up -d

# Create virtualenv and install deps (dev file pulls in requirements.txt too)
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys

# Run server
uvicorn app.main:app --reload --port 8000

Verify

# Health check
curl http://localhost:8000/health

# Trigger the RSS pipeline (refreshes all categories; there is no category-scoped refresh)
curl -X POST http://localhost:8000/pipeline/refresh \
  -H "Content-Type: application/json" \
  -H "X-Pipeline-Key: dev-key" \
  -d '{"force": true}'

# Multi-source comparison
curl -X POST http://localhost:8000/analyze/compare \
  -H "Content-Type: application/json" \
  -H "X-Pipeline-Key: dev-key" \
  -d '{"topic": "Federal Reserve interest rate decision", "sources": ["reuters", "bbc", "associated press"]}'

API

All endpoints are available at both /v1/... (preferred) and legacy paths (for backwards compatibility).

Method	Endpoint	Description
GET	/	Service info + available endpoints
GET	/health	Health check + DB status + last pipeline run. Also carries model_overrides when LLM_MODEL_OVERRIDES has moved a stage off its default model — omitted entirely when nothing has
GET	/docs	Interactive API documentation (Swagger UI)
GET	/redoc	Alternative API documentation (ReDoc)
POST	/v1/pipeline/refresh	Trigger RSS pipeline (auth required)
POST	/v1/analyze/compare	Multi-source comparison via LangGraph
POST	/v1/analyze/compare/stream	Same comparison as SSE: stage / source-done / results / error events
POST	/v1/analyze/compare/daily/refresh	Force-regenerate the anonymous daily compare example (auth required)

Project structure

sift-api/
├── app/
│   ├── main.py              # FastAPI app, health, background scheduler
│   ├── config.py            # pydantic-settings
│   ├── db.py                # asyncpg connection pool
│   ├── models.py            # Pydantic schemas
│   └── routers/
│       ├── pipeline.py      # POST /pipeline/refresh
│       └── compare.py       # POST /analyze/compare
├── workflows/
│   ├── pipeline_workflow.py # LangGraph: fetch_rss→dedup→summarize→context→primer→embed→link_entities→store
│   ├── story_workflow.py    # LangGraph: fetch_articles→extract_entities→cluster→synthesize_and_store (nested in pipeline)
│   └── compare_workflow.py  # LangGraph: search_sources→extract_and_compare→format_response
├── services/
│   ├── rss.py               # 59 RSS feeds (56 outlets) across the spectrum, feedparser, image extraction
│   ├── summarizer.py        # Claude Haiku 4.5 batch summarization
│   ├── context_generator.py # why_it_matters line + importance score
│   ├── primer_generator.py  # context_primer: background + glossary terms
│   ├── quality_gate.py      # deterministic cliché/restatement gate (#90)
│   ├── judge.py             # offline LLM judge for the quality eval (#90)
│   ├── embedder.py          # Voyage AI embeddings (voyage-3-lite, 512-dim)
│   └── deduplicator.py      # Postgres dedup check
├── tests/
├── docker-compose.yml       # Postgres 16 + pgvector (local dev)
├── init.sql                 # DB schema (9 tables; civic profile tables via migrations/)
├── Dockerfile               # Production image
├── railway.toml             # Railway deployment config
└── .github/workflows/ci.yml # Ruff + pytest on PR/push

Database

Schema source of truth for a fresh database is init.sql. Additive changes after the initial schema are layered on via two parallel mechanisms:

Where	Form	Applied by
migrations/NNN_*.sql	CREATE ... CONCURRENTLY IF NOT EXISTS	Operator running psql -f manually
app/db.py:_apply_migrations	Same DDL, non-CONCURRENTLY, idempotent	FastAPI startup (the prod path on Railway)

When adding a migration, write both. The SQL file is the CONCURRENTLY-safe version for live ops; the Python hook is what actually runs on every deploy.

Feed queries and the indexes that serve them

The user-facing /api/news feed is served by queries that live in sift/lib/db.ts in the sift repo, not here. This service owns the write path and the indexes; the frontend owns the read queries. The partial indexes below (defined in migrations/004_feed_indexes.sql + app/db.py:_apply_migrations) exist to match the exact predicates those queries use.

Query (sift/lib/db.ts)	Purpose	Index
:36 getArticlesByCategory	category fallback feed	idx_articles_feed
:85 stories + LEFT JOIN	top stories per category	idx_stories_feed + idx_articles_story_feed
:121 story articles	articles belonging to a story	idx_articles_story_feed
:150 standalone articles	articles outside any story	idx_articles_feed

Client abort budget is API_TIMEOUT_MS = 10_000 in sift/lib/constants.ts; exceeding it surfaces as "We hit a snag pulling today's stories." If a category tab starts timing out, these indexes or these queries are the place to look.

Diagnosing feed-query performance

python scripts/explain_feed_queries.py            # summary table
python scripts/explain_feed_queries.py --verbose  # full EXPLAIN JSON

Runs EXPLAIN (ANALYZE, BUFFERS) across all 10 categories × 3 query shapes against DATABASE_URL. Warns at 2000 ms, fails (exit 1) at 8000 ms.

CI wires the same script into the feed-perf job (.github/workflows/ci.yml), triggered only on PRs that touch app/db.py, migrations/, or the script itself. Requires a DATABASE_URL repo secret set to the prod Neon URL.

Funding edges (990 Schedule I / R)

python scripts/ingest_funding_edges.py --year 2025 --dry-run   # parse + report
python scripts/ingest_funding_edges.py --year 2025 --apply     # write funding_edges
python scripts/ingest_funding_edges.py --year 2025 --dry-run --only 237327730

Builds edges between organizations — grants paid (Schedule I Part II) and declared related tax-exempt orgs (Schedule R Part II) — for the ten curated think tanks in ORGS. Run on demand only; deliberately not on the pipeline heartbeat, so an empty funding_edges table is the normal state.

Every edge stores an ein_name_agrees verdict (agrees / review / ein_absent) computed at ingest by services/funding_edges.py, which compares the counterparty name as filed against the IRS's own name for that EIN. A counterparty EIN is a join key a human typed at the filing org, and in the first 121-edge pull one was wrong — Brookings listed "Urban League of Louisiana" under The Urban Institute's EIN. Only agrees edges are publishable; review means unadjudicated, not false.

Requires stream-unzip (in requirements-dev.txt, not shipped to prod): the IRS bulk zips use Deflate64, which Python's zipfile, bsdtar and macOS ditto all refuse — and bsdtar fails by writing a zero-byte file.

Environment variables

Variable	Required	Description
DATABASE_URL	Yes	Neon Postgres direct connection string
ANTHROPIC_API_KEY	Yes	Claude API key (summaries + comparison)
VOYAGE_API_KEY	Yes	Voyage AI key (embeddings)
PIPELINE_API_KEY	Yes	Shared secret for pipeline auth
ENVIRONMENT	No	development or production (enables background scheduler)
PORT	No	Server port (default: 8000, Railway injects 8080)
LOG_LEVEL	No	debug, info, warning, error
SENTRY_DSN	No	Sentry DSN for error monitoring. Inert unless set; no PII sent
SENTRY_TRACES_SAMPLE_RATE	No	Sentry tracing sample rate (default: 0.1)
AI_COST_GUARD_ENABLED	No	Enable the daily AI cost ceiling (default: false)
DAILY_AI_COST_LIMIT_USD	No	Daily Claude + Voyage spend ceiling, USD (default: 10.0)
AI_COST_ALERT_THRESHOLD_RATIO	No	Budget fraction that triggers an alert (default: 0.8)
WHY_IT_MATTERS_JUDGE_ENABLED	No	Run the runtime LLM-judge on generated why_it_matters lines (default: false; adds a paid call per kept line, increases suppression)
ENTITY_LINKER_REGEX_GATE_ENABLED	No	Send an article to the LLM entity linker only when the free regex matcher finds a candidate surface form (default: true — a kill switch, not an opt-in, since it saves rather than spends). Forwards ~26% of articles and retains 98.11% of the links the ungated path produced, measured by scripts/eval_linker_gate.py. Set false to restore one LLM call per article
LLM_MODEL_OVERRIDES	No	Per-stage model overrides against services/model_registry.py, as operation=catalog_id pairs — e.g. summarizer.batch=sonnet-4-6,judge.batch=haiku-4-5. Empty (the default) runs every stage on the model it used before the registry existed. A malformed entry, unknown operation, unknown model, or a model lacking a capability the stage requires is dropped with an ERROR and that stage stays on its incumbent — never a hard failure, never silent. Active overrides are logged at WARNING on startup and echoed in GET /health

Which model is a stage on?

Ask the service, don't infer it from the env var:

curl -s https://<host>/health | jq .model_overrides

null means every stage is on its default. services/model_registry.py holds the mapping and ai_usage_daily records the model each call actually ran on, so spend per stage per model is a SQL query rather than an assumption.

Tests

pytest
ruff check .

Deployment

Deployed to Railway. Push to main triggers automatic deploy. CI runs ruff + pytest on every PR via GitHub Actions.

Production URL: sift-api-production.up.railway.app (target port: 8080)