Complete guide for running the Deproceduralizer analysis pipeline on a state's regulations.
1. Scrape → regulatory_documents table
1b. Exclude → marks non-regulatory sections (exec orders, military code, data tables)
2. Regulatory Status → marks repealed/reserved/expired sections
3. Definitions → definitions table
4. Link Definitions → definition_document_occurrences table
5. Cross-References → cross_references table (internal citations between documents)
6. Summaries → summary, keywords, section_type on regulatory_documents
7. Compliance Scan → compliance_issues table (regex matches)
8. Research Pipeline → compliance_issue_analyses table (LLM Tier 1-3)
9. IBR Scan → incorporations_by_reference table
9b. Fines Scan → fines_fees_penalties table (LLM structured extraction)
10. Barrier Pipeline → barrier_analyses + barrier_issues tables (LLM single-pass)
11. Refresh Stats → precomputed counts on regulatory_documents
Separate pipeline:
12. Guidance Import → guidance_documents table (from nys-documents repo)
Step dependencies:
- Steps 1-4 are sequential prerequisites
- Step 5 (cross-references) can run anytime after Step 1 (only needs scraped documents)
- Steps 6 and 7 can run in parallel (no dependency on each other)
- Step 8 depends on Steps 5 and 6 — the research pipeline uses document summaries and cross-reference summaries as LLM context
- Steps 9-9b are independent of 8 and can run anytime after Step 1
- Step 10 depends on Steps 5 and 6 — Barrier uses summaries, cross-refs, and definitions as rich LLM context
- Step 11 should run last (after any step that changes counts)
Replace NY/ny with NJ/nj for New Jersey or NYC/nyc for New York City. All commands run from the project root unless noted.
python backend/scripts/ny_regulations_scraper.py- Scrapes NYCRR from Cornell Law's Legal Information Institute
- Creates JSON cache in
data/scraped_data/ - Automatically runs
migrate_regulations_to_db()+refresh_all_stats() - Progress tracked in
progress.json,failed_urls.json,scraped_from_urls.json - Use
--freshto clear progress and start over
For NJ statutes (separate from NJAC regulations):
python backend/scripts/nj_statutes.pyFor NY statutes (from OpenLegislation API):
# First: get a free API key at https://legislation.nysenate.gov/
# Set NY_OPEN_LEG_API_KEY in .env
# Test with child care-relevant laws first:
python backend/scripts/ny_statutes.py EXC SOS PBH EDN
# All consolidated laws (~95 laws):
python backend/scripts/ny_statutes.pyKey flags:
| Flag | Description |
|---|---|
EXC SOS PBH EDN |
Specific law IDs (positional args) |
--fetch-only |
Only cache API data, skip DB |
--skip-fetch |
Use cached JSON, skip API calls |
--skip-db |
Skip database operations |
--hierarchy-only |
Only update parent_id relationships |
--fresh |
Clear all caches and re-fetch |
--delay 0.5 |
Custom delay between API calls (default: 0.2s) |
| Cost | Time |
|---|---|
| Free (API key required) | Hours (network-bound) |
After ingesting statutes, scan regulation text for statute citations and link them:
cd backend
# Scan NYCRR for NY state law citations (default):
python3 scripts/link_ny_statutes.py --dry-run
python3 scripts/link_ny_statutes.py
# Scan RCNY for NY state law citations:
python3 scripts/link_ny_statutes.py --source nyc --dry-run
python3 scripts/link_ny_statutes.py --source nyc
# Filter by law or title:
python3 scripts/link_ny_statutes.py --law SOS --title 18
python3 scripts/link_ny_statutes.py --source nyc --title 47| Flag | Description |
|---|---|
--dry-run |
Preview citations found without writing to DB |
--source ny|nyc |
Which regulations to scan: ny (NYCRR, default) or nyc (RCNY) |
--law SOS |
Only link citations to a specific law |
--title 18 |
Only scan regulations from a specific title |
| Cost | Time |
|---|---|
| Free | Minutes |
Parsed from local HTML files:
cd backend
# Parse HTML and save to JSON (no DB):
python3 scripts/nyc_statutes.py --parse-only
# Full pipeline (parse → JSON → DB → hierarchy):
python3 scripts/nyc_statutes.py
# Use cached JSON (skip re-parsing):
python3 scripts/nyc_statutes.py --skip-parse| Flag | Description |
|---|---|
--parse-only |
Only parse HTML + save JSON, skip DB |
--skip-parse |
Use cached JSON, skip HTML parsing |
--skip-db |
Skip database operations |
--hierarchy-only |
Only update parent_id relationships |
--charter-only |
Only parse Charter sections |
--admin-only |
Only parse Administrative Code sections |
| Cost | Time |
|---|---|
| Free (local HTML file) | ~30s parse, ~1min DB ingest |
Parsed from ALP XML export. Manual download from American Legal Publishing: https://codelibrary.amlegal.com/codes/newyorkcity/latest/NYCrules/0-0-0-1 → "Rules of the City of New York" → "Download" → select XML format
Extract XML.zip to data/rcny_xml/ (the XML files should be at data/rcny_xml/XML/*.xml).
cd backend
# Parse XML and save to JSON (no DB):
python3 scripts/rcny_parser.py --parse-only
# Full pipeline (parse → JSON → DB → hierarchy):
python3 scripts/rcny_parser.py
# Use cached JSON (skip re-parsing):
python3 scripts/rcny_parser.py --skip-parse
# Parse only Title 1 (for testing):
python3 scripts/rcny_parser.py --title 1| Flag | Description |
|---|---|
--parse-only |
Only parse XML + save JSON, skip DB |
--skip-parse |
Use cached JSON, skip XML parsing |
--skip-db |
Skip database operations |
--hierarchy-only |
Only update parent_id relationships |
--title N |
Only parse a specific title number |
| Cost | Time |
|---|---|
| Free (local XML files) | ~4s parse, ~2min DB ingest |
Parsed from ALP HTML export. The Health Code (Articles 1-205 under Titles I-V) covers child care, environmental sanitation, vital statistics, etc.
Manual download from American Legal Publishing: https://codelibrary.amlegal.com/codes/newyorkcity/latest/NYCrules/0-0-0-1 → Navigate to "New York City Health Code" → use browser "Save As" to save the full page HTML.
Save as newyorkcity-ny-1_health.html in the project root.
cd backend
# Parse HTML and save to JSON (no DB):
python3 scripts/health_code_parser.py --parse-only
# Full pipeline (parse → JSON → DB → hierarchy):
python3 scripts/health_code_parser.py
# Use cached JSON (skip re-parsing):
python3 scripts/health_code_parser.py --skip-parse
# Parse only Title III (child care articles):
python3 scripts/health_code_parser.py --title III| Flag | Description |
|---|---|
--parse-only |
Only parse HTML + save JSON, skip DB |
--skip-parse |
Use cached JSON, skip HTML parsing |
--skip-db |
Skip database operations |
--title ROMAN |
Only parse a specific title (e.g., III) |
| Cost | Time |
|---|---|
| Free (local HTML file) | ~1s parse, ~1min DB ingest |
Note: Health Code records use the same rcny_* DB fields as RCNY data, with
source_index offset at 100,000 to avoid collision. state='NYC' for both.
cd backend
# Scan NYCRR for NYC citations (default):
python3 scripts/link_nyc_statutes.py --dry-run
python3 scripts/link_nyc_statutes.py
# Scan RCNY for NYC citations:
python3 scripts/link_nyc_statutes.py --source nyc --dry-run
python3 scripts/link_nyc_statutes.py --source nyc
# Filter by title:
python3 scripts/link_nyc_statutes.py --title 9
python3 scripts/link_nyc_statutes.py --source nyc --title 47| Cost | Time |
|---|---|
| Free | ~15s |
# Preview what would be excluded:
python backend/scripts/batch_exclude_ny_sections.py --dry-run
# Apply exclusions:
python backend/scripts/batch_exclude_ny_sections.py- Batch-excludes NY sections that are not administrative regulations
- Sets
exclude_from_analysis = TRUEwith documented reasons per category - Categories: executive orders, military code, chemical/hazmat tables, water classification tables, shellfish/geographic enumerations, fee schedules
- Idempotent — safe to re-run (skips already-excluded sections)
- See
.claude/skills/database/schema.mdfor full category breakdown
| Cost | Time |
|---|---|
| Free | Seconds |
python backend/scripts/detect_regulatory_status.py --state NY- Parses titles to mark sections as reserved, repealed, recodified, or expired
- Sets
regulatory_statusandrecodified_tocolumns onregulatory_documents - Use
--dry-runto preview,--resetto clear all statuses first
| Cost | Time |
|---|---|
| Free | Seconds |
python backend/scripts/batch_definitions.py submit --state NY # or NJ, NYC
python backend/scripts/batch_definitions.py status # poll until "ended"
python backend/scripts/batch_definitions.py process --batch-id <id>- Uses Anthropic Batch API (50% cost savings vs realtime)
- Finds sections with "definition", "glossary", "meaning of" in title/content
- Results cached in
data/outputs/batches/ - Use
--skip-doneto avoid reprocessing,--title 3Ato filter by title - Subcommands:
submit,status,process,resume(retry failed requests),continue(extract from truncated sections) - Resume failed docs:
python backend/scripts/batch_definitions.py resume --batch-id <id> - Continue truncated:
python backend/scripts/batch_definitions.py continue --state NY --dry-run
| Cost | Time |
|---|---|
| ~$ (Haiku) | Hours (batch queue, up to 24h) |
python backend/scripts/link_definitions.py run --state NY # or NJ, NYC
python backend/scripts/backfill_definition_scope.py --state NYlink_definitionscreatesdefinition_document_occurrences— links definitions to documents where terms appear (word-boundary regex matching)backfill_definition_scopeinfersscope_levelandscope_document_idfor definitions with null scope- Use
--forceto reprocess ignoring cache,--dry-runto preview
| Cost | Time |
|---|---|
| Free | Minutes |
cd backend/scripts
python3 -m cross_references.run_extraction --state NY --output db
python3 -m cross_references.run_extraction --state NYC --output db # RCNY → NYCRR links- Extracts internal citations between regulatory documents (e.g., "18 NYCRR 505.14", "section 624.8 of this Part")
- Regex-based detection → resolution to target document IDs
- Reference types: NYCRR, NJAC, NY short-form, same-section
- NYC mode: scans RCNY text for NYCRR citations, resolves against NY regulation documents
- Use
--limit 50to test on a subset,--dry-runto preview without writing - Use
--output jsonto save results to file instead of DB,--output consolefor quick summary - Can filter by title:
--title 18(NYCRR title for NY, RCNY title for NYC)
| Cost | Time |
|---|---|
| Free | Minutes |
Runs bottom-up through 5 hierarchy phases (sections → subchapters → chapters → parts → titles).
cd backend
# Run all phases for a single title (submit → poll → process → import to DB)
python3 -m scripts.summarization.cli --state NY --title 18 --run-all-phases
# NYC (flat hierarchy — only sections phase runs)
python3 -m scripts.summarization.cli --state NYC --title 3 --run-all-phases
# Dry run (preview what would be submitted)
python3 -m scripts.summarization.cli --state NY --title 18 --run-all-phases --dry-run
# Manual phase control (if needed)
python3 -m scripts.summarization.cli --state NY --title 18 --phase sections --submit
python3 -m scripts.summarization.cli --state NY --status --watch
python3 -m scripts.summarization.cli --state NY --phase sections --process- Uses Anthropic Batch API (Haiku, 50% savings)
--run-all-phaseshandles everything: submit, poll, process, import to DB for each phase- Each parent phase uses child summaries as input — must run in order
- Requires
--title <N>to prevent accidental 50K+ doc submissions - Use
--dry-runto preview what would be submitted - Can also be triggered as a post-pipeline step via the research pipeline CLI:
python3 -m scripts.research_pipeline.cli --state ny --all-issues --execute --title 18 --summarize
- Import updates
regulatory_documentswith:summary,section_type,keywords,applies_to,has_form,has_ibr
IMPORTANT: Step 6 must complete before Step 8. The research pipeline (Step 8) uses summaries as context at multiple stages:
- Tier 1 includes each document's summary in the analysis prompt
- Tier 2 fetches summaries for related documents (siblings, cross-references) to provide broader context
- Proactive context pre-fetches sibling and cross-reference summaries for richer LLM analysis
Without summaries, the pipeline falls back to truncated content excerpts or titles, which significantly degrades analysis quality. Steps 6 and 7 can run in parallel since the regex scan (Step 7) doesn't use summaries.
Multiple titles can be summarized concurrently — each run_phase call passes its batch_id to process_results, which uses batch_mapping.json for exact directory lookup instead of globbing by phase name.
Recovery: If --run-all-phases stalls or crashes after batches were submitted (e.g., deadlock on DB import, API throttling), use the recovery script to pull completed results directly from the Anthropic API and import them:
cd backend
# Recover all missing summaries for a state
python3 -m scripts.summarization.recover_batches --state NY
# Recover for a specific title
python3 -m scripts.summarization.recover_batches --state NY --title 16
# Preview without importing
python3 -m scripts.summarization.recover_batches --state NY --dry-run
# Scan more batches (default: 100, max depends on API history)
python3 -m scripts.summarization.recover_batches --state NY --limit 500Note: --run-all-phases will resubmit batches for any phase without local results.json files, even if the batch already completed on Anthropic's side. Use the recovery script first to avoid duplicate submissions.
| Cost | Time |
|---|---|
| ~$100/state, ~$2-5/title (Haiku) | Hours per phase (batch queue) |
cd backend
# Dry run — preview what would be submitted
python3 -m scripts.guidance_summarization --dry-run
# Run full pipeline (submit → poll → process → import to DB)
python3 -m scripts.guidance_summarization --run
# Filter by agency or doc type
python3 -m scripts.guidance_summarization --run --agency OCFS
python3 -m scripts.guidance_summarization --run --doc-type ADM --limit 100
# Check status / show progress
python3 -m scripts.guidance_summarization --status
python3 -m scripts.guidance_summarization --stats- Single-phase batch (guidance docs are flat, no hierarchy)
- Uses cross-referenced regulation summaries as context
- Updates
guidance_documentswith:summary,section_type,keywords,applies_to,has_form,has_ibr - Truncates very long docs (>100K chars) with head+tail strategy
| Cost | Time |
|---|---|
| ~$50-100 for all 12K docs (Haiku) | Hours (batch queue) |
cd backend
python3 -m scripts.readability_metrics --run- Computes 8 readability indices + word/sentence counts using
textstatlibrary - Pure local computation (no LLM, no API costs)
- Resumes automatically (skips docs with
word_count IS NOT NULL) - Skips documents with <100 words or <3 sentences
Key flags:
| Flag | Description |
|---|---|
--run |
Compute and save metrics to the database |
--dry-run |
Compute metrics without saving (preview mode) |
--stats |
Show current computation status |
--agency OCFS |
Filter by agency |
--limit N |
Max documents to process |
| Cost | Time |
|---|---|
| $0 (local computation) | ~1 min for 13K docs |
python backend/scripts/scan_compliance_issues.py --state ny- Regex-only scan — finds candidate matches for 49 compliance issue types
- Patterns defined in
prompts/compliance_issues.yaml - Use
--issue wet_signaturefor a single type,--dry-runto search without saving - Use
--list-issuesto see all available issue types,--limit Nto cap matches per issue type - Use
--tag scan:complianceto only scan documents with that tag (requiresdocument_tagstable, see migration 0048) - Supports
--state nycfor NYC/RCNY regulations
| Cost | Time |
|---|---|
| Free | Minutes |
cd backend
python3 -m scripts.research_pipeline.cli --state ny --all-issues --execute # or nyc, njThis is the most expensive step. Always ask before running.
- 3-tier LLM analysis on regex matches from Step 7:
- Tier 1: Filter false positives
- Tier 2: Classify burden + assign severity (1-10)
- Tier 3: Web search verification (opt-in with
--web-search)
- Default is dry-run — must pass
--executeto actually run - Default uses batch API (50% savings) — use
--realtimefor immediate results at 2x cost
Key flags:
| Flag | Default | Description |
|---|---|---|
--execute |
off | Actually run (default is dry-run preview) |
--limit N |
1000 | Max docs per issue type |
--global-limit N |
100 | Max total docs across all issues |
--max-tier 1|2|3 |
2 | How far to process |
--web-search |
off | Enable Tier 3 web search |
--realtime |
off | Use realtime API (2x cost, immediate) |
--no-cache |
off | Disable LLM API response cache |
--title N |
none | Filter by regulatory title |
--force-rerun |
off | Reprocess already-analyzed docs |
--source |
regulation |
regulation or guidance — analyze sub-regulatory guidance docs |
--agency |
none | Filter guidance by agency (e.g., OCFS, NYSDOH) |
--doc-type |
none | Filter guidance by doc type (e.g., ADM, DAL) |
--statute-mandate |
off | Run statute mandate checks after pipeline (NY only) |
--run-statute-mandate-only |
off | Run only the statute mandate check step |
--generate-summaries / --no-generate-summaries |
on with --execute |
Generate review summaries after pipeline completes |
--summaries-only |
off | Standalone: regenerate summaries for existing analyses without re-running the pipeline |
Guidance Documents (requires migration 0058):
cd backend
# OCFS childcare guidance
python3 -m scripts.research_pipeline.cli --state ny --all-issues --source guidance --agency OCFS --execute --limit 10Review Summary Generation (post-pipeline, default on with --execute):
After the pipeline completes, generates 1-2 sentence plain-language summaries for each analysis using Haiku via Batch API. Cost: ~$0.05-0.10 per 500 analyses. Summaries are saved to the review_summary column in compliance_issue_analyses.
Important: The default --source is regulation. Guidance analyses require a separate --source guidance run. The Anthropic batch API limit is 10,000 requests per batch — if you have more, run with --global-limit 10000 first, then run again (already-summarized rows are auto-skipped).
cd backend
# Standalone: generate summaries for all existing regulation analyses
python3 -m scripts.research_pipeline.cli --state ny --all-issues --summaries-only --execute --limit all --global-limit no-limit
# Standalone: generate summaries for guidance analyses (must specify --source guidance)
python3 -m scripts.research_pipeline.cli --state ny --all-issues --summaries-only --execute --source guidance --limit all --global-limit no-limit
# Regenerate existing summaries after prompt changes (--force-rerun overwrites non-null summaries)
python3 -m scripts.research_pipeline.cli --state ny --all-issues --summaries-only --execute --force-rerun --limit all --global-limit no-limit
# Filter by title or agency
python3 -m scripts.research_pipeline.cli --state ny --issue mail_only --summaries-only --execute --title 18
python3 -m scripts.research_pipeline.cli --state ny --all-issues --summaries-only --execute --source guidance --agency OCFS
# All jurisdictions (loops NY, NJ, NYC)
python3 -m scripts.research_pipeline.cli --state dangerously-run-all --all-issues --summaries-only --execute --limit all --global-limit no-limitStatute Mandate Checks (NY only, post-pipeline):
# Preview what would be checked
cd backend
python3 -m scripts.research_pipeline.cli --state ny --all-issues --run-statute-mandate-only
# Run checks on first 10 pairs
python3 -m scripts.research_pipeline.cli --state ny --all-issues --run-statute-mandate-only --execute --limit 10See .claude/skills/research-pipeline/SKILL.md for full documentation including state machine, tier triggers, and cost estimates.
| Cost | Time |
|---|---|
| $1-60+ depending on scope | Hours (batch), minutes (realtime) |
python backend/scripts/ibr_scanner.py --state ny # or nj, nyc- Detects external standards incorporated by reference (NFPA, ICC, ASTM, CFR, etc.)
- Regex search → LLM confirmation → saves confirmed IBRs
- Use
--batchfor Anthropic Batch API,--dry-runto search without saving
| Cost | Time |
|---|---|
| ~$ (Haiku) | Minutes to hours |
# Basic scan (regulations only, async)
python backend/scripts/fines_scanner.py --state ny
# Include guidance documents
python backend/scripts/fines_scanner.py --state ny --include-guidance
# Batch API (recommended for full scans)
python backend/scripts/fines_scanner.py --state ny --include-guidance --batch
# Poll a batch until complete, then auto-process results
python backend/scripts/fines_scanner.py --state ny --include-guidance --poll <batch_id>
# Process a completed batch manually
python backend/scripts/fines_scanner.py --state ny --include-guidance --process-batch <batch_id>
# Filter to specific title/part (useful for targeted scans)
python backend/scripts/fines_scanner.py --state ny --title 18 --part 413
# Dry run (regex only, no LLM)
python backend/scripts/fines_scanner.py --state ny --include-guidance --dry-run- Detects monetary charges (fines, fees, penalties, surcharges, forfeitures)
- Regex search → LLM structured extraction → saves to
fines_fees_penaltiestable - Extracts: amount, type, who pays, responsible agency, violation trigger
- Injects document metadata (NYCRR title agency, guidance doc agency) into LLM prompt for better agency attribution
- Uses Haiku model for cost efficiency
- Resume-safe: skips documents already in the database
| Flag | Description |
|---|---|
--state |
Jurisdiction (ny, nj, nyc) — required |
--include-guidance |
Also scan guidance_documents table |
--batch |
Submit to Anthropic Batch API (50% cheaper) |
--poll <id> |
Poll batch until complete, then process + save |
--process-batch <id> |
Process a completed batch |
--title N |
Filter to NYCRR title number |
--part N |
Filter to NYCRR part number |
--limit N |
Cap number of matches to process |
--dry-run |
Regex search only, no LLM calls |
| Cost | Time |
|---|---|
| ~$1-3 (Haiku batch) | ~6 min for 5,500 docs |
# Dry run (default) — shows section count, no execution
cd backend && python3 -m scripts.barrier_pipeline.cli --state ny --title 18
# Filter by part (e.g., child care Parts 413-419)
cd backend && python3 -m scripts.barrier_pipeline.cli --state ny --title 18 --part 414
# Execute pipeline (limit 20 sections)
cd backend && python3 -m scripts.barrier_pipeline.cli --state ny --title 18 --execute --limit 20
# Realtime mode (faster, 2x cost)
cd backend && python3 -m scripts.barrier_pipeline.cli --state ny --title 18 --part 414 --execute --realtime
# Re-run specific document(s) by ID
cd backend && python3 -m scripts.barrier_pipeline.cli --state ny --document-id 81899 --execute --realtime
# Multiple document IDs
cd backend && python3 -m scripts.barrier_pipeline.cli --state ny --document-id 81899 78696 --execute --realtime- Single-pass enriched LLM pipeline analyzing regulatory sections for implementation complexity
- Uses 18-category framework: 15 core problem categories + 2 structural + OTHER catch-all
- Full rich context upfront: ancestor summaries, sibling summaries, cross-ref summaries, full definition text
- Results in
barrier_analyses(one row per document) +barrier_issues(one row per category finding) - Depends on: Step 5 (cross-references) and Step 6 (summaries) for context
- Uses Anthropic Batch API (Sonnet 4.6) for cost efficiency
| Cost | Time |
|---|---|
| ~$15-40/title | Hours (batch processing) |
CLI flags:
--state ny|nj|nyc— Jurisdiction to analyze (required)--title N— Filter by title number--part N— Filter by part number (e.g., 413 for child care)--limit N— Max sections (default: 100)--execute— Actually run (default is dry-run)--include-all-types— Include reserved, definitions, scope sections (excluded by default)--force— Re-analyze sections that already have Barrier analyses--realtime— Use realtime API instead of batch (faster but 2x cost)--no-cache— Disable LLM API response cache--document-id ID [ID ...]— Target specific document ID(s) (auto-disables exclude_analyzed)
python backend/scripts/refresh_hierarchy_stats.py --state NY # or NJ, NYC- Precomputes hierarchy statistics for frontend performance
- Updates:
child_count,descendant_count,descendant_analyzed_count,compliance_issue_count,descendant_compliance_issue_count - Run after any step that changes document counts or analysis results
- Use
--structure,--analysis,--compliance, or--statute-linksto refresh only specific stat types - Use
--all-statesto refresh all states at once
| Cost | Time |
|---|---|
| Free | Seconds |
Separate from the regulation pipeline. Imports sub-regulatory guidance documents (ADMs, LCMs, INFs, GIS, TSBs, etc.) from the nys-documents repo into the guidance_documents table.
Prerequisite: Clone the nys-documents repo as a sibling directory (i.e., ../nys-documents relative to this project root).
cd backend
# Preview what would be imported
python3 -m scripts.guidance_pipeline --dry-run
# Import all guidance documents
python3 -m scripts.guidance_pipeline --execute
# Import only a specific agency
python3 -m scripts.guidance_pipeline --execute --agency OCFS
# Gap analysis: identify missing docs by agency/year
python3 -m scripts.guidance_pipeline --gap-analysis
python3 -m scripts.guidance_pipeline --gap-analysis --gap-format json| Option | Description |
|---|---|
--dry-run |
Preview without inserting |
--execute |
Run the import (or execute clean-refs/detect-supersedure) |
--gap-analysis |
Run gap analysis on imported data |
--clean-refs |
Clean junk canonical_ref values (dry-run by default, add --execute to apply) |
--detect-supersedure |
Detect supersedure relationships (dry-run by default, add --execute to apply) |
--supersedure-report |
Print summary of all supersedure relationships in the DB |
--agency OCFS |
Filter by agency |
--source-project childcare |
Filter by source project |
--limit 100 |
Limit number of docs |
--source-dir PATH |
Path to nys-documents repo (default: ../nys-documents) |
--classify |
Rule-based classification of guidance docs (dry-run by default, add --execute to apply) |
--regex-scan |
Scan guidance docs for compliance issue regex patterns |
--extract-content-refs |
Extract NYCRR + guidance cross-references from document content |
--doc-type TYPE |
Filter regex scan to a specific doc_type |
Extract NYCRR and guidance-to-guidance cross-references from guidance document content:
cd backend
# Preview what would be extracted
python3 -m scripts.guidance_pipeline --extract-content-refs
# Extract cross-references from guidance document content
python3 -m scripts.guidance_pipeline --execute --extract-content-refs
# Filter by agency
python3 -m scripts.guidance_pipeline --execute --extract-content-refs --agency OCFSClassify guidance documents by policy category and scan for compliance patterns:
cd backend
# Preview classification distribution
python3 -m scripts.guidance_pipeline --classify
# Apply classification to DB
python3 -m scripts.guidance_pipeline --classify --execute
# Scan classified docs for compliance issue patterns (preview only, no DB writes)
python3 -m scripts.guidance_pipeline --regex-scan
# Scan and save matches to compliance_issues table
python3 -m scripts.guidance_pipeline --regex-scan --execute
# Filter scan by agency or doc_type
python3 -m scripts.guidance_pipeline --regex-scan --agency OCFS
python3 -m scripts.guidance_pipeline --regex-scan --execute --doc-type ADMAfter importing guidance documents, detect which docs supersede/replace others:
cd backend
# Step 1: Clean junk canonical_ref values (preview first)
python3 -m scripts.guidance_pipeline --clean-refs
python3 -m scripts.guidance_pipeline --clean-refs --execute
# Step 2: Detect supersedure relationships (preview first)
python3 -m scripts.guidance_pipeline --detect-supersedure
python3 -m scripts.guidance_pipeline --detect-supersedure --execute
# Step 3: View report
python3 -m scripts.guidance_pipeline --supersedure-reportDetection uses three strategies (in order):
- Revision chains: Docs with
-R1/-R2suffix linked to base ref - Content regex: Scans first 3000 chars for "Cancelled", "replaces", "supersedes", "revision of" patterns
- Title keywords: Titles containing "revised", "amended", "cancels" etc.
Sets supersedes, superseded_by, revision_of columns and status='superseded' on replaced docs.
| Cost | Time |
|---|---|
| Free (no LLM calls) | ~2-5 minutes for full import |
For specific client-requested documents at known URLs, a manifest-based workflow is used:
Step 1: Scrape (in nys-documents repo)
cd ../nys-documents
npm run scrape -- --scraper=curated-manifest # reads data/raw/curated-manifest.json
npm run merge # adds to master_catalog.csv
npm run download # fetches files (Playwright for sites that need a browser)
npm run extract # extracts textStep 2: Import (in this repo)
cd backend
python3 -m scripts.guidance_pipeline --execute # imports from enriched_catalog.jsonStep 3: Check coverage
cd backend
python3 -m scripts.guidance_pipeline --import-manifest data/client_requested_docs.json
python3 -m scripts.guidance_pipeline --import-manifest data/client_requested_docs.json --group H| Option | Description |
|---|---|
--import-manifest PATH |
Check DB coverage against a manifest JSON file |
--group X |
Filter to a specific group within the manifest (A-H) |
Manifests live in two places:
nys-documents/data/raw/curated-manifest.json— source of truth for scraping (used bycurated-manifest.tsscraper)backend/data/client_requested_docs.json— tracking copy for coverage checks
| Group | Agency | Count | Notes |
|---|---|---|---|
| A | OCFS Policy Statements | 11 PDFs | Needs Playwright |
| B | OCFS CCDBG pages | 5 HTML | Needs Playwright |
| C | OCFS Inspection Tools | 4 PDF/DOCX | Needs Playwright |
| D | OCFS ADMs/INFs | 2 PDF | Needs Playwright |
| E | Justice Center | 2 HTML | New agency |
| F | DOS | 2 HTML | Already in DB |
| G | NYSED | 4 mixed | Partially in DB |
| H | NYC DOHMH | 9 mixed | New agency (NYC jurisdiction) |
| Cost | Time |
|---|---|
| Free (no LLM calls) | ~5-15 minutes for download + extract |
To process a single title (e.g., NY Title 18) through the full pipeline:
# 1. Cross-references (extracts internal citations for pipeline context)
cd backend/scripts && python3 -m cross_references.run_extraction --state NY --title 18 --output db
cd ..
# 2. Summaries (batch API — submit and wait for completion)
python3 -m scripts.summarization.cli --state NY --title 18 --run-all-phases
# 3. Research pipeline through Tier 3 with web search (after summaries complete)
python3 -m scripts.research_pipeline.cli --state ny --all-issues --title 18 \
--max-tier 3 --web-search --execute \
--limit all --global-limit no-limit
# 4. Statute mandate checks (NY only, after research pipeline)
python3 -m scripts.research_pipeline.cli --state ny --all-issues --title 18 \
--run-statute-mandate-only --execute
# 5. Refresh hierarchy stats
python backend/scripts/refresh_hierarchy_stats.py --state NYOrder matters: Cross-references and summaries must finish before the research pipeline starts (the pipeline uses both as LLM context). The statute mandate check runs after the pipeline since it operates on pipeline results. Refresh stats runs last.
Estimated cost per title: ~$10-25 (summaries ~$2-5, pipeline ~$5-15, mandate checks ~$1-3)
NYC pipeline is the same as NY, but note:
- NYC has RCNY + Health Code data (both use
state='NYC') - RCNY hierarchy is flat (title/chapter/section — no subchapter/part)
- Health Code hierarchy is also flat (sections only, no chapter/title rollup in DB)
- Summarization only runs the sections phase for flat hierarchies
- Use
--skip-summary-checkon the research pipeline if many parent stub docs have no content
cd backend
# 1. Cross-references
(cd scripts && python3 -m cross_references.run_extraction --state NYC --output db)
# 2. Summaries (sections phase only for flat hierarchy)
python3 -m scripts.summarization.cli --state NYC --title 3 --run-all-phases
# 3. Research pipeline through Tier 3
python3 -m scripts.research_pipeline.cli --state nyc --all-issues --title 3 \
--max-tier 3 --web-search --execute \
--limit all --global-limit no-limit --skip-summary-check
# 4. Refresh hierarchy stats
python backend/scripts/refresh_hierarchy_stats.py --state NYCNote: --skip-summary-check is needed because many NYC docs are structural parent stubs with no content, making the summary percentage look low even when all summarizable sections are done.
LLM-based classification scripts that add normalized fields to existing data. All use the shared EnrichmentRunner (lib/enrichment.py) — supports batch API (Haiku, cheaper) and realtime with automatic retry + Sonnet escalation for failures.
| Command | Purpose | Cost |
|---|---|---|
python3 -m scripts.enrich_who_pays --execute --batch |
Normalize who_pays into 15 categories |
~$1-2 |
python3 -m scripts.enrich_fines_activity --execute --batch |
Classify fines by 17 activity categories + volume + justification | ~$2-4 |
python3 -m scripts.enrich_guidance_audience --execute --batch |
Classify guidance audience direction/expertise/tags | ~$6-7 (full) |
All scripts default to dry-run. Use --limit N to test on a subset first.
Filters Barrier findings into actionable recommendations through multi-round LLM evaluation.
| Command | Purpose | Cost |
|---|---|---|
cd backend && python scripts/barrier_tournament.py --full |
Full pipeline: Pass 1 + Pass 2 + top examples + JSON | ~$15 |
cd backend && python scripts/barrier_tournament.py --domain childcare |
Childcare-specific themes + fixes | ~$1 |
cd backend && python scripts/barrier_tournament.py --domain permitting |
Permitting-specific themes + fixes | ~$1 |
cd backend && python scripts/barrier_tournament.py --verify |
Web search verification on Pass 1 semifinalists | ~$5 |
cd backend && python scripts/barrier_tournament.py --build-json |
Rebuild tournament_results.json (no API calls) | $0 |
cd backend && python scripts/import_tournament_results.py |
Import tournament_results.json into DB tables | $0 |
cd backend && python scripts/classify_topics.py --execute |
Multi-label topic classification (50 topics) | ~$3-5 |
cd backend && python scripts/validate_verbatim.py --execute |
Fix is_verbatim flags on text_examples | $0 |
cd backend && python scripts/fix_text_examples.py --execute |
Re-extract verbatim text_examples via batch | ~$3 |
Outputs:
backend/outputs/barrier_strategies/tournament_results.json— unified results for frontendbackend/outputs/barrier_domains/{domain}/analysis.json— domain-specific analysis- DB tables:
tournament_patterns,tournament_examples,tournament_named_findings
Prompts: prompts/tournament.yaml (all prompts loaded from YAML)
| Command | Purpose |
|---|---|
cd backend && python3 -m scripts.fix_fabricated_quotes --execute |
Fix remaining fabricated justification_quotes in fines activity classifications |
cd backend && python3 -m scripts.research_pipeline.resume_guidance_run |
Resume a guidance research pipeline run safely after a crash (checks in-flight batches, polls to completion, then re-runs pipeline) |
cd backend && python3 -m scripts.research_pipeline.resume_guidance_run --check-only |
Check status of in-flight batches without running anything |
cd backend && python3 -m scripts.research_pipeline.recover_tier1_batch --batch-id <id> --state NY |
Recover Tier 1 results from Anthropic batch when pipeline crashed before import |
These scripts help plan runs without making API calls or writing to the database.
| Command | Purpose |
|---|---|
python backend/scripts/estimate_compliance_run.py |
Token/cost/time estimates for research pipeline |
python backend/scripts/analyze_compliance_coverage.py |
Document overlap across issue types |
python backend/scripts/analyze_patterns.py |
Regex pattern coverage analysis (DuckDB) |
python backend/scripts/verify_langfuse.py |
Test Langfuse observability integration |
| Directory | Contents |
|---|---|
data/scraped_data/ |
NY scrape cache + progress files |
data/nj_scraped_data/ |
NJ scrape cache + progress files |
data/outputs/batches/ |
Batch processing results by issue type |
backend/summaries_staging/ |
Summarization batch results before DB import |
| File | Purpose |
|---|---|
progress.json |
Visited URLs and total count |
failed_urls.json |
URLs that failed to scrape |
scraped_from_urls.json |
Maps each URL to its referring page |
| File | Purpose |
|---|---|
batch_info.json |
Batch API ID, status, timestamps, counts |
errors.log |
Failed document IDs (for resume) |
- All scripts in
backend/scripts/ - Writes directly to PostgreSQL via
psycopg - No connection to Next.js app
- Uses
backend/lib/modules for DB, LLM, and parsing utilities
- Server actions in
frontend/apps/web/src/lib/actions/ - Queries PostgreSQL via Drizzle ORM (
frontend/packages/database/src/client.ts) - Does NOT call Python backend
- All data access is read-only (review updates go directly to DB)
python backend/scripts/nj_regulations_scraper.py --freshpython backend/scripts/refresh_hierarchy_stats.py --state NJ
# Or refresh specific stat types:
python backend/scripts/refresh_hierarchy_stats.py --structure --state NJ
python backend/scripts/refresh_hierarchy_stats.py --analysis --state NJ
python backend/scripts/refresh_hierarchy_stats.py --compliance --state NJpython backend/scripts/batch_definitions.py statuspython backend/scripts/batch_definitions.py resume --batch-dir data/outputs/batches/NJ_full_20251226One-time script to re-fetch HTML from Cornell Law and re-run the improved table converter
for documents whose content contains markdown tables. Needed after the table_to_markdown()
rewrite to fix colspan/rowspan handling.
cd backend
# Preview what would change:
python -m scripts.fix_table_content --dry-run
# Update DB (re-fetches from Cornell, ~1 req/sec):
python -m scripts.fix_table_content --execute
# Filter by state or specific doc:
python -m scripts.fix_table_content --execute --state NY
python -m scripts.fix_table_content --execute --id 200391Note: Some source HTML tables on Cornell have inconsistent cell counts (data rows with
fewer <td> elements than the header). These render misaligned on Cornell too — our converter
matches the source faithfully. See issue #16 for details.
All prompts live in the /prompts/ directory:
- Core analysis prompts:
prompts/core.yaml(definitions) - Compliance issues:
prompts/compliance_issues.yaml(49 issue types with patterns + prompts) - Frontend copy:
frontend/apps/web/src/data/prompts.yaml(copied fromprompts/core.yamlvia prebuild)
TypeScript types are auto-generated from YAML - run python scripts/generate_prompt_types.py after modifying prompts/compliance_issues.yaml.
Run pnpm prebuild in frontend/apps/web/ if the /prompts page fails in development.
Approximate LLM costs per state (as of February 2025):
| Step | Model | Est. Cost |
|---|---|---|
| 3. Definitions | Claude Haiku | ~$10-20 |
| 6. Summaries | Claude Haiku | ~$100 |
| 8. Research Pipeline (full) | Claude Haiku/Sonnet | ~$50-60 (observed, per title) |
| 9. IBR Scan | Claude Haiku | ~$5-10 |
| 9b. Fines Scan | Claude Haiku | ~$1-3 |
| Total | ~$166-193 |
Costs depend on document count and average length. NY and NJ each have ~60K regulatory documents.
| Jurisdiction | Table | Records | Source |
|---|---|---|---|
| NY (NYCRR) | regulatory_documents |
~61,540 | Cornell Law scraper |
| NJ (NJAC) | regulatory_documents |
~60,000 | NJ.gov scraper |
| NYC (RCNY) | regulatory_documents |
7,777 | ALP XML export |
| NY statutes | statutes |
41,348 | OpenLegislation API (94 laws) |
| NJ statutes | statutes |
~16,000 | NJ.gov scraper |
| NYC Charter | statutes |
850 | Local HTML |
| NYC Admin Code | statutes |
10,241 | Local HTML |
| Source → Target | Links | Match Rate | Script |
|---|---|---|---|
| NYCRR → NY laws | 6,437 | 77% matched to DB | link_ny_statutes.py |
| RCNY → NY laws | 398 | 93% matched | link_ny_statutes.py --source nyc |
| RCNY → NYCRR | 3,953 | 99.8% resolved | run_extraction.py --state NYC |
| RCNY → NYC laws | 1,626 | 92% matched | link_nyc_statutes.py --source nyc |
| State | Matches | Issue Types |
|---|---|---|
| NJ | 11,343 | 41 |
| NY | 9,571 | 41 |
| NYC | 2,008 | 41 |
| Run | Tier 1 | Tier 2 | Confirmed | Cost |
|---|---|---|---|---|
| NY Title 18 (NYCRR) | 100 | 26 → 4 burdensome | 12 mandate checks (all not mandated) | $2.13 |
| NYC Title 24 (RCNY) | 100 | 5 → 2 burdensome | — | ~$0.30 |
| NYC Title 3 (Health Code) | 140 | 53 → 15 burdensome, 25 justified | T3: 6 burdensome, 6 justified | — |
- 28,544 definitions extracted (NY only)
- 89% linked to source documents (25,724 of 28,544)
- 3,146 unlinked — see issue #31
- NJ extraction: infrastructure ready, not yet run (see
DEFINITIONS_REMAINING_WORK.md)