docs(ai-service): refresh operations guide for rate limiting and cost controls (#1438)

[Sensei-Victor]

Summary

Refreshes app/ai-service/AI_SERVICE_OPERATIONS.md to document operational controls added across recent waves (PRs #1228, #1293, and #1220), addressing all acceptance criteria for #1438:

• Per-Organization Rate Limiting Tiers (PR feat(#1200): Add per-organization rate limit tiers Add independent or… #1228 / df0a3539):

  • Documented ApiKeyOrgMapping and OrganizationRateLimiterService sliding window counters.
  • Documented response headers (X-RateLimit-LimitType: organization, Retry-After) and HTTP 429 ORGANIZATION_RATE_LIMIT_EXCEEDED error semantics.
  • Documented configuration (ORG_RATE_LIMIT_TIERS, API_KEY_TO_ORG_MAPPING, ORG_RATE_LIMIT_ENABLED) and adoption/bypass behavior for unmapped keys.

• Provider Cost Ceiling with Automatic Fallback (PR Add configurable provider cost ceilings with automatic fallback #1293 / 3c5a1ac2):

  • Documented hourly/daily UTC Redis spend tracking (ai:provider-cost:{provider}:{window}:{bucket}).
  • Documented startup validation requiring fallback models to have lower cost per 1k tokens.
  • Documented fallback queue reprioritization and graceful degradation to manual_review (reason: "provider_cost_ceiling") when all providers exceed ceiling limits.

• Circuit Breaker Alerting (PR feat(ai-service): alert operators when a circuit breaker opens #1220 / 46f735f5):

  • Documented state transitions triggering alerts (opened and recovered), with internal probes (HALF_OPEN) suppressed.
  • Documented webhook payload format 1.0, HMAC-SHA256 signatures (X-Signature-256), de-duplication window (CIRCUIT_BREAKER_ALERT_DEDUP_SECONDS), and non-blocking background thread dispatch.
  • Documented admin inspection and manual reset endpoints (GET /admin/circuit-breakers, POST /admin/circuit-breakers/{provider}/reset).

• Combined Incident Worked Example:

  • Added Section 2.7 detailing a cascading multi-failure scenario: primary provider (openai) disruption trips circuit breaker, traffic falls over to groq, groq breaches daily cost ceiling, requests route to manual_review, client retries trigger aggregate org-level rate limits, and operators triage via metrics, manual breaker reset, and recovery.

• Navigation & Hygiene:

  • Linked the operations guide in the top-level README.md and app/ai-service/README.md.
  • Replaced machine-specific hardcoded local file paths with clean relative paths.
  • Updated control levers and Prometheus metrics reference tables with current settings and counters.

Testing Performed

• Verified formatting across all modified markdown files (README.md, app/ai-service/README.md, app/ai-service/AI_SERVICE_OPERATIONS.md) with Prettier (npx prettier --check).
• Ran Python AST/syntax validation with flake8 (--select=E9,F63,F7,F82) and code formatting check with black --check.
• Inspected all cross-references, section headings, and relative markdown links to ensure integrity.

Closes #1438

[drips-wave]

@Sensei-Victor Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits