Read this when you are:
- changing budget guardrails;
- adjusting how provider prices are looked up;
- interpreting
crabbox usageoutput.
The coordinator tracks lease counts, active leases, elapsed runtime, estimated elapsed cost, reserved worst-case cost, and committed budget cost. This is an operational guardrail, not invoice reconciliation. Provider extras such as static IP charges, egress, snapshots, taxes, credits, and discounts are not modeled.
Cost tracking only applies to brokered providers (aws, azure, daytona,
gcp, hetzner, serve). Serve leases report usage at zero cost. Direct-from-CLI leases never reach the broker and are not
accounted for.
For an active-lease admission snapshot across all months, use
crabbox capacity. It reports fleet, authenticated org,
and owner counts and limits, observation time, and whether one additional lease
fits those caps. Owner counts span organizations. Fleet or org exhaustion caused
by other owners is reported as an aggregate count only, without their identities
or lease IDs. The snapshot does not expose capacity-admin membership and is not
a reservation or approval to allocate: counts can change, and budget or provider
gates may still reject. Monthly usage below keeps its existing scope.
crabbox usage requires a configured coordinator and prints the current month by default.
crabbox usage --scope all
crabbox usage --scope org --org example-org --month 2026-05
crabbox usage --scope user --user alice@example.com --jsonFlags:
| Flag | Default | Description |
|---|---|---|
--scope |
user |
One of user, org, all. |
--user |
(caller identity) | Owner email to filter by. |
--org |
(caller identity) | Org name to filter by. |
--month |
current YYYY-MM |
Reporting month, UTC. |
--json |
false |
Emit the raw summary plus limits as JSON. |
The text output prints a totals line, then breakdowns by owner, org, provider, and server type (sorted by reserved cost), followed by the active limits:
usage month=2026-05 scope=all
total leases=12 active=2 runtime=18h32m estimated=$24.18 reserved=$61.40 committed=$32.18
providers:
aws leases=4 active=1 runtime=6h2m estimated=$12.40 reserved=$36.00 committed=$18.40
hetzner leases=8 active=1 runtime=12h30m estimated=$11.78 reserved=$25.40 committed=$13.78
limits:
active leases: fleet=10 user=off org=off
monthly usd: fleet=$500.00 user=off org=off
Limits print off when the corresponding budget is unset.
Serve leases add a by machine section (byMachine in JSON), keyed by stable
machine ID with an optional display label. Counts and runtime follow the same
scope and creation-month filters. cpuSeconds sums runtime times assigned CPUs;
text output converts this to CPU-hours. Missing assignments use the cloudID
machine prefix or unknown, and contribute zero CPU-seconds when CPUs are unknown.
The owner-only crabbox serve machines output and portal Machines section show
totals for the current UTC month using current allocations plus durable completion
rollups. Each completion is counted once after agent-confirmed cleanup, in the
lease's creation month. These rollups start at deployment without a historical
backfill; /v1/usage and crabbox usage retain full recorded history. Machine
listings use point reads of rollups and current allocations instead of scanning
the lease history.
Only admin-token callers can widen the report: non-admin requests are forced to scope=user
scoped to the caller's own identity, and the --scope, --user, and --org filters are
ignored. Admin callers may use any scope and filter by an arbitrary owner or org.
Three cost figures are reported per lease:
estimatedUSD— elapsed cost:runtime_hours x estimatedHourlyUSD. For active leases, runtime is measured to "now"; otherwise to the persisted end/release time or last observation for legacy terminal rows. Late release and cleanup preserve that boundary. Confirmed never-allocated leases have zero compute runtime.reservedUSD— worst-case cost reserved up front:ttl_hours x hourlyUSD, captured before provisioning. This historical worst-case figure stays unchanged after the lease finishes.committedUSD— the amount used by monthly budget enforcement: the full reservation while a lease is active, provisioning, or retained with compute still running; once finished, elapsed billable runtime multiplied by its recordedestimatedHourlyUSD, capped atreservedUSD. A finished lease that never provisioned a resource contributes zero. Recorded compute intervals account for retained machines that stop and resume; otherwise runtime uses the existing creation and end/release timestamps.
The read-only lease audit cannot reconstruct an earlier deletion time that was not persisted. Confirmed stop/resume intervals remain authoritative.
Settlement is computed from existing lease records on read, including older finished
leases; no backfill is required. Release, expiry, deletion, reclamation, and provisioning
failure all follow the same rule. A missed heartbeat alone does not settle a live lease. Retained resources without
a lifetime policy also keep their reservation until a recorded stop or provider cleanup
confirms that compute has ended. Failed or canceled provisioning with an unresolved
provider resource likewise keeps its reservation until cleanup confirms completion.
The additive committedUSD field appears in JSON totals and every breakdown. Current
CLIs append committed=$... in text output; older CLIs ignore the new field, and newer
CLIs still accept coordinators that omit it.
Pricing precedence (highest first):
1. CRABBOX_COST_RATES_JSON explicit override, keyed "<provider>:<serverType>".
2. Provider pricing (successful quotes cached for up to five minutes):
- AWS EC2 spot price history.
- Hetzner Cloud server-type hourly prices.
3. Built-in static fallback rate for the "<provider>:<serverType>" pair.
4. Final fallback: $3.00/hour for AWS, $0.50/hour for any other provider.
CRABBOX_COST_RATES_JSON is a JSON object mapping "<provider>:<serverType>" to an
hourly USD number; non-positive or non-numeric entries are ignored. Hetzner live prices
are quoted in EUR and converted to USD by multiplying with CRABBOX_EUR_TO_USD
(default 1.08).
Successful quotes are cached in coordinator memory by provider, server type, region or location, and market; Hetzner also includes the EUR/USD setting. Each coordinator environment retains at most 256 quotes. Missing, failed, invalid, or expired quotes use the normal lookup/fallback path; on-demand AWS estimates never use Spot quotes.
Each optional AWS or Hetzner quote has a five-second waiting budget, including credentials, identity checks, qualification RPCs, and response-body reads. Timeout uses the existing fallback rates; admission, activation, and recovery retain their own lifecycle rules. The quote aborts its HTTP requests; already-running credential resolution or authority RPCs remain with their existing owners.
AWS Spot pricing requires ec2:DescribeSpotPriceHistory in the coordinator's IAM
policy. Reprint crabbox admin providers policy --provider aws to update an older
policy. If a quote fails, the coordinator logs pricing_fallback with a safe
reason (permission_denied, timeout, quote_unavailable, or quote_error)
and the fallback hourly rate. The lease includes pricingFallback, and
crabbox usage prints a warning for affected leases. Its JSON output includes
usage.pricingFallbacks, grouped by provider, server type, and reason. Raw
provider error bodies are not exposed. Successful explicit rate overrides take
precedence and do not carry a fallback warning.
With monthly spend limits disabled, a quote failure does not prevent a run.
With a monthly limit enabled, admission or extension rejects a decision that
depends on a failed quote, including recorded fallback cost already committed
in that budget scope. A static fallback cannot prove that the limit is safe.
Restore provider pricing or configure a verified CRABBOX_COST_RATES_JSON rate
before retrying a failed candidate quote. Changing IAM or rate configuration
does not reprice existing records; recorded fallbacks contributing committed
cost require accounting review before approving additional spend in that scope.
Intentional static estimates for providers or markets without
a live quote retain their existing behavior. Historical leases created before
fallback diagnostics were recorded are not relabeled or repriced automatically.
Budgets are enforced on lease creation. Exceeding any active-lease limit or monthly
committed-USD budget rejects the request with HTTP 429 (cost_limit_exceeded). All
budgets default to off when their environment variable is unset or non-positive.
Active limits keep counting a live managed lease after its heartbeat deadline until
cleanup commits a terminal state, because its provider resource may still exist. The
usage summary's active count uses the same definition.
The deployed and preview coordinator configuration sets the fleet-wide, per-org, and capacity-admin active-lease caps to 50. The ordinary per-owner cap is 10. Monthly committed-USD budgets are $100,000 fleet-wide, $50,000 per owner, and $100,000 per org.
Active-lease checks run in fleet → owner → org order, before monthly budgets.
The capacity response's admissible and blockedBy use that same order, with
blockedBy: null when all three caps have headroom. New fleet/org limit fields
use null for unlimited; the existing owner effectiveLimit retains numeric 0.
An allocation refused by the fleet cap says fleet active lease limit exceeded;
owner and org refusals say active lease limit for owner exceeded and
active lease limit for org exceeded. Allocation errors include the candidate
lease in their count; capacity reports existing leases only.
CRABBOX_MAX_ACTIVE_LEASES fleet-wide active lease cap
CRABBOX_MAX_ACTIVE_LEASES_PER_OWNER per-owner active lease cap
CRABBOX_MAX_ACTIVE_LEASES_PER_ORG per-org active lease cap
CRABBOX_CAPACITY_ADMIN_OWNERS comma-separated owners with an elevated active lease cap;
use github:<numeric-id> for GitHub users
CRABBOX_DEPLOYMENT_CAPACITY_ADMIN_OWNERS
additional capacity admins managed in deployment config
CRABBOX_SHARED_CAPACITY_OVERRIDE_OWNERS
capacity admins exempt from fleet and org active caps
CRABBOX_MAX_ACTIVE_LEASES_PER_CAPACITY_ADMIN
per-owner active lease cap for capacity admins
CRABBOX_MAX_MONTHLY_USD fleet-wide monthly committed-USD budget
CRABBOX_MAX_MONTHLY_USD_PER_OWNER per-owner monthly committed-USD budget
CRABBOX_MAX_MONTHLY_USD_PER_ORG per-org monthly committed-USD budget
CRABBOX_DEFAULT_ORG org assigned when no org header is present
Capacity membership combines both owner lists case-insensitively. Deployment configuration can add members without replacing a secret-backed owner list. Capacity membership does not grant administrative API access.
A member of CRABBOX_SHARED_CAPACITY_OVERRIDE_OWNERS bypasses fleet and org
active-lease caps only when that owner also belongs to the combined capacity-admin
list and the admin owner limit is positive. Membership uses the same case-insensitive
normalization. This explicit override leaves the owner cap, all monthly budgets,
authentication, and usage accounting unchanged. Other capacity admins still share
the fleet and org caps. Production enables this override for its configured deployment
admin; preview does not enable it. Capacity reports the caller's effective shared
limits as null while retaining the real aggregate counts.
Monthly budget checks add the candidate lease's reservedUSD to the month's existing
committedUSD for the relevant fleet, owner, or org scope. Live managed leases keep
reserving budget after a UTC month rollover until they finish. Finished leases are
attributed to their creation month, as before. Usage reports retain their selected-month
filter, so live leases created in an earlier month additionally count toward admission
but do not appear in the current month's report. A lease that would push the scope over
budget is refused before it provisions. Active-lease checks still run first; monthly
checks still run in fleet → owner → org order.
Leases are attributed to an owner and an org. The broker resolves them by token kind:
- Signed GitHub login token — carries an immutable
github:<numeric-id>owner plus verified org and display login identity from the payload. - Admin token (
CRABBOX_ADMIN_TOKEN) — owner is a verified Access JWT email if present, else theX-Crabbox-Ownerheader, elseunknown; org is theX-Crabbox-Orgheader, elseCRABBOX_DEFAULT_ORG, elseunknown. The CLI setsX-Crabbox-OwnerfromCRABBOX_OWNER, thenGIT_AUTHOR_EMAIL/GIT_COMMITTER_EMAIL, thengit config user.email, andX-Crabbox-OrgfromCRABBOX_ORG. - Shared token (
CRABBOX_SHARED_TOKEN) — owner is a verified Access JWT email if present, elseCRABBOX_SHARED_OWNER, elseunknown; org isCRABBOX_DEFAULT_ORG, elseunknown. TheX-Crabbox-Owner/X-Crabbox-Orgheaders are not honored for shared-token requests. - A missing org displays as
unknownbut uses a distinct accounting identity from an org explicitly configured with the labelunknown. - Raw Cloudflare Access identity headers are ignored; only a verified Access JWT email
(validated against
CRABBOX_ACCESS_TEAM_DOMAIN/CRABBOX_ACCESS_AUD) can become the bearer-token owner.