|
| 1 | +# AGENTS.md |
| 2 | +## What this is |
| 3 | + |
| 4 | +the braintrust java sdk: tools for writing evals and tracing ai apps, built on opentelemetry. we publish three artifacts (all under the `dev.braintrust` group): |
| 5 | + |
| 6 | +- `braintrust-sdk`: programmatic sdk code to write evals and instrument popular ai frameworks |
| 7 | +- `braintrust-java-agent`: autoinstrumentation (no-code instrumentation of popular ai frameworks) |
| 8 | +- `braintrust-otel-extension`: more niche. applies to autoinstrumentation users who are also running the open telemetry java agent |
| 9 | + |
| 10 | +## Architecture |
| 11 | +- `braintrust-sdk` : core sdk code. contains evals and instrumentation |
| 12 | +- `braintrust-sdk/instrumentation/*` : instrumentation modules, one per supported library version (e.g. `openai_2_15_0`) |
| 13 | +- `braintrust-api` : autogenerated client against the braintrust openapi yaml |
| 14 | +- `braintrust-java-agent` : autoinstrumentation. attaches as a `-javaagent` and rewrites bytecode to apply the instrumentation modules automatically |
| 15 | +- `braintrust-otel-extension` : fat jar packaging the sdk as an extension of the upstream otel java agent |
| 16 | +- `test-harness` : shared testing framework and utils |
| 17 | +- `btx` : spec runner that executes shared sdk specs (see below) |
| 18 | + |
| 19 | +the agent and the manual wrappers (`BraintrustOpenAI.wrapOpenAI` etc.) share the same instrumentation code under `braintrust-sdk/instrumentation/*`: the agent discovers modules via ServiceLoader and applies them with bytecode transforms, while manual users call the wrapper directly. either way, spans are exported to braintrust over otel. |
| 20 | + |
| 21 | +## See also: spec repo |
| 22 | + |
| 23 | +there's a cross-sdk spec repo that describes general sdk behavior shared across languages. we keep a shallow checkout of it to run some automated spec tests (see the btx section). |
| 24 | + |
| 25 | +repo: https://github.com/braintrustdata/braintrust-spec |
| 26 | + |
| 27 | +the ref we check out is pinned in gradle.properties as `braintrustSpecRef` |
| 28 | + |
| 29 | +the `fetchSpec` gradle task downloads the pinned ref into `btx/build/spec/` (it runs automatically before `btx:test`). the spec files themselves live under `btx/build/spec/test/llm_span/` — read those locally if you want to see what behavior is being asserted. to run against a local checkout instead, set `BTX_SPEC_ROOT=/path/to/braintrust-spec/test/llm_span`. |
| 30 | + |
| 31 | +## Build, test, lint |
| 32 | + |
| 33 | +- formatter: `./gradlew spotlessApply` |
| 34 | +- build: `./gradlew compileJava compileTestJava` |
| 35 | +- test: `./gradlew test` (scope to a module while iterating, e.g. `./gradlew :braintrust-sdk:test`) |
| 36 | +- full ci gate: `./gradlew check` |
| 37 | + |
| 38 | +use `check` as a one-time final sanity check before opening a PR, not as part of the normal dev loop. it runs every muzzle check (slow, and downloads a lot of artifacts). while iterating, prefer the scoped `build` / `test` commands above. |
| 39 | + |
| 40 | +## Testing |
| 41 | + |
| 42 | +tests are mostly written with junit5. there are a few exceptions in cases where we have to manage the test env carefully (e.g. autoinstrumentation global java logging test requires a custom env that junit can't guarantee). |
| 43 | + |
| 44 | +### Test Harness |
| 45 | + |
| 46 | +TestHarness.java provides a consistent environment for unit testing. It allows us to: |
| 47 | + |
| 48 | +- set up a fresh otel state |
| 49 | +- inspect exported otel traces |
| 50 | +- fully simulate the real braintrust backend |
| 51 | +- fully simulate various AI vendors (openai, anthropic, etc) |
| 52 | + |
| 53 | +External calls are simulated with the VCR.java test utility (see section below). |
| 54 | + |
| 55 | +### VCR (recorded HTTP) |
| 56 | + |
| 57 | +VCR allows us to: |
| 58 | + |
| 59 | +- make real external calls: `VCR_MODE=off ./gradlew test` |
| 60 | +- make real external calls and record the results to "cassette" files: `VCR_MODE=record ./gradlew test` |
| 61 | +- replay previously recorded calls instead of making real external requests: `VCR_MODE=replay ./gradlew test` |
| 62 | + |
| 63 | +The default vcr mode is `replay`. this allows us to fully test the sdk with zero external calls |
| 64 | + |
| 65 | +in replay mode, tests must pass quickly (~30 seconds) |
| 66 | + |
| 67 | +a full re-record of cassettes should be done every once in a while (1-2 months) to ensure the sdk still works with external AI vendors. |
| 68 | + |
| 69 | +to re-record all cassettes: `./scripts/re-record-cassettes.sh`. this script will take a long time to run. it deliberately turns off concurrency to reduce the chances of being rate-limited by llm providers. If a full re-recording is botched, it may require waiting over an hour before trying again because some tests exercise token caches and need that amount of time before the cache is in a state where the test can be retried (note: `off` mode should always pass because it uses nonces for cache tests). |
| 70 | + |
| 71 | +## BTX spec test runner |
| 72 | + |
| 73 | +LATER |
| 74 | + |
| 75 | +## Adding or updating instrumentation |
| 76 | + |
| 77 | +LATER |
| 78 | + |
| 79 | +## Conventions & guardrails |
| 80 | + |
| 81 | +LATER |
| 82 | + |
| 83 | +## Releasing |
| 84 | + |
| 85 | +Note: Releases require a human and should not be automatically done by an agent. |
| 86 | + |
| 87 | +Releases are driven end-to-end from a single GitHub Actions workflow. |
| 88 | + |
| 89 | +To cut a release: |
| 90 | + |
| 91 | +1. Make sure everything you want included is merged to `main` and CI is green. |
| 92 | +2. Go to **Actions → Release → Run workflow**. |
| 93 | +3. Enter: |
| 94 | + - `version`: the release version as `vX.Y.Z` (semver, no `-SNAPSHOT`). |
| 95 | + - `sha`: the **full 40-character commit SHA** on `main` you want to release. Copy it from the commit page on GitHub using "Copy full SHA". A branch name is intentionally not accepted — pinning to a SHA prevents commits that land on `main` during the approval gate from sneaking into the release. |
| 96 | +4. The job runs in the protected `release` GitHub Environment and will pause for **required-reviewer approval** before doing anything. Approve from the workflow run page (or the repo's Deployments tab). |
| 97 | +5. Once approved, the `Release` workflow will, in one job: |
| 98 | + - Validate the version and the SHA, and verify the SHA is reachable from `origin/main`. |
| 99 | + - Check out the pinned SHA and run `./gradlew check`. |
| 100 | + - Create and push the annotated tag `vX.Y.Z` pointing at the SHA (using the default `GITHUB_TOKEN` — no separate bot identity is needed since the publish steps are in the same workflow). |
| 101 | + - Check out the tag, re-run `./gradlew check`, and build release artifacts. |
| 102 | + - Create the GitHub Release with the SDK, agent, and OTel extension jars attached. |
| 103 | + - Publish to Maven Central via Sonatype, signed with the project GPG key. |
| 104 | + - Poll Maven Central until the new version is visible (this can take many hours). |
| 105 | + |
| 106 | +The Sonatype and GPG signing secrets (`SONATYPE_USERNAME`, `SONATYPE_PASSWORD`, `GPG_SIGNING_KEY`, `GPG_SIGNING_PASSWORD`) are scoped to the `release` environment. |
| 107 | + |
| 108 | +The SDK version is computed from git tags at build time (see `generateVersion()` in `build.gradle`) and embedded into `braintrust.properties`, so there are no version constants to bump in source. |
| 109 | + |
| 110 | +### Re-publishing a failed release |
| 111 | + |
| 112 | +If the workflow fails partway through, re-run **Release** with the same version. The workflow detects that the tag already exists, skips tag creation, and resumes from the build/publish steps against the existing tag. GitHub Release asset uploads use `--clobber` so partial uploads from a prior run are replaced. |
0 commit comments