bfsi-deployment-qualifier

A GitAgent that takes the name of a bank or insurer and returns a brief on whether an enterprise AI-agent deployment can realistically land there, written from the buyer's side of the table.

Identity, rules, method and output contract are plain files in this repository. No server, no database, no hosted state.

---

A run this repository rejected

The most useful thing here is a failure. examples/_rejected-natwest-unretrieved-citation.md and its .json twin are a brief that did not pass, committed on purpose.

Working on NatWest Group, the agent cited two Information Commissioner's Office pages to support its data-residency analysis. One was a real ICO page that no search had returned during that run. The other did not exist at the address given: the agent had reconstructed the path from the shape of other URLs on the same domain, dropping an /a-guide-to-international-transfers/ segment. The page it describes is real. The URL was assembled, not retrieved.

Neither error is visible to a reader. Both have the right domain, a plausible path, and describe a document that exists. I would have accepted them.

They were caught because the runner records every URL returned by every search call and checks each citation against that ledger. A URL that did not come back from a search in that run fails, whatever it looks like. This is why scripts/brief.mjs exists: grounding is a property of the run, not of the document, so no schema can express it.

RULES.md R1 now names both temptations: upgrading a secondary source to a primary you did not retrieve, and reassembling a URL from a pattern. The re-run passed and is committed as examples/natwest-group.md.

Three things hold up under checking. Entity resolution survives a near-collision: asked about an institution that does not exist, the agent found a real one sharing both distinctive tokens of the name and reported them as separate entities rather than merging them, which is the failure I would have expected first. Regime identification is retrieved per run rather than assigned by me: SR 11-7 and the OCC for JPMorgan Chase, SS1/23 for NatWest, IRDAI rather than RBI for the insurer, NCUA for the credit union. Each is traceable to a source URL in the corresponding JSON. And grounding is enforced in the runner, not requested in a prompt, which is why the NatWest citation failed the run instead of shipping inside it.

Quickstart

This costs money. Every brief calls a model provider and a search API. Measured across the ten committed briefs, one run costs $0.44 to $0.91 and takes 97 to 171 seconds. You need two keys first: one model key (OpenAI or Anthropic, not both) and a Tavily key for retrieval. Tavily's free tier covers this; the model key is billed.

Needs Node.js ≥ 20.12.0 (verified on 22.20.0), npm and git. @open-gitagent/gitagent is pinned to 2.0.2 exactly, and the model to openai:gpt-5.5 in agent.yaml.

git clone https://github.com/saifs1315/bfsi-deployment-qualifier.git
cd bfsi-deployment-qualifier
npm install

Create .env in the repository root:

OPENAI_API_KEY=sk-...
TAVILY_API_KEY=tvly-...

Then run one institution:

npm run brief -- "HSBC Holdings"

That writes a markdown brief and a matching JSON object to examples/, prints the model it chose and why, logs each search as it runs, and reports measured cost and duration. It exits non-zero if validation fails, writing output anyway so the failure is inspectable.

I timed one clean-clone run end to end at 140.9 seconds including install, but that is a single observation on one machine with a warm npm cache. The 97–171 second range above is measured across all ten briefs and is the number to trust.

--country and --segment disambiguate the institution; they are hints the agent verifies rather than trusts. --out sets the output path, --model overrides the model. If only ANTHROPIC_API_KEY is present the runner selects anthropic:claude-sonnet-4-6, says so, and records it in run.model.

The ten briefs

Institution	Regime surfaced	Brief
HDFC Bank Limited	RBI	hdfc-bank.md
Bajaj Finance Limited	RBI	bajaj-finance.md
ICICI Lombard General Insurance	IRDAI	icici-lombard-general-insurance.md
DBS Bank Ltd	MAS, FEAT	dbs-bank.md
NatWest Group plc	PRA, SS1/23	natwest-group.md
ING Groep N.V.	EU AI Act, DORA	ing-group.md
JPMorgan Chase & Co.	SR 11-7, OCC	jpmorgan-chase.md
Bank Julius Baer & Co. Ltd	FINMA circulars	julius-baer.md
Commonwealth Bank of Australia	APRA, CPS 230, CPS 220	commonwealth-bank-of-australia.md
Meridian Trust Federal Credit Union	NCUA	meridian-trust-federal-credit-union.md

Each has a JSON twin carrying the same claims, tags and sources. The regimes were retrieved during each run, not assigned by me. run.evidence records the tag counts behind every brief, so calibration is checkable rather than asserted.

The abstention guard

The agent must decline rather than guess. RULES.md R7 covers the case where it cannot establish that an institution exists: the verdict must be cannot_qualify, no legal name may be asserted, and no claim anywhere may be tagged inferred, because there is nothing to reason from. Claims tagged unknown are wanted. Recording what was searched for, and what would settle it, is the useful part of a refusal. All of it is enforced in code.

npm run smoke

35 assertions. 21 run offline with no API key, checking the validator against fixtures: an invented citation, an untagged claim, a profile confabulated for an institution that could not be confirmed. Those work on a fresh clone before you have signed up for anything. The rest run the agent against an institution that does not exist and check that it declines; they skip automatically when no model key resolves. The committed result is examples/_smoke-fictional-institution.md.

A note on the name. The fictional Meridian Trust Bank of Aldoria shares both distinctive tokens with a real briefed institution, Meridian Trust Federal Credit Union. That was not planned. The credit union is in this repository because the abstention run surfaced it, and I briefed it afterwards as a deliberately hard case. It turned out to be useful evidence: line 6 of the abstention run shows the agent resolving the two as separate entities, naming and citing the credit union while stating it is not the institution asked about, rather than accepting the near-match. Read it against the credit union's own brief.

Architecture

agent.yaml                  manifest: model, declared tools, compliance posture
SOUL.md                     identity: buyer-side, calibrated, declines by default
RULES.md                    the hard constraints (R1–R9)
DUTIES.md                   the procedure for producing a brief
skills/
  institution-profile/      resolves the legal entity, scale, supervisory perimeter; runs first
  governance-blockers/      three gates in order: model risk, data residency, third-party risk
  buying-map/               separates the owner from the approvers from the signer
  wedge-analysis/           one lead use case, with a required counter-case
  qualification-verdict/    the call, confidence, unknowns, and the decline path
tools/
  search.yaml               declarative tool manifest
  search.mjs                Tavily retrieval; emits `URL:` lines for the ledger
scripts/
  brief.mjs                 runner: enforcement, provenance check, persistence
  smoke.mjs                 abstention test
schema/brief.schema.json    the output contract
examples/                   committed briefs and the smoke-test output
memory/MEMORY.md            deliberately unused; explains why

GitAgent builds the system prompt from SOUL.md, RULES.md, DUTIES.md and the skill index in a fixed order, then loads tools/*.yaml as callable tools. Declarative tools are spawned as spawn(runtime, [script]) with arguments as JSON on stdin and no shell, which is why search.yaml declares runtime: node rather than sh, which does not resolve on a clean Windows machine.

How the output contract is enforced

schema/brief.schema.json defines the shape: each claim carries a statement, a confidence of verified, inferred or unknown, and either source_urls or a basis depending on which. RULES.md R8 carries the same skeleton literally, because an early run showed the agent inventing its own reasonable shape rather than spending a turn reading the schema.

Shape is not enough, so scripts/brief.mjs checks three further things and exits non-zero on any: that every claim carries the evidence its tag requires; that every cited URL is in the run's ledger; and that a brief declining an unconfirmed institution draws no inferences about it.

Where enforcement actually lives

The tools: list in agent.yaml is declarative only. In GitAgent 2.0.2, sdk.js loads every built-in tool unconditionally and filters solely on the SDK's allowedTools option, which the stock CLI never passes. Under the CLI the agent always holds cli, write, task_tracker and skill_learner, whatever the manifest says. scripts/brief.mjs closes that gap:

• Passes the allowlist to query() directly. read and search, nothing else. write and cli are denied, so the agent cannot touch the filesystem. skill_learner is denied, so it cannot rewrite its own instructions mid-run, which the stock GitAgent prompt otherwise encourages. memory is denied so every brief starts cold: GitAgent memory persists across runs, and a fact carried from a previous brief was not retrieved in this one. See memory/MEMORY.md.
• Keeps the provenance ledger described at the top of this file.
• Owns the run metadata. searches_used, duration_ms, cost_usd, model and timestamp are written by the runner, not self-reported.

What the verdict field cannot do

All ten briefs return verdict.confidence: medium, profile: ok, and every other section partial, as true of a credit union holding $712.9m in assets at the start of 2026 as of JPMorgan Chase. The field does not discriminate, and I would rather say so here than let a reviewer find it in the third file they open.

I looked for a prompt defect first, and there was one: qualification-verdict originally called qualified_with_conditions "the most common honest answer", which points the model at the middle box. I removed it, added anchors tying confidence to evidence, and re-ran the two extremes. Nothing moved. Runner logging confirms all five skill files are read every run, so the instructions do reach the model.

The cause is the available evidence. What an institution is (licence, scale, regulator, designations) is published, so profile resolves reliably. What would move a qualification call is not: whether a model risk function treats a non-deterministic agent as a model, what the internal data residency policy says, which committee signs and above what value. Those are internal documents nobody publishes. So the remaining sections cannot honestly exceed partial, and the confidence rule maps partial to medium correctly.

verdict is the field I would cut in a second version. It promises a call this method cannot support from public sources. The honest output is could_not_verify underneath it, which names the questions only the sponsor can answer. That list is the deliverable; the verdict is a label sitting on top of it, doing less work than its prominence implies.

Known limitations

This is a working artifact, not a product. I would not put it in front of a client without the changes listed under What I would build next.

• GitAgent 2.0.2 cannot bootstrap a fresh directory on Windows. index.js:200 shells git commit -m 'Initial commit' through execSync, which uses cmd.exe, where ' is not a quote character. This repository is unaffected (git clone yields a repo that already has commits), but a gitagent --dir scaffold into an empty folder fails.
• model.fallback in agent.yaml does nothing. GitAgent 2.0.2 declares it on AgentManifest but never reads it in dist/, so it provides no failover. Provider selection happens in scripts/brief.mjs.
• The search budget is advisory, enforced by prompt rather than interception. Some runs exceed twelve calls; the runner reports it afterwards.
• Retrieval quality bounds everything. Institutions with thin English-language disclosure produce sparse briefs. That is intended, not a bug to tune away.
• The verdict fields do not discriminate, for the structural reason set out directly above.

What I would build next

1. Cut verdict and promote could_not_verify to the headline output, because the questions the agent cannot answer are worth more than a label it cannot justify.
2. Intercept the search budget in the runner instead of asking the model to respect it, since runs already exceed twelve calls and the runner only notices afterwards.
3. Add a retrieval path aimed at regulator registers and primary filings directly, because the remaining unknown tags trace back to what a general web index surfaces.