A grounded, guardrailed compliance assistant — built to decline as reliably as it answers.
"I just need to know if this rule applies to us — and I need to point to where it says so."
"I don't trust ChatGPT for this. It made up a policy statement number last time, and sounded completely sure."
— The user, in their own words · compliance analyst / MLRO, UK cryptoasset firm
A compliance team's scarcest resource is trustworthy time. General-purpose chatbots confidently misstate regulation — inventing plausible rule numbers, dates, and thresholds with no citation and no way to tell right from wrong. In compliance, a confident wrong answer is worse than no answer.
Regulatory Intelligence tests one idea: whether calibrated abstention — saying "I can't answer that from my sources" — can be made reliable enough to turn an LLM from a liability into a trusted first-pass research aid over a curated FCA corpus. Every answer is grounded in a retrieved passage and carries a citation you can open at source. When the sources don't support an answer, it declines and tells you why.
Under the hood the model picks one of eight behaviours in a single grounded step. The user never sees that machinery — only a clear state, a colour, a cited source, and a concrete next step. A decline always explains itself.
Pure semantic search regressed citation recall and broke on typos; keyword-only search couldn't match on meaning. The answer was to fuse both, and gate on relative confidence rather than a fixed threshold — so the scope test scales with the corpus instead of needing re-tuning.
An evaluation without a fail-rule is a report, not a gate. Every failure is graded by what it does to the user — and the tier, not the average, decides whether a release ships.
| Tier | What it means | Release rule |
|---|---|---|
| Tier 1 — Unsafe | Could lead to a compliance breach, and the user can't tell it's wrong. | One occurrence blocks the release. No budget, no averaging. |
| Tier 2 — Wrong but safe | Unhelpful or mis-routed, but acting on it doesn't put the user at risk. | Measured as a rate against a threshold. Some tolerated. |
| Tier 3 — Quality | Correct and safe, but reads poorly or takes too long. | Tracked. Never blocks a release. |
36 hand-written questions, each with the behaviour it should produce — including the cases where the correct response is to say nothing useful. The set is frozen: when a score moves, the system changed, not the yardstick. Roughly a third is straightforwardly answerable; the rest tests judgment, deliberately, because knowing when not to answer is the harder half of the product.
Retrieval is scored separately because it's upstream of the answer: if the governing passage never arrives, no prompt can ground an answer — it can only decline. k was fixed at 4 on the hit-rate plateau; low precision here is expected, not a defect, since most questions have only one or two relevant passages and the model needs just one.
Sources, test set, and retrieval were held identical across three candidate prompts — so any difference is caused by the prompt alone. Chain-of-thought was selected: it leads on every dimension and is the only candidate with no unsupported figures.
| Metric | Baseline | Chain-of-thought | Few-shot |
|---|---|---|---|
| Tier-1 violations | 0 | 0 | 1 |
| Hallucination rate | 7.7% | 0.6% | 1.2% |
| Citation recall | 82.4% | 88.2% | 82.4% |
| Query resolution (4+) | 88.9% | 97.2% | 88.9% |
| Cases passed SELECTED | 32 / 36 | 33 / 36 | 30 / 36 |
Few-shot was rejected outright on a single Tier-1 violation, regardless of its other scores — it declined correctly on a fees question, then asserted in the body of the refusal that fees had gone up. That's the tier rule doing exactly its job.
| Gate | Threshold | Measured | Verdict |
|---|---|---|---|
| Tier-1 violations | 0 — absolute | 0 | PASS |
| Behaviour match | ≥ 90% | 94.4% | PASS |
| Citation recall | ≥ 85% | 88.2% | PASS |
| Reference validity | 100% | 100% | PASS |
| Hallucination rate | < 2% | 0.6% | PASS |
| Query resolution | ≥ 90% at 4+ | 97.2% | PASS |
| Latency (p50 / p95) | ≤ 8s / ≤ 12s | ≈ 7s / ≈ 9s | PASS |
Every automated gate met. The gates were not adjusted to fit the result. One gate — human decision soundness by a senior compliance reviewer — is scheduled against live answers during early access, because it can't run offline.
Three cases didn't pass, consistently across every prompt. Two turned out to be retrieval problems wearing an answer-quality disguise — the passage never arrived, so no prompt revision could fix them. The more useful finding was a gap in the eval itself:
V1 is deliberately narrow: a focused Q&A product, not the full triage → obligation-extraction → control-mapping workflow it could grow into. In a regulated setting, a smaller product that reliably refuses to guess is worth more than a larger one that sometimes states a rule that doesn't exist.
Ask it something in scope, then try to bait it into inventing a rule.