Skip to main content

Benjamin
Charity

Published: October 7, 2026

Regulated AI Has Two Problems. Neither of Them Is the Model.

Reading time: 16min

Who authorized this, and can you prove what happened. Healthcare has asked those two questions of every system that touches a chart for thirty years. Agents didn't change the questions. They changed who has to answer them.

When AI in healthcare comes up, the first questions are about the model. Does it hallucinate. Is it accurate enough. Will the vendor sign a BAA. Those are real questions, and none of them is the first one I ask.

Every system that has ever written to a patient record had to answer two things before anyone trusted it. Who was allowed to do this. And afterward, can we prove what happened. EHRs answer them. Pharmacy systems answer them. Billing systems answer them (badly, but they answer them). An AI agent that reads a chart, decides a referral is needed, and sends an email inherits both questions, and inherits them in a harder form, because the thing acting isn't a person and isn't quite a program either.

I come at this from pharma, where controlled substance ordering, PDMP reporting, and DEA audits put both questions to every system involved. I've spent the last few years building for this gap. What follows is the shape the two problems take once you have to ship, the design decisions that survived contact with production, and the places where an honest system admits what it can't do.

An old brass key tied to a blank paper tag, resting on an open handwritten ledger in a dim records room

Problem one: authority

The service-account trap

The first instinct, when you give an agent the ability to act, is to give it an API key. A key is a familiar object. Every integration has one.

A key says: whatever is holding this may call these endpoints. It doesn't say who decided that, within what bounds, or until when. And it gives nobody a way to stop it short of rotating the key and hoping nothing else was using it.

For a nightly job that copies claims to a data warehouse, that's fine. For something that reads a chart and then takes an action in the world, it's a liability with no record of who authorized it. At that point the agent is a long-lived credential with no consent behind it.

An agent in a regulated setting needs a different object: a credential with four properties.

Delegated. A named human sits at the root of every agent's authority. A role, a team, or "the system" doesn't count. Liability flows to that person, and the credential records who they are. When an agent delegates to another agent, the root doesn't change. The second agent acts under the same human's authority, with a chain recording every hop.

Scoped, structurally. "Read access" is not a scope. A scope is a structured object: which entities, which fields, which filter, at what rate, through which tools. Read and write are separate grants even on the same entity, because a routine regulated workflow is an agent that must read protected data to make a routing decision and must not be able to write it. OAuth's Rich Authorization Requests spec, RFC 9396, gets the shape right: authorization as a list of typed objects, not a string of space-separated words.

Any variable in a scope, such as "records where the assigned clinician is the delegating user," is resolved when the credential is issued, not when the agent acts. The credential carries the already-resolved scope. An agent that could re-resolve its own bindings at runtime could, in principle, argue its way into broader authority. It should never be in a position to argue.

Time-bound, with no refresh. Every credential expires. There's no permanent agent credential, and there's no refresh token. The refresh token is the one that surprises people, because refresh is standard OAuth. The reason to reject it is the audit chain: a refresh severs the link between the consent event and the actions taken under it. Which consent covers an action taken after the third refresh? If the answer takes more than one sentence, the record is ambiguous, and ambiguous records are what auditors are paid to find. Issue a new credential. Record a new consent.

Revocable, immediately, and never retroactively. Revocation takes effect at once. Two modes exist, and both should be chosen at issuance: kill, which cancels in-flight work and is what you want in an incident, and drain, which lets in-flight work finish but admits nothing new, which is what you want at the end of a shift. Revocation cascades down a delegation chain. It never reaches backward. A write that completed before revocation stays completed, and the record says so. Revocation prevents future actions; it doesn't unwind history, because history is the one thing an audit system must never edit.

Authorize the action, not the intent

The easy mistake here is about where the check runs. It's structural, and configuration won't fix it.

The model decides what to do. Then a tool executes. The authorization decision has to run between those two moments, against the resolved call: the actual tool, the actual arguments, the actual target. The prompt and the model's stated plan aren't what you check. You check what is about to happen.

There are two controls here, and a serious system has both. The first is what the model is offered: filter the tool list to what the credential permits, so the model is never shown a capability it can't legitimately use. The second is what the model is permitted to execute: when it emits a tool call, check that call against the scope before anything runs. The first reduces wasted turns. The second is the actual gate. Systems that only do the first are trusting the model to be well-behaved, which is a different thing from being authorized.

When the gate fails, it must fail closed and it must fail loudly, with a typed error the agent can read. A silent no-op is the worst outcome: the model learns nothing, retries or improvises, and the audit record can't tell a refused action from one that was never attempted. A typed refusal lets the agent adapt: escalate to a human, fall back, or stop. And it leaves a record of what was attempted and why it was refused.

The last control is counterintuitive until you've been burned by it. A tool's arguments don't tell you how many records it will return. "Find patients matching this filter" is a legitimate call whether it returns four rows or forty thousand. So the bulk-egress control can't live on the request. It has to bound the result: count the record-shaped items coming back, and refuse the result if it exceeds the tool's ceiling, before anything downstream sees it. Record the size of the attempted read without inspecting the content. That's your primary defense against an agent draining a database one well-formed query at a time.

The line that cannot be delegated

Everything above is about bounding what an agent may do. There's one category where the answer isn't "bound it" but "never."

Authority to do something irreversible can be delegated. An organization can decide that a specific agent may take a specific destructive action, scoped to that one capability, bounded by the authority of whoever granted it. That's a policy choice and a reasonable one.

Authority to make a significant decision about a person cannot be delegated. No grant, however broad, satisfies it. If an action's effect on a patient is the reason consent exists in the first place, then a non-human caller taking that action requires a contemporaneous human decision, or it doesn't happen. These two properties are independent. Deleting a draft is irreversible and not decisional. Denying a referral is decisional and not irreversible. The system has to know the difference and offer no mechanism to pre-authorize the second.

And when the system does route to a human, silence is refusal. Every suspension waiting on a person carries a deadline. When the deadline passes with no answer, the recorded outcome is a timeout, which is its own disposition and is never rewritten as approval. An author can choose, explicitly and on the record, that a particular step proceeds on timeout. But proceeding is a choice someone made, visible in the audit trail, never a default the runtime picked because nobody was looking. The most dangerous failure of a consent control is the one where the work visibly succeeds and the control did nothing.

Problem two: evidence

Tamper-evident to whom?

"Immutable audit log" is a standard line on a vendor security page, and it's rarely a precise claim. Immutability isn't a single property. It's a ladder, and each rung defeats a specific adversary. So the useful question is which rung you're on, and who the adversary is on the next one.

Rung one: append-only by convention. No update method on the model, no delete path in the code. This defeats well-behaved code and nothing else. It's an easy place to stop.

Rung two: immutability enforced by the database. Triggers that reject any update or delete of a sealed row, regardless of how the SQL arrived. This defeats application bugs, ORM mistakes, and anyone writing through the application. It doesn't defeat anyone with a database credential.

Rung three: a hash chain. Each entry carries a hash of its own canonical content and the hash of its predecessor. Rewrite a row and its hash no longer matches. Delete a row and its successor points at something that isn't there. This makes direct edits detectable. It doesn't make them impossible, and it doesn't, on its own, tell you who did it or when.

Rung four: a signature on every entry, under a key the writer doesn't hold in plaintext, with the public half published somewhere the attacker doesn't control. This is the rung that separates "tamper-evident to ourselves" from non-repudiation. A per-tenant signing key means a forged entry requires that tenant's private key. Publishing the public key on an independent, unauthenticated route means a verifier can compare the key inside an export against a key fetched from a source the exporter couldn't have edited. The RFC 8615 well-known URI pattern works for this. Sign the entry's hash, not the raw content, so the chain check and the signature check share one canonical form and there's no second serialization to drift.

Rung five: an external timestamp over a commitment to the whole history. RFC 3161 timestamping from an authority whose clock you don't control. The subtle part is what you timestamp. A single row's hash attests that one row. A running fold over every hash in order, from genesis, attests the content, the order, and the count of everything up to that point. Submit the fold periodically; now a rewrite of anything inside the anchored prefix can't be reconciled with a token the adversary can't forge or re-date. This is the rung that defeats the operator of the database (including you).

Rung six: independent verification. A stranger, holding the export, the published key, and standard tooling, with none of your software in the loop, reaches the same verdict you do. Open the scheme. Publish the verifier. If verification requires trusting the party being verified, it isn't verification, it's a demo. Kerckhoffs had this right in 1883 and it hasn't stopped being right.

What the honest version admits

A system that has climbed this ladder knows what it doesn't cover, and says so.

Between two external anchors there's a window. Entries newer than the latest anchor are chained and signed but not yet time-attested. A deletion confined to that window, with the successor re-linked, isn't caught by any per-row check. The window is bounded by how often you anchor, and it has to be monitored: if the anchoring job stops, the window grows and nothing alarms unless you built the alarm.

Any vendor who tells you their log is immutable and can't tell you the size of that window hasn't built the thing they're describing.

There's a second class of problem that only shows up when the log has to live for seven years, and in healthcare that's a normal ask. HIPAA's documentation rule alone runs six years, and state record laws and customer contracts can add to it. How do you add a field to the record without rehashing history you've already exported to customers. How does a retention purge delete the oldest rows without breaking a commitment rooted at genesis. What's the difference between rotating a key and revoking one, and why does a retired key verify forever while a revoked one needs independent proof that the signature predates the compromise. Each of those is its own post. The short version is that an audit chain designed for the first week and an audit chain designed for the seventh year are different objects, and you only get to choose once.

They are one system

If you take one thing from this, I want it to be the point where the two problems stop being two.

Every authorization decision is an evidence entry. The decision to allow or refuse a tool call is itself a row in the chain, carrying who the caller was, under whose delegated authority, what capability, against what resource, and which policy layer decided. It's written durably before the action proceeds. If the log can't be written, the action doesn't happen.

The operational consequence is uncomfortable: the availability of your audit infrastructure becomes the availability ceiling of the entire system. You can build around that with dedicated storage, replication, and per-tenant failure isolation so one tenant's audit trouble doesn't halt another. What you can't do is make it optional. A pre-action record that is written "eventually" is a record that might not exist when the action already has.

With that in place, the trace of a single agent action reads, from the leaf back: the tool result, the tool call, the authorization decision, the credential it ran under, every delegation hop above it, and the consent event where a named human granted the authority at the root. An auditor walks that path from any point back to a person. Not to a role. Not to a service account. A person.

One guarantee has to be stated carefully. When an outside party asserts an identity into the record, say "this request was made on behalf of Dr. Jones," binding that assertion into the hash proves provenance of record, not truth of content. It proves this caller asserted this identifier on this entry, at write time, and nobody altered it since. It doesn't prove the named human acted. State the guarantee at that tier and no higher. Overstating it is how untrusted input becomes a false audit record; stating it precisely is what lets you put untrusted input into the chain at all.

And for the compliance officer reading over your shoulder: HIPAA's audit control standard, 45 CFR 164.312(b), asks for mechanisms that record and examine activity in systems that contain or use electronic PHI. Its integrity standard, 164.312(c)(1), asks you to protect that data from improper alteration or destruction. Neither tells you how, and in practice you can't show either one unless you can reconstruct what a record looked like before and after a change. If "before" and "after" snapshots are a per-event discretion left to each developer, that reconstruction is luck. If they're the structural shape of every state-changing record, it's a guarantee.

What it costs

None of this is free, and designs that show no cost haven't met production.

Pre-action durability puts a synchronous write on the critical path of every gated action. Per-tenant signing means a key resolved and cached on every write path, wrapped at rest, never in plaintext. Chain writes for a single tenant must be serialized, or two concurrent writers fork the chain. Verification has to stream in bounded memory, because a busy organization's chain reaches millions of rows well before its retention window ends. Scope evaluation runs on every tool call in every agent workflow, so grant counts must be bounded by policy and arguments kept small by design.

Each of those is a line item. Together they're the price of being able to say, to a regulator, with a straight face, "here is what happened, here is who authorized it, and you do not have to take our word for either."

The questions I ask now

When I look at any agentic system in a regulated setting, the model is the last thing I ask about. These come first:

  1. Who is the human at the root of this agent's authority, and where is the record that they granted it?
  2. Can this agent read the data it needs without being able to write it? Show me the grant.
  3. When the agent emits a tool call outside its scope, what happens, and what does the record say?
  4. How large can a single tool result be before something refuses it?
  5. What is the adversary your audit log defeats, and what is the adversary it doesn't?
  6. How far back can a stranger verify your export without running your code?
  7. What is the window between external anchors, and what alarms when it grows?

Vendors with real answers to these have built the thing. Vendors with "immutable" and "enterprise-grade" have built a security page.

The model is the easy part.


Build, Scale, Succeed

Join others receiving expert advice on
engineering and product development.

Newsletter Subscription

No data sharing. Unsubscribe at any time.