Skip to main content

Benjamin
Charity

Published: November 1, 2026

Six Levels of a Tamper-Evident Audit Log

Reading time: 22min

Every audit log can be edited by someone. The design question is who, and whether it would show.

The cheapest tamper-evident audit log is a table the application only inserts into. There is no update method and no delete path anywhere in the code. That log is tamper-evident against your own well-behaved code, and that is a real thing to protect against. When a security page says "immutable audit log" and nothing more, it hasn't said who the log is protected against. Usually the answer is the company's own code. The hard part is making the log tamper-evident against the person who runs the database. If you run the system, that's you.

A stack of documents bound with a green ribbon and sealed with red wax on a dark wooden desk

Part one of this series, Regulated AI Has Two Problems. Neither of Them Is the Model., described immutability as six levels, each one stopping a different adversary. This article works through each level: how it works, who it stops, and who gets past it. Then two decisions that sit outside the levels: who runs the verifier, and how much one chain should cover.

The reason this matters comes from part two, An Agent Is Not a Service Account. An agent's credential is the consent record, and every authorization decision gets written to the log before the action runs. If someone can edit that log without it showing, the consent record proves nothing, and neither does any decision that was made on it.

Level one: append-only by convention

The model has no update method and no delete path, and reviewers reject any pull request that adds one. This is where most audit tables start, and it does something real. It stops the well-meaning engineer who would otherwise clean up a log entry during an incident because it looked wrong.

It does not stop anyone who can skip the application: a migration script, a maintenance mode in the data layer, a database console. A convention holds only as long as every engineer follows it, and it leaves no record of whether they did. Nobody outside the team can check.

Level two: the store refuses the write

Move the rule into the store. In a relational database, that means a trigger that rejects any update or delete on a committed row no matter how the statement arrived, plus a database role for the application that doesn't have update or delete privileges on the table at all. In an object store, it means a retention lock. Either way, an ORM mistake, a bad migration, and a developer using the application's credentials all get the same refusal.

This is a real step up, and it is cheap until you need to change the table. A migration that has to touch committed rows goes through a path that disables the trigger, and that path has to leave its own evidence. If it doesn't, the people with the most access are the only ones who can write to the log without leaving a record.

Level two stops anyone writing through the application. It does not stop anyone with a privileged connection: the operator, the database administrator, whoever has the cloud console. They can drop the trigger, make the edit, and put the trigger back, and the row looks the same afterward. A restore from last quarter's backup rewinds the whole table without anyone touching a row.

Level three: a hash chain, and the fork you have to prevent

Part one covered the basic hash chain. Each entry stores a hash of its own content and the hash of the entry before it, so a rewritten row no longer matches its hash, and a deleted row leaves the next one pointing at a hash that no row has anymore.

One detail part one didn't spell out changes what the chain can catch. The previous entry's hash is stored as its own field and is not included in the row's own hash. That is a deliberate departure from the textbook chain, where each hash covers the one before it and a rewrite anywhere forces you to recompute everything after it. The reason is retention. A log that has to live for seven years has to purge its oldest rows on a schedule without breaking the chain, and a later part of this series covers how. The cost is that the newest row's hash vouches only for that one row, so a rewrite means fixing one link instead of recomputing the rest of the chain. Level five closes that gap with a running digest.

A verifier can only trust the chain if two things were done right at write time: every row was hashed in one fixed canonical form, and only one writer appended at a time. The first is canonical form. The hash covers one fixed serialization of the row, the same one a verifier will rebuild from an export a year from now, and it includes the chain's identity and the row's position, so two identical events never produce the same hash. If your writer orders keys in insertion order and your verifier orders them alphabetically, every row fails, and you find out when an auditor asks you to verify two years of history. Write the canonical form down and treat it like a public API. Once it's out, changing it is a versioned, announced change, not a refactor.

The second is concurrency. Two writers read the same latest entry, each builds a new entry pointing at it, and both insert. Every row verifies on its own. A whole-chain check can see that two rows claim the same predecessor, but it can't tell you which branch is the real history, and the running digest you'll meet at level five has no order to follow. You can detect the fork, but you can't resolve it, so prevent it. Let only one writer append to a chain at a time: take a lock on the chain itself before you read the latest entry and hold it through the commit, or put a uniqueness constraint on the pair of chain and predecessor hash so the second writer fails and retries from the new latest entry. Locking the latest row isn't enough, because two writers can both read it before either locks it. Serializing caps how fast any one chain can grow, which is one reason to scope each chain to a single tenant.

The chain has limits. A hash needs no secret, so anyone with the database and the hashing code can rewrite a row, recompute its hash, and fix the one link that pointed at the old value. The verifier accepts the result. The same goes for reordering: swap two rows, re-point the links, and every row still checks. And if someone deletes a row and points the next row's link at the one before it, every per-row check passes too. I'll call that the re-linked deletion, because it comes back at levels four and five. The chain also says nothing about who changed what or when. Level three catches careless edits. Someone who understands the chain can rewrite the row and its hashes together, and the next three levels exist to catch them.

Level four: a signature from a key that never leaves the key service

Part one said to sign the hash rather than the content, so the chain check and the signature check agree on what they're checking. If the signature covers a different serialization than the hash does, one day the two checks will disagree, and the auditor will see a valid chain and an invalid signature, with nobody able to say which serializer drifted.

The private key lives in a key service that signs on request. The application sends a hash and gets a signature back, and the key never leaves the service. That protects less than it sounds like it does. It doesn't stop anyone who can act as the application, because the application can ask for a signature on any hash it likes, and a leaked application credential is enough to do that. What it does buy is that there's no key file to steal, and that every signature the service issues is itself a record you can reconcile against the number of rows written. Use one key per tenant, so a leaked or revoked key is one customer's problem and rotating it doesn't touch anyone else's history. What rotation and revocation do to verification years later is another seven-year problem.

Then publish the public half at a fixed route, separate from the export. The well-known URI pattern in RFC 8615 works for this. The route is still yours, so publishing there proves nothing on its own. What it gives a customer is a fixed address to pin: one who fetched the key last year can tell whether it has changed since, and if the key's fingerprint goes into the first anchored digest at level five, the timestamp dates it too. Publish the full set of keys with the dates each one was valid, or every export older than the last rotation fails for the wrong reason.

So who does this level stop? A rewrite now needs a new signature, so the rewrite-and-relink move from level three is gone for anyone who can write to the store but can't get a signature: a customer's admin, a database administrator whose access stops at the database, anyone who walks off with a dump. The re-linked deletion and the reorder still pass, because each remaining row's signature covers its own hash and not its link. And anyone who can get a signature out of the service is untouched, which includes the application's own operators. A signature proves the key holder produced this entry. It can't prove the key holder didn't produce a different one last month.

This level has a cost. Every write now makes a network call to the key service, and when that service is down, nothing gets written. Part one said the audit store's availability caps the whole system's; the key service is now a second cap. Per-tenant keys mean the key count grows with customers, and so does the rotation work.

Level five: a timestamp from someone else's clock

To reach the people who can get signatures, you need something they can't make: a timestamp from a clock outside the company. RFC 3161 defines the protocol. You send a hash to a time-stamping authority, and it returns a signed token that, in the RFC's words, establishes "evidence indicating that a datum existed before a particular time." The authority never sees your data, only the hash. From here on, sending a hash and keeping the token is what I mean by anchoring.

What you anchor matters more than how often. Because the previous entry's hash sits outside the row hash, the newest row's hash only vouches for that one row. So keep a running digest: start from a fixed genesis value and fold in each row hash in order. The digest after row ten thousand commits to the content, the order, and the count of all ten thousand rows. Anchor the digest on a schedule and store each token next to the digest it covers.

Two verifier rules turn that into a real check. The first is the date. The authority will stamp any hash you send, including a digest over a rewritten history, so a fresh token on old rows gives it away. Every new row has to be covered by a token dated within one anchoring interval of when the row was written. An idle chain can go longer between anchors, so check the cadence separately from the coverage. The second is custody. A token only you hold is a token you can delete, and a verifier can't tell "deleted" from "never anchored." So send a copy out of your custody on the same schedule, to the customer or to an independent escrow. Publishing it at the well-known route helps only if someone outside your company archives what's there, because the route is still yours. With both rules in place, any rewrite, reorder, or deletion of a row older than the latest anchor produces a digest that no surviving token matches. That includes the re-linked deletion from level three, as long as the row was anchored before it was deleted.

This is the level that reaches the database operator, which is you. The DEA already names your own staff. Its rules for electronic prescriptions say a hosted pharmacy application has to treat the alteration or destruction of controlled substance prescriptions "by any agent or employee of the application service provider" as an auditable event. The rule doesn't say how the audit trail has to resist those employees. But once a regulator has told you to log your own staff, a log your staff can edit is hard to defend.

RFC 3161 notes that an organization might run an authority for its own internal time-stamping. That is fine for ordering events inside one company. But an authority you run is just another key you hold, so it only convinces people inside the company.

Then there is the window. Everything after the latest anchor is chained and signed but not yet timestamped, and inside that window the re-linked deletion and the reorder still pass. You can't close the window to zero without a synchronous call to a third party on every write, and part one already said the audit store's availability is the ceiling for the whole system. Put a time-stamping authority on the write path and your uptime is now their uptime.

You can make the window small, though. An anchoring job off the write path can run every few seconds, and the authority's rate limit and bill are what cap it. Pick an interval you can defend to whoever asks, then watch it, because when the anchoring job stops, nothing else breaks. The window keeps growing until someone notices.

Level six: a stranger can verify it without your code

Everything so far still runs on your software. Level six is where that stops. A verifier holding an export, the published keys, the anchor tokens, and standard tooling reaches the same verdict you do, with none of your code in the loop. You have to publish three things for that to work.

  • The scheme itself: canonical form, hash function, chain rule, digest rule, signature algorithm, where the keys live, what a token looks like.
  • A reference verifier short enough to read in one sitting, so anyone could rewrite it from the scheme alone.
  • Exports as plain data, in a format that doesn't need your product to open.

The principle Kerckhoffs wrote down in 1883, that a system shouldn't depend on its design staying secret, is usually aimed at the people designing the cipher. Here it is a requirement. Outsiders can only check a scheme they can read.

There is a working example of this that every browser already uses. Certificate Transparency, RFC 6962, is an append-only log that browsers and domain owners verify without trusting the log operator. Its spec also names the failure level six has to handle: a log that breaks its append-only promise by presenting "two different, conflicting views" at different times or to different parties. A verifier who only ever receives exports from you has no way to know whether another verifier, or the same verifier last year, received something else. Each export can pass every check on its own, and the two can still disagree.

Certificate Transparency's answer is that clients gossip and compare what each was shown. Your answer is the anchor from level five, and comparing anchors takes a little setup. Two verifiers line up anchors by sequence number and by the row count each one covers, compare the digest each token commits to, and let the token dates say which came first. The second verifier can be the same customer holding last year's export. Browsers run this comparison automatically. Your customers won't, so give them something to compare against: publish the list of anchor tokens at the same fixed route as the keys. Anyone who saved a copy of the list can check whether a token has changed or disappeared since.

Independent verification means two strangers can check whether you showed them the same history. One stranger checking your math is only half of it. Crosby and Wallach's 2009 paper on tamper-evident logging defines the property the same way: the log has to prove it is "consistent with how it was seen in the past." Their data structure makes that comparison cheap. The comparison still only happens if someone runs it.

This level has a cost too. Publishing the scheme freezes it. Every change from then on is public and versioned, and the reference verifier is code you maintain forever, where a bug is a wrong verdict that somebody else trusts.

What the six levels catch, and what they don't

Every level leaves something uncaught, and you should be able to name what that is before a customer asks.

Caught Not caught
A row edited, reordered, or deleted once it's behind an anchor Garbage in. If the application logs something false, or logs nothing, the chain preserves that.
A rewrite anywhere, by anyone who can't get a signature New entries forged under a compromised signing key. Entries behind an anchor dated before the compromise still stand, and showing that is a seven-year problem.
A careless edit, or a deletion that leaves a dangling link, anywhere, even inside the window An insider who can edit rows, get them signed, and delete the tokens that covered them, when nobody outside your custody holds a copy.
A forged or altered export A time-stamping authority that is compromised or colluding. Anchoring to more than one is the mitigation.
A history that differs between two verifiers who compare anchors Anything written after the latest anchor and edited, reordered, or deleted before the next one. Editing takes a signature; the other two don't. Deleting the newest rows takes nothing at all, because no link points at them yet.

Wholesale destruction is in neither column. The levels tell you the store is gone. Only backups bring it back.

The garbage-in row is the easiest one to skip. The six levels prove the log wasn't changed after it was written. They say nothing about whether what was written was true. If the application records that a clinician approved a referral because a request header said so, the chain preserves the header's claim exactly as written. Part one drew that line as provenance of record versus truth of content, and all six levels sit on the provenance side.

The verifier has to run on a schedule

A tamper-evident log that nobody has verified hasn't proven anything yet. The chain, the signatures, and the anchors are all passive. They only say something when a verifier reads them, so the verifier has to run, on a schedule that doesn't depend on anyone remembering.

I've spent the last few years in pharma, where this is already written down for one class of record. The DEA's rule for pharmacy applications that handle controlled substance prescriptions requires the application to analyze its audit trail "at least once every calendar day" and generate an incident report. The pharmacy then has one business day to report anything it judges a security incident to the DEA. Whatever the number is in your domain, the shape is the same. A scheduled run, a report, and a deadline for responding.

Treat every verification failure as an incident until someone explains it, and "explained" means someone can point at the record that caused it. A planned migration that touched committed rows, a restore from backup, a key change: each of those should have written its own evidence when it happened, so the person triaging can match the failure to a record instead of to someone's memory of what happened.

There are two ways to get this wrong. Skip the schedule, and the first time anyone runs the verifier is the day a customer asks, which is the worst possible day to learn that a serializer drifted in March. Run the schedule without recording the planned changes, and every migration fails verification, people learn to close the alert, and you've trained the on-call rotation to ignore the one alarm that matters.

The verifier has two more jobs. Verify before every export, so what you hand a customer or a regulator has already passed. And alert on how old the latest anchor is, because the anchoring job is the one component whose failure breaks nothing else.

Then put a name on it. One person owns the verifier and the exports. At a small company that is the tech lead. Later it is platform or security. It is not a committee.

One chain per tenant

The chain, the signing key, the digest, and the anchor all have a scope, and the default is one of each per tenant, for three reasons.

  • Serialization. Appends to one chain run one at a time, so a single chain for the whole system puts every write behind one lock, and a busy customer slows a quiet one.
  • Verification. A customer's history verifies on its own, and the export you hand them contains only their rows. On a global chain with a linear digest, verifying anyone means verifying everyone, and one customer's export can't stand alone without the neighbors' hashes, which also tells them how busy everyone else is. A tree-shaped commitment shrinks that to a proof path, which is the trick the batched anchor below uses, but the lock and the blast radius don't change.
  • Blast radius. A compromised key, a forked chain, or a verification failure is one tenant's incident, and the other tenants' chains keep anchoring while you sort it out.

A global chain buys one thing per-tenant chains can't, which is cross-tenant ordering. Per-tenant chains can't prove that one tenant's event happened before another's, and the anchors only tell you that down to the anchoring interval. If that ordering is itself something you have to prove, or you run a single-tenant deployment with nobody to isolate from, a global chain is the right call. Per tenant is the default because when something goes wrong, it goes wrong for one customer instead of all of them.

The per-tenant cost is operational. Everything you run per chain, you now run per tenant: the verifier, the anchor-age alert, the key rotation, and one timestamp request per tenant per interval. If that is too many requests, build one small digest over all the tenant digests, anchor that once, and include each tenant's proof path in its export, along with the digest and row count the batch recorded for them. It is cheaper, and the export gets a little more complicated. Pick based on how many tenants you have and how often you anchor.

Which level you need

The right level depends on who has to be convinced.

If the only person who will ever read the log is your own engineer during an incident, levels one through three are enough, and most of the protection comes from level two. If a customer's security team is going to ask, level four gives you something to hand them, and level five answers "what if your operator edited it." If a regulator, a court, or a customer whose lawyers have no reason to trust you is on the other side, you need five and six, because the point is that they don't have to take your word.

HIPAA's integrity standard has one implementation specification, and it is addressable. It asks for an electronic mechanism to corroborate that protected health information "has not been altered or destroyed in an unauthorized manner." It doesn't say who has to be convinced, and it leaves whether you build the mechanism at all to your risk analysis. That part is yours to decide, and it is the part that sets the cost.

Stopping too low is the familiar failure. A team sits at level two with "immutable" on the security page. Going too high gets less attention. Anchoring every write to a third party, demanding a public verifier for an internal admin tool, building all six levels for a log nobody outside the team will ever ask about. Each level costs something, and paying for one that defeats an adversary you'll never face buys you nothing.

Name the adversary, and the level picks itself.

The question I'd put to any vendor, and the one I put to myself, is tamper-evident against whom. If the answer is a specific person with a specific level of access and a specific reason not to trust you, you know which level you need. If the answer is "it's immutable," ask again.


Build, Scale, Succeed

Join others receiving expert advice on
engineering and product development.

Newsletter Subscription

No data sharing. Unsubscribe at any time.