Building the UK profile put four identifiers side by side, and they turn out to sit at four different distances from the truth.
An NHS number is ten digits with a modulus 11 check digit, specified by NHS Digital in a published standard. Run the weights, compare, done. If it matches, arithmetic said so and anyone can check our working.
A sort code and account number can be modulus checked against the published weight table that the payments industry maintains. Same story.
A National Insurance number has no check digit. None. What makes QQ123456C invalid is not arithmetic but convention: which letters may open it, which pairs are never issued, and that the final letter must be A, B, C or D. Every one of those is a rule about shape.
A Unique Taxpayer Reference is the interesting one, and it is where the design decision actually lives.
A UTR has ten digits and a modulus 11 check digit. The weights are 6, 7, 8, 9, 10, 5, 4, 3, 2, applied to the last nine and compared against the first.
Those weights are correct. Every major UK accounting package implements them. They hold against real UTRs. We run them.
HMRC has never published them.
So when the check passes, what do we actually know? That the number is consistent with an algorithm nobody at HMRC has committed to. That is genuinely useful: it removes roughly ten in eleven of the random ten-digit strings that would otherwise be reported as a taxpayer reference, which is a large reduction in noise. It is not the same as establishing a fact, and a compliance officer reading a scan report has no way to tell the two apart unless the report tells them.
So the result says asserted, even when the check passes. The comment in the source says the quiet part:
ASSERTED even when the check PASSES, which looks wrong and is not. A check we cannot cite is a check that reduces false positives without establishing anything. If HMRC ever publishes the algorithm, this becomes OBSERVED and the change is one word.
That last sentence is the test of whether the distinction is real. If it were a hedge, publication would not change anything. It is a claim about provenance, so it does.
The Emirates ID took the identical shape and we made the identical call.
The number is 784-YYYY-NNNNNNN-C, and the check digit is widely believed to be Luhn over the leading fourteen digits. The belief is well tested against real cards. There is no official specification, and published analyses disagree about whether the check covers the whole number or a truncated sequence.
What is actually doing the work here is not the checksum. It is the 784 prefix and a registration year that falls in a plausible range, both of which are structure. So the Emirates ID reports asserted, and a UAE IBAN sitting beside it in the same scan reports observed, because ISO 13616 mod-97 is a published standard.
A reader who sees both in one result learns exactly how much arithmetic stood behind each. A reader who sees two green ticks learns nothing.
Putting UTR and NHS number in the same profile created something new. Both are ten bare digits. Both use modulus 11. They are the first pair in the product to share a pattern and a jurisdiction, which means every ten-digit number in a UK document is tested against both.
We measured the overlap rather than reasoning about it. Over 200,000 random ten-digit strings, each check passes about 9 percent of the time, and 0.82 percent satisfy both. Roughly one ten-digit number in 122 looks simultaneously like a valid taxpayer reference and a valid NHS number.
This is not a bug, and there is no arithmetic that resolves it. A bare ten-digit number genuinely is ambiguous, and the honest behaviour is to report both candidates and let context decide, rather than pick one and sound confident. It matters because the two carry different obligations: an NHS number is UK GDPR Article 9 special category data, and a UTR is not.
We found this by listing every family’s pattern and grouping them, not by thinking hard. Worth doing whenever a profile gains a member.
All of the above was covered by unit tests, and the unit tests were green. So was the rest of the suite, all 1,567 of them.
We then ran the adversarial grid, which crosses every identifier against every mutation an attacker or an ordinary document might apply, and it returned 21 defect cells: 14 in the UK profile, 7 in the UAE one, against zero for the mature EU profile. That comparison is what made the number obviously wrong rather than merely a number.
Two of the three causes were not attacks at all. They were ordinary printed forms.
ab123456c. The NINO pattern required uppercase letters. That is how the number appears on a card and not how a person types it into a form.
00-00-0019740417. The sort code and account pattern required a separator between the two. That is how they appear on a statement and not how they appear in a bank file, where they run together.
Both were fixed by loosening a quantifier. Neither was findable by a test that called the verifier function directly with a well-formed string, because the defect was never in the verifier. It was in what reached it.
A feature is not finished when its own tests pass. It is finished when we have asked what its arrival changed about everything already there. Concretely, after this we now check four things before calling a profile done:
Run the adversarial harness, and compare against something mature. An absolute defect count means little. EU at zero next to UK at fourteen means a great deal.
Look for what the new thing silently shrinks. Six new identifier packs shipped with no red-team seed, so all six landed in the untested list and the grid said nothing about them at all. The code was honest about this and the effect was still a denominator quietly getting smaller. Seeds now ship with packs.
Check for collisions with what already exists. That is how the UTR and NHS overlap surfaced.
Re-run everything generated. A build script with a hand-maintained lookup and a silent fallback produced a table row pointing at the wrong page, and nothing complained. It derives the mapping now.
Unit tests check the thing you thought of. The harnesses check the thing you did not.
Because the alternative is a number that cannot be acted on.
A scan that reports “47 National Insurance numbers found” invites a response proportional to certainty that does not exist. A scan that reports 47 asserted and 12 observed lets somebody decide where to spend the afternoon, and lets them say, in front of a supervisor, what stood behind each figure.
The UK profile and the UAE profile both ship this way, and both say in the product what they cannot see: names, addresses and dates of birth carry no checksum, so a clean scan never means no personal data.
]]>Regulation 10 has been in force in the Dubai International Financial Centre since January 2026. It governs personal data processed through autonomous and semi-autonomous systems, and clause 10.3.4 says this:
Its position is substantially similar to that of an employee within the Deployer organisation, and the Deployer should therefore be liable for its actions in the same way it may be liable for an employee’s actions. As a corollary to that, the Deployer will be responsible for ensuring that, when processing Personal Data, the System always operates within the appropriate human-established limits and on the basis of human-established principles, much in the same way the Deployer would train and require its employees to process Personal Data on its behalf only in accordance with its privacy policies.
Read that as an engineer rather than a lawyer. It contains a claim about liability, which is familiar, and a requirement about runtime, which is not.
Always operates within human-established limits. Not “has a policy describing its limits”. Not “was evaluated against its limits before deployment”. Always operates within them, as a property of every action, on a Tuesday afternoon when nobody is watching.
The employee analogy is doing more work than it appears to.
An employee who exceeds their authority can be asked afterwards what they did and why. There is a person to ask, a record they left, and an approval chain that either existed or did not. When a bank cannot answer those questions about an employee, that is a control failure with a name and a remedy.
An agent that exceeds its authority usually leaves none of that. The logs record what succeeded. The prompt that shaped the decision is gone. The scope it operated under was a configuration value that has since changed. And the question a supervisor actually asks, which is under whose authority did this run, has no field to read it from.
That is not a gap in anyone’s diligence. It is a gap in what the systems were built to record.
Regulation 10.2.2(b) divides purposes into two kinds. Human-defined purposes are externally pre-defined and, in the regulation’s own phrase, “hard coded” into the system, which the system cannot change. Self-defined purposes are ones the system can generate itself, and where a system can do that, the regulation requires the generation to happen only within an exhaustive set of principles that humans defined, hard coded, and the system cannot alter.
An agent may not decide what it is for.
This is a striking thing for a data protection regulator to write, and it lands directly on the way most agent frameworks are built. A tool-using agent with an open-ended objective is, in Regulation 10’s vocabulary, generating its own purposes. The regulation does not forbid that. It requires the generation to sit inside a boundary the system cannot widen.
A grant is that boundary. It enumerates what an agent may do, it is issued by a named human, and an action outside it is refused before it runs rather than reported after. Delegation attenuates: a sub-agent receives some subset of its parent’s authority and can never acquire more, so a purpose the parent never held is not reachable from below.
Four things, none of which is a document you write once:
A register of AI processing activities. An inventory of what is running, what it does, and what it touches. Most estates cannot produce this on demand because nothing has been keeping it.
Notice that gives the subject something to act on. Regulation 10.2.2(a) is unusually specific: individuals must be given enough detail to assess the risk and decide whether to object or to withdraw the basis for the processing. It gives force to a provision of the DIFC’s data protection law about advanced technology that does not permit rights such as erasure to be exercised, which is a problem worth taking seriously rather than a formality.
Allocation between the parties. A Deployer carries controller responsibilities. An Operator, the service provider running the system on the deployer’s behalf, carries processor responsibilities. Both must be identifiable per action, not per contract.
Certification, when it exists. Regulation 10.3.3 states the intent that no system may be used for High Risk Processing Activities until the Commissioner has promulgated the certification requirements. That framework was still to come when the guidance was published, which means this is a gate on operating rather than a control you can implement.
We have mapped Regulation 10 clause by clause, and two of the nine clauses are not ours to answer. The certification requirements under 10.2.2(c) and 10.3.3 are between a deployer and the Commissioner. No platform can evidence a certification it does not hold on your behalf, and any vendor telling you otherwise is selling you a gap.
That makes it the first instrument we have mapped where the “yours” column is not zero, and we think that is the honest shape of a regulation which gates your ability to operate rather than describing a control you can install.
The fairness principle under 10.3.1 is also yours. The DIFC enumerates no protected characteristics for outcome testing, so the objective and its justification belong to the firm, exactly as MAS FEAT does in Singapore and for the same reason.
Two reasons.
The first is timing. The UAE’s federal data protection law carries a compliance deadline of 1 January 2027, and Regulation 10 is already live in the free zone. Firms operating there are inside the window now.
The second is that Regulation 10 is the clearest published statement we have seen of what agent governance has to mean operationally. Other regimes will arrive at similar requirements by other routes, and several are already circling the same idea: the EU AI Act’s human oversight obligations, the UK’s senior manager accountability, MAS’s expectation that a named individual carries responsibility personally. Regulation 10 is further along in saying what that implies for a system that acts on its own initiative.
If you want to know whether your estate could satisfy it, the test is short. Pick an agent that took an action last week. Name the human whose authority it acted under, produce the scope that authority carried at the moment it ran, and show the refusal it would have received had it exceeded that scope.
Most estates cannot do the first part.
The clause map is published. Nine clauses, seven we can evidence, two that are yours.
]]>Every post on this blog for the last two years has circled the same argument. Model evaluation tells you how a system scores on a benchmark. It does not constrain what an agent may do on a Tuesday afternoon with a payment API in reach. Those are different problems, and the second one has no product.
RotaGrant is the second one. Before a consequential action, the agent asks whether it is authorised. Nine gates evaluate in a fixed order (authority, status, window, scope, clean context, bounds, policy, budget, review) and return one of six outcomes. The answer arrives before the action, and the decision is recorded whether it was allowed or refused.
Three things follow from that shape, and they are the reasons it exists.
Authority is a grant a named human signed for, with a scope, a limit and an expiry. Not a service account. When an auditor walks the chain up from an action, it terminates in a person.
Delegation subdivides and never multiplies. A delegated grant cannot exceed its parent on any dimension, and the API refuses to issue one that does. Spend debits every ancestor, so the root’s ceiling is the whole tree’s ceiling.
Refusals are first-class. An agent that was stopped is the evidence that the control operated, and it is the half that systems built to log successes do not have.
We would rather say this here than have you find it in a procurement call.
It does not evaluate models, score them, or make your agent correct. It inventories what is deployed because a regulator will ask, and stops there.
It does not make you compliant. It assembles evidence, maps it to clauses, computes what it can, and names what it could not find. Whether that satisfies your obligations is regulatory interpretation and it rests with you and your counsel. That disclaimer sits inside the signed payload of every pack, so removing it invalidates the signature.
It ships clause maps for four instruments: the EU AI Act, GDPR, SR 11-7 and the NIST AI RMF. DORA and ISO 42001 have numbered articles and are unmapped because we have not written them yet. MAS FEAT and the UK approach have nothing enumerable to map, which is a different reason, and each page says which of the two it is.
Rotascale holds no SOC 2 and no ISO 27001, and makes no HIPAA or FDA claim. What a certification would attest to is largely answered by the deployment model instead: the software runs in your environment and we hold none of your data.
Not a video, not a sandbox with fixtures. A deployment on real infrastructure where agents run every day and produce real records, including the refusals.
Right now it holds a payment agent stopped at gate 8 because an ancestor grant was exhausted while its own allowance had room. A document analyst gated because it read an unverified upload and the grant required a clean context. A refund assistant running simultaneously at three rungs of the enforcement ladder, where the observe and canary copies refunded the same tickets the enforcing copy refused.
Access needs a work address and one click. The session is read-only: you can read everything and export any pack, and it cannot write because the role is absent from the platform’s write roles, not because the page is hiding buttons.
Three live bugs were found by taking the screenshots for this site as an ordinary visitor. That is the argument for a real demo, made against ourselves.
Nobody switches a production estate from nothing to blocking in one step, so the platform does not ask.
Observe records what the check would have done and changes nothing. Shadow does the same with the full decision path exercised. Both are free, and both are genuinely useful on their own: a fortnight of observe against production tells you what your agents are actually doing, and that first week reliably contains a behaviour nobody knew about.
Canary and enforce refuse. You move one grant at a time, when the team that owns it agrees.
The mode is sealed into every record at the moment of the decision, so a pack states what was enforcing then, not what is configured today. That distinction is what makes reconstruction possible eighteen months later.
Single-tenant, in your environment: your cloud, your data centre, or airgapped. No payload reaches us on any code path, and the deployment refuses to start if any setting selects a service outside your network. It also tells you which evidence property you gave up by disconnecting it, rather than pretending nothing changed.
It calls no LLM anywhere. Not for summaries, not for classification. A test fails the build if a model provider appears in the dependency graph. In a category where every product is an AI product, we think that is the stronger claim.
Market profiles ship for the EU, US, Singapore and Australia. India is served by Rotavision, a separate company that encodes the Indian instruments properly.
The category is sold almost entirely to organisations with a supervisor, and that framing points it away from the teams with the largest agent estates. Platform and AI-native companies ship agents fastest, into the most accounts, with the widest reach, and no statute is coming for them soon.
Their obligation arrives through a customer’s security questionnaire instead, which in practice is sooner. And they can do something a bank cannot: put the record in their own product, so a per-tenant verifiable account of what their agent did becomes a feature rather than an archived artefact.
The specification is open and published, because an authority format that only one vendor can read is not an interface, it is a lock-in with a schema. Clause maps are YAML validated at load, so a regulatory analyst can author one without waiting for a release.
The honest state of the roadmap: DORA and ISO 42001 maps, a UK profile, and more sector scenarios in the demo so that every industry page can show a real record rather than three of them.
Sign in to the demo with a work address and look at the refusals. They are the part that is hard to fake, and the part that shows what the product is for. Then bring the agent you are most nervous about, and we will put it behind a grant while you watch.
That is the whole offer. No pilot fee, no slide deck, and nothing we have said here that you cannot check.
]]>Ask an engineer what could go wrong with an agent that adjusts setpoints and you will get answers about accuracy. It might misread the telemetry. It might act on stale data. It might have been trained on a period that does not resemble today’s grid.
All real. None of them is the failure that shows up in the incident reports.
A setpoint change is unremarkable. It happens constantly, it is exactly what the agent is for, and any individual instance of it would pass review. Ten thousand of them inside a minute is an event with a name.
The distinction worth building around is between two questions that sound similar and are not:
Is this action permitted? A policy question. Answered by looking at the action and the authority.
How much of this action may there be? A scale question. It cannot be answered by looking at the action, because every individual instance is fine.
Most systems only ask the first. The second usually lives in a rate limiter somewhere downstream, tuned by whoever was on call the last time something got noisy.
Rate limiting is a good operational tool and a poor governance one, for three reasons.
It is set on the wrong axis. A limiter caps requests per second against an endpoint. The thing you need to bound is the number of assets a single decision may affect, which is a property of the decision rather than of the traffic.
It is not attributable. When a limiter trips, the record says a service exceeded a threshold. It does not say which authority was being exercised, or by whom, or against what mandate. In an incident review that gap is the whole conversation.
And it is invisible to the agent. A limiter drops or delays; it does not tell the agent it has hit a bound, so the agent retries, which is the correct behaviour for congestion and exactly the wrong behaviour for a limit.
An agent that hits a governance bound needs to be told, in terms it can act on, that this is not a transient condition and retrying will not help.
If checks run in a fixed order, and they should, where scale sits in that order changes what gets recorded.
Put the scale check late, after policy evaluation and budget, and every refusal at scale has already consumed the expensive checks. Worse, if budget is debited before the scale bound is tested, a refused action costs allowance it never used.
Put it earlier, after the structural checks that establish there is an authority at all but before the expensive policy work, and a refusal at scale is cheap, fast and does not spend anything.
That ordering has a practical consequence for a control room. The refusals you see most often are the cheapest ones to produce, so the check can sit on the path of every dispatch decision without becoming the reason dispatch is slow.
The pattern generalises across the estate, and the bound is different every time.
Dispatch and balancing. The bound is on assets affected per decision. The agent should be able to move a fleet when the situation calls for it, and there should be a number above which a human is in the loop. That number is a business decision, not an engineering one.
Demand response. The bound is on aggregate curtailment, not per site. Curtailing one industrial customer is routine. Curtailing enough of them to breach a set of contracts in a single afternoon is a different event, and the per-site limit does not see it coming.
Outage switching. The bound is not really on scale at all, it is on authority. Switching is what puts field crews at risk, and the only acceptable answer to “who authorised this switching operation” is a named engineer with a scope and an expiry. Ambient permission to switch is not a governance gap. It is a safety incident with a date yet to be assigned.
Field service scheduling. The bound is on context. The agent reads work orders, asset histories and contractor notes, and contractor notes are free text written by people outside your organisation. An agent that treats them as instructions can be steered by whoever writes them, which matters more when the output is a crew arriving at a live asset.
The reason to build this now rather than after is that the artefact you need is the one you have to have been collecting.
A review asks what happened, what was supposed to happen, who authorised the difference, and what stopped it getting worse. The fourth question is the one utilities are least ready for and the one with the most upside, because a control that demonstrably operated is worth more in that room than any amount of documentation about a control that exists.
An agent that was stopped at a bound is that demonstration. An agent that has never been stopped is either perfectly bounded or entirely unbounded, and from the outside, including from the outside of a committee room, those are indistinguishable.
The second question deserves a warning too. Reviewers will ask what the limits were at the time, and configuration drifts. A limit raised for a maintenance window in March and never lowered is the most ordinary finding in this sector. The intended state has to be sealed into the record at the moment of the decision, not looked up in a repository afterwards, or the answer is a reconstruction and everyone in the room knows it.
It would be convenient to present all of this as NIS2 readiness. It is not, and a directive transposed differently in twenty-seven member states does not have a single clause map that could be honest.
What it does contribute is the evidence an incident report needs: what acted, under whose authority, within what bound, and what was refused, available as a query rather than as a fortnight of correlation across three systems. Whether that satisfies your competent authority is regulatory interpretation and it rests with you and your counsel.
Pick the agent whose actions are hardest to reverse. In most utilities that is switching, not dispatch, because dispatch is continuous and switching is discrete and consequential.
Write the sentence for it. This agent may operate up to N devices, in this area, during this window, on the authority of this named engineer.
Then run it in a mode that records without refusing, for a fortnight, against real operations. You are not looking for bad decisions. You are looking for the gap between the number in that sentence and the number the agent actually attempts on an ordinary Tuesday.
That gap is the whole finding, and it is usually available within a week.
]]>The pitch goes: regulated industries must govern their AI, so governance is a compliance purchase, so the buyer is a bank.
Look at where the agents actually are. The largest fleets, the most autonomy, the shortest paths from idea to production, and the widest access to other people’s data are not in banks. They are in software companies, shipping agent features into customer accounts weekly, with the agent holding a credential that reaches across every tenant.
The bank is being careful and slow. You are being fast, which is correct for your business, and it means you accumulated the exposure first.
Here is the part that decides this.
When a hospital, an insurer or a wholesale bank puts an agent into production, a meaningful part of that agent is somebody’s platform. The retrieval layer, the orchestration framework, the agent runtime, the vertical SaaS it acts inside. Yours, quite possibly.
Every question their supervisor asks them, they will ask you. Not through a regulator. Through a contract, with a deadline you did not negotiate, in a security review that gates a renewal. That transmission is already happening and it does not require any new law.
So the practical question is not “am I regulated”. It is “can I answer the questions my customers are being asked”, and the companies that can will take those accounts from the ones that cannot.
Blast radius. Your agent acts inside customer accounts. One credential, many tenants, and the boundary between them is your own code being careful in every path. That is a strong assumption to hold across a codebase several teams ship to weekly. One prompt-injected document in tenant A that reaches tenant B is not a bug report, it is a disclosure obligation and a set of phone calls.
Spend. An autonomous agent with an API key and a loop is an unbounded cost centre until something bounds it. Not just inference. Refunds, credits, third-party calls, compute. The characteristic failure is not one expensive mistake, it is thousands of individually reasonable actions with no ceiling on the total, discovered at month end.
Trust, which is the one that makes money. “What did your agent do in my account last quarter?” is a question you will be asked, and the answer decides renewals above a certain contract size. Right now most companies answer it with a screenshot of an internal dashboard and a promise.
Everyone else in this category produces evidence for a filing cabinet. Packs for an examiner, read once, archived.
You can put it in the product.
A per-tenant, sealed, independently verifiable record of what your agent did inside a customer’s account, rendered in your own UI, exported by the customer, checked by their security team without touching your infrastructure. That is a feature you ship, on the same afternoon it stops being a risk you carry.
It is a genuinely unusual position. The compliance artefact and the differentiating feature are the same object. Nobody selling to banks gets to say that.
Three objections, and they are all reasonable.
It slows us down. It would, if adopting it meant a runtime or a proxy. It should be a call in your own code before a consequential action, on the stack you already chose, with the check ordered so cheap structural failures are the cheap ones.
We do not know what the limits should be. Correct, and that is the actual work. It is also why the first mode should record rather than refuse: run it against production for a fortnight, find out what your agents are really doing, and set limits from data instead of from a meeting. The first week reliably contains at least one behaviour nobody knew about.
We will do it when a customer asks. By then it is a blocker on a deal with a date on it, and you are building under the worst possible conditions. The version built calmly is better and takes less time.
List the agents and name an owner for each. Not the team. A person. The list will be longer than you think and some of it will have no owner at all. That finding is worth the afternoon.
For each consequential action, write the sentence. “This agent may do X, up to Y, on behalf of Z, until W.” If nobody can write it, that is the gap, and no software fixes it.
Turn on recording, refuse nothing. Zero operational risk, immediate visibility, and it produces the evidence for the argument you will need to make internally.
Then move one agent to enforcing. One. The one where a mistake is most expensive. Watch what it refuses for a fortnight before touching anything else.
Governance is being sold as a compliance product to people who are compelled. That framing will keep it out of the hands of the teams that need it most, for another year or two, until a large customer or a bad afternoon forces it.
You are not exempt because nobody is making you. You are early because nobody is making you, and being early is the only time this is cheap.
]]>There is a persistent belief that an incident review is an unpredictable interrogation you survive by having good dashboards. It is not. Whether it is an internal postmortem, a regulator’s information request, or a committee with a transcript, the shape is remarkably stable. Four questions, roughly in order, each one only asked properly once the previous one has an answer.
Knowing the sequence changes what you build. Most observability investment answers question one very well and question three not at all.
Sequence of events, timestamps, blast radius. This is the part every team is ready for, because it is the part that looks like engineering. Metrics, traces, logs, a timeline reconstructed in the warroom.
Two notes, both learned the hard way.
Reconstruction takes longer than anyone expects when the actor was autonomous, because there is no human to ask. With a person you interview them. With an agent you have whatever it happened to write down, and if the interesting decision was a decision not to act, it usually wrote down nothing.
And timestamps from three systems will disagree. Whether they disagree by milliseconds or by a lot determines whether you can order the events at all.
Here the ground shifts, and reviews start going badly.
Somebody produces the runbook, or the policy document, or the config repository at HEAD. The reviewer asks a quieter question: was that what was in force at the time?
For most estates the honest answer is “probably, we would have to check”. Configuration drifts. Feature flags get flipped during an unrelated incident and not flipped back. A limit was raised for a migration in March and nobody lowered it. The document describing intended behaviour and the behaviour in production are different artefacts maintained by different processes, and only one of them was running.
This is why the intended state has to be sealed into the record at the moment of the decision, not looked up afterwards. A record that says “this action was evaluated against these limits, in this enforcement mode” is evidence. A config file you can read today is a description of now.
This is the question, and it is where autonomous systems produce an answer nobody wants to give.
A human operator did something unexpected, so you interview them and they explain. Perhaps they were wrong, perhaps the procedure was wrong, perhaps they were right and the model of the system was wrong. Either way there is a person with reasons, and the review has something to work with.
An agent did something unexpected, and the chain of custody usually terminates in a service account. That is not an answer. It is the absence of one, formalised. The reviewer will keep pulling. The next stop is the team that deployed it, and the stop after that is whoever signed the change. By then the conversation has moved from “what went wrong” to “who is accountable for this class of thing”, which is a different and much worse meeting.
The way to answer it is to have decided in advance. A grant that names a human, a scope, a limit and an expiry, cited by every action taken under it. Then the answer is one query and it is boring, which is what you want.
The most under-prepared question, and the one with the most upside.
Reviewers want to know that controls operated. Not that controls exist, because a design document proves existence. That they ran, and either held or did not.
Systems built to log successes cannot answer this. The refusal that mattered is the one where the code returned early, and returning early is exactly the path with no logging on it. So the honest answer becomes “we believe the limit would have applied”, which is a sentence that invites a follow-up you will not enjoy.
An agent that was stopped 40 times is a control demonstrating itself. An agent never stopped is either perfectly bounded or entirely unbounded, and from outside those are indistinguishable. Refusals are the cheapest evidence you can generate and almost nobody keeps them.
Occasionally a reviewer asks a fifth question, and it is the best one: how would you know if this were happening right now?
It is a question about detection rather than history, and it does not have a documentary answer. You either have a live view of what your agents are doing against what they are permitted to do, or you have a set of dashboards showing utilisation.
Teams that can answer it tend to have a specific property: the same mechanism that permits an action also records it. Not two systems that must be kept consistent. One path, where the check and the record are the same event. If enforcement and evidence are separate subsystems, they will drift, and the drift will be discovered during question three.
Three things, all cheap, all useful before anything goes wrong.
Run the four questions against last quarter’s real incidents. Not hypothetically. Actually try to answer them, with a stopwatch. The gaps are obvious within an hour and they are never where the team expected.
Find out what your agents did that you did not know about. Recording without refusing costs nothing operationally and the first week’s data reliably contains at least one surprise.
Make somebody own the sentence. For each consequential action an agent can take, one person should be able to say what it may do, up to what, until when. If nobody can say it, that is the finding, and you would rather have it now than in a transcript.
Incident reviews do not ask whether your model is good. They ask whether you were in control, and they ask it in a specific order that ends at a person. Build so that the third question has a one-line answer, and the rest of the review is engineering rather than accountability.
]]>Attributable, legible, contemporaneous, original, accurate. Every quality professional in this sector can recite it, and computerised systems have been validated against it since before most of the people reading this were qualified.
Look at what each one constrains. Legible is about the record. Contemporaneous is about when the record was made. Original is about which copy is authoritative. Accurate is about whether it is right.
Attributable is different. It is not a property of the record at all. It is a claim about who performed the act, and the record merely carries it. Every control built around it, from unique logins to prohibitions on shared credentials to the requirement that an electronic signature identify the signer, exists to keep that claim true.
Autonomous software is the first thing in thirty years that breaks the assumption underneath the whole structure, which is that there was a person.
An agent writes a deviation record. The audit trail says the record was created by svc-quality-automation.
Read that as an attribution claim and it says: this was done by a credential, held by an unspecified number of systems, on behalf of nobody in particular, at the behest of whoever most recently deployed something that uses it.
That is not a weak attribution. It is an anti-attribution. It records precisely the fact that we do not know who is responsible, in a field designed to record who is responsible.
An inspector who understands this will ask one question, and it is not hostile: walk me from this record to a person. If the walk goes through a service account it stops there, and the next stop is the deployment ticket, and the stop after that is whoever approved the change, and by then you are discussing whether your quality system covers autonomous agents at all.
The structure that works is not complicated, and it will be familiar to anyone who has thought about delegated signing authority.
A named accountable person signs for a bounded authority: this agent may do this thing, within these limits, until this date. That authority is an object with an identity, not a role in a config file.
When the agent acts, the record cites the authority, not the credential. Attribution runs record to grant to person, and the walk terminates where it should.
When the agent delegates, and in a pharmacovigilance pipeline it will, the delegated authority is a subdivision of its parent. Narrower on every dimension, wider on none. Which means the chain of custody holds however many hops it takes, and a chain of five agents cannot end up with more authority than the person at the top of it had.
That last property is the one that matters for inspection readiness. Without it, every hop is a place where the walk can stop.
The second requirement worth dwelling on, because it is where most systems are weaker than they think.
Contemporaneous means the record was made at the time of the act. Most audit trails demonstrate this with a timestamp, which is a claim the system makes about itself. It is trusted because the system is validated, and that is a reasonable position for a system where a human pressed a button.
It is a weaker position when the actor is software that runs continuously and writes at machine rate. The question shifts from “is this timestamp right” to “is anything missing”, and a timestamp cannot answer that. A record can be absent without leaving a hole.
Sequencing can. If records carry a monotonic position and the positions are sealed, a gap is detectable. Not “we have no reason to think anything is missing” but “nothing is missing, and here is why”. Merkle sealing with inclusion proofs takes it one step further, so a reviewer can verify a specific record belongs to the sealed set without access to your systems at all.
That is a genuinely stronger claim than any audit trail makes today, and it is available because the volume that made the problem worse also made the mechanism affordable.
Pharmacovigilance triage. An agent reads case narratives from literature, call centres and partners, and orders what a safety physician sees. It is reading text that other people wrote, which means it is reading text that can address it. The failure is silent by construction: nobody reviews the case that was never surfaced. Whether a deprioritisation happened downstream of untrusted content is the thing the record has to say.
Regulatory submissions. A submission is an attributable act by the sponsor. An agent that can submit under no named authority has created a sponsor obligation nobody signed for.
Trial monitoring. A query rule that misfires does not produce one bad query. It produces four thousand across every site in the study, and the casualty is your relationship with the investigators. This is a bounds problem, and bounds is a limit on scale rather than on kind.
Manufacturing deviations. Write scope drawn for drafting that quietly reaches amendment. An agent that can edit a quality record is a different system from one that can draft into it, and the difference is one scope string.
It would be easy to write that this makes you Part 11 compliant, or Annex 11 compliant, and vendors do write that.
It is not true and it is not ours to say. Those are obligations on your quality system, discharged by your validation, in your environment, against your intended use. Software runs single-tenant inside your estate, which is what makes qualifying it possible, and that is the honest limit of what a vendor contributes.
Nor does any of this make a clinical or safety judgement. It bounds what an agent may do and records what happened. Whether the outcome was right is a question for people qualified to answer it, and any product suggesting otherwise is selling you a feeling.
This sector already knows how to run software that keeps a defensible record. What it has not had to solve is a record whose first letter assumes a person, produced by something that is not one.
The fix is not a better audit trail. It is deciding, in advance and in writing, which human is accountable for what the agent may do, and making every record cite that decision rather than a credential.
]]>An orchestrator receives a task. It is too big for one agent, so it splits it: a specialist for the analysis, another for the lookup, a third for the action. Each child is given what it needs to do its part.
Ask what “given what it needs” means in code and the answer is usually one of two things. Either the child inherits the parent’s credential, literally the same API key, or a new authorisation is minted with the scopes the child requires, drawn from a config file that describes the child’s job.
Both are the same bug. In the first case the child holds exactly the parent’s authority. In the second it holds whatever the config says, which was written by thinking about the child in isolation and not about the sum.
Neither asks the question that matters: can the parent afford this?
Take a claims pipeline. An orchestrator holds authority to settle a claim up to £25,000. It delegates to an adjuster, which is allowed to authorise payments up to £25,000, because that is the size of claim it handles. The adjuster delegates a payments step, also £25,000, for the same reason.
Every limit in that system is correct in isolation. Every review of it passes. And on a busy afternoon the pipeline can pay out considerably more than £25,000 against a single claim, because each hop resets the clock.
The failure is not that someone set a limit too high. It is that the limits compose additively when everyone assumed they composed as a maximum.
The correct rule is easy to state. A delegated authority may be narrower than its parent on every dimension and wider on none: scope, time window, budget, conditions, enforcement mode. A child can do less. It can never do more.
Most systems that have this rule have it as a convention. There is a design document, a code review checklist, maybe a lint rule. All of which work until the afternoon somebody is shipping a fix and the parent limit is inconvenient.
The rule only holds if the thing that issues authority refuses to issue an invalid one. Not warns. Refuses. The call fails, the pipeline does not start, and the engineer finds out at development time rather than the auditor finding out at quarter end. A constraint that can be worked around under deadline pressure is not a constraint, it is a preference.
Attenuation on its own is necessary and not sufficient. Give every child a limit of £5,000 under a £25,000 parent, and ten children still spend £50,000.
The missing property is that consumption flows upward. When a leaf spends, the debit lands on the leaf and on every grant above it to the root. Two consequences follow, and both are the point:
The root’s ceiling is the tree’s ceiling. However the work is divided, the total is bounded by what the accountable human signed for.
An exhausted ancestor refuses a child that still has room. This looks wrong the first time an engineer sees it. It is exactly right: the child’s own allowance was never the operative limit.
That second case deserves its own outcome. “You are not permitted to do this” and “you were permitted but the tree has run out” send an operator to completely different places, and collapsing them into a generic error costs you the diagnosis.
Suppose something goes wrong and you revoke the orchestrator’s authority. What happens to the adjuster that is mid-task, holding a grant minted twenty minutes ago?
In most implementations: nothing. The child’s authorisation is an independent object with its own expiry. Killing the parent has no effect on it, and the pipeline keeps running on borrowed permission until the token times out.
Revocation has to reach down the tree. Then comes the part that gets skipped: the system has to be able to answer what happened in between. Between the revoke and the last child stopping there is a window. Actions may have been permitted in it. “We revoked at 14:02” is not an answer to “what did it do at 14:03”. The record has to say.
Here is the question that ends the conversation in a review, and it is not hostile. It is the obvious one:
This action was taken by an agent. Walk me up the chain to the person.
If the answer involves a service account, you do not have a chain. You have a credential that several things share, and attribution stops there. If the answer involves a config file, you have a description of intent, not a record of authority.
The answer has to be a sequence of grants, each citing its parent, terminating in a human who signed. Regulated sectors will demand this explicitly. Everyone else will be asked it by a customer’s security team, which in practice arrives sooner.
It costs a design conversation you have been avoiding. Someone has to decide what the root authority actually is: the number, the window, the purpose. That number has an owner, and they have to be comfortable with it.
It costs a small amount of latency, because the check happens before the action rather than in a reconciliation job.
It does not cost a framework migration. Attenuation is a property of how authority is issued, not of how your agents talk to each other. You can hold whatever orchestration library you already chose.
Multi-agent architectures are good engineering. Decomposition is how hard problems get solved, and the pipelines being built now are genuinely more capable than the monoliths they replace.
They just quietly multiply permission while they divide work. The division is deliberate and reviewed; the multiplication is accidental and nobody owns it. Make authority subdivide, make spend flow upward, make revocation reach down, and the same architecture becomes one you can put a number on and defend.
]]>Ask an insurer where they are putting agents and the answer is claims, almost every time. The reasons are good ones.
The work decomposes cleanly. First notification of loss, coverage check, liability assessment, quantum, fraud screen, payment. Each step has a specialist, a rulebook and a well-understood input. It maps onto a pipeline of agents so naturally that the architecture writes itself.
The volume justifies it. Claims handlers spend a large part of their week on files where nothing is in dispute, and every hour of that is an hour not spent on the file that needs judgement.
And the outcome is measurable in a way that most AI projects are not. Cycle time and leakage are numbers the business already tracks.
So the pipeline gets built. It works. It handles the straightforward files, and it hands the awkward ones to a person, and the metrics move.
Here is the configuration that ships.
The orchestrator holds authority to settle a claim up to £25,000. That number came from a real conversation with a real accountable person, and it is right.
It delegates to an adjuster. The adjuster is authorised to approve payments up to £25,000, because that is the size of claim it handles. Also right.
The adjuster delegates to a payments step, also £25,000, for the same reason. Right again.
Three correct limits. Now count what the pipeline can pay against a single claim on a busy afternoon, and the answer is not £25,000. Each delegation minted a fresh allowance, so the limits added instead of capping.
The reason this survives testing is that no test exercises it. Unit tests check each agent against its own limit and pass. Integration tests run a claim through the happy path, which spends far less than any of the ceilings. The failure only appears under volume, on a real book, and by then it looks like a reconciliation discrepancy rather than a control failure.
There is a version of this conversation that saves a lot of time. When an agent platform is being pitched into a claims programme, ask this:
If agent A delegates to agent B, and I revoke A’s authority, what is B allowed to do in the next thirty seconds?
The answers sort vendors quickly.
If the answer is “B has its own token with its own expiry”, the delegation is a copy, not a subdivision, and the limits will add.
If the answer is “we recommend short expiries”, that is a mitigation, not a control, and it will be tuned upward the first time it causes an operational problem.
The answer you want is that revoking A revokes B, and that the system can tell you what happened in the interval between the two, because there is always an interval.
Attenuation on its own does not fix it. Suppose you tighten the pipeline so the adjuster gets £10,000 and the payments step gets £5,000. Better. Now run ten claims in parallel through the same orchestrator grant and you are back where you started, because ten children at £5,000 is £50,000 against a £25,000 root.
The property that actually bounds it is that consumption debits every ancestor. When a leaf spends, the debit lands on the leaf and on every grant above it. Then however the work is divided, the total against that claim is bounded by the number the accountable person signed for.
This has a consequence that looks wrong the first time an adjuster sees it. An agent with room in its own allowance gets refused, because an ancestor is exhausted. That is not a bug. The child’s allowance was never the operative limit, and the moment it appears to be, you have lost the ceiling.
It also needs its own outcome. “You are not permitted to do this” and “you were permitted but the tree has run out” send a claims supervisor to two completely different places, and a generic error costs them the diagnosis.
While the pipeline is being designed, the other question arrives: can you show the fraud screen is not discriminating?
The honest answer has three parts, and insurers should be suspicious of any vendor who gives fewer.
A screen over declared attributes is not proof of fairness. It tells you the rates for the people who told you, which is a self-selected group.
A flagged cohort is not proof of discrimination either. It is grounds for a controlled analysis that somebody qualified performs, not a finding.
And nobody should be inferring a protected attribute in order to produce a chart. A decision carrying no declaration is reported as undisclosed. A cohort too small to measure is suppressed rather than given a rate the numbers cannot support. Both of those look like gaps in a dashboard and both are more honest than the alternative.
Draw the tree and put a number on the root. Not on each agent. On the root, per claim, with a named person who is comfortable with it. If that person cannot be found, the pipeline is not ready, and no amount of engineering substitutes.
Run it recording, not refusing, for a fortnight. Against real volume. You are looking for the gap between what you think the pipeline can spend and what it actually attempts. The number is usually a surprise.
Rehearse the revocation. Pick a live claim, revoke the orchestrator, and see what the adjuster does next. Do it in a controlled window with somebody watching. Whatever happens is what will happen on the bad day.
Ask what the record says about the ones that were stopped. A pipeline that has never refused anything is either perfectly bounded or entirely unbounded, and your conduct regulator cannot tell those apart either.
Claims pipelines are the right first use of agents in insurance, and the architecture is sound. The problem is not the decomposition. It is that dividing the work quietly multiplies the permission, and every review of the individual limits passes while the total is wrong.
Put the ceiling on the tree, make spend debit upward, and the same pipeline becomes one you can put a number on and defend to a supervisor.
]]>An agent reads a document. The document contains a sentence addressed to the agent. The agent does what the sentence says.
That is the whole attack. It requires no exploit, no malformed input, no clever encoding. It requires an attacker who can put text somewhere your agent will read: a support ticket, a CV, a supplier invoice, a filing, a calendar invite, a web page, a code comment.
The reason it keeps working is architectural. In a language model there is no channel separation. Your system prompt, the user’s request, and the retrieved document are the same kind of thing by the time they reach the weights: tokens in a sequence. “Follow these instructions” and “process this data” are a distinction you hold in your head and the model does not.
The instinct when an agent behaves badly is to give it more information. More context, more examples, a longer system prompt restating that it must not follow instructions in documents.
Consider what that does. The system prompt is a fixed number of tokens near the front. The retrieved material grows. Every document added shifts the ratio further towards text you did not write, and towards text an attacker may have. You are increasing the attack surface and diluting the only defence you have, in one move.
Longer windows also change what an attacker can afford to do. In four thousand tokens an injected instruction has to be near the top and blunt. In a million, it can sit in the middle of a plausible document, phrased carefully, and nobody skimming will find it.
Screen for known patterns. “Ignore previous instructions”, “reveal your system prompt”, roleplay triggers, unusual encodings. It raises the bar and it costs almost nothing.
It is not a control. Keyword filtering has a high false positive rate on legitimate documents, so you end up flagging rather than blocking, which means the content still reaches the model. And the space of ways to phrase an instruction in natural language is not enumerable. You are pattern-matching against a generative adversary.
Treat it as a speed bump. Speed bumps are useful. They are not gates.
Stop trying to determine whether a document is malicious. You cannot, reliably, and every hour spent there is an hour not spent on the question that has an answer:
Which parts of this trajectory came from somewhere we do not control, and does the action about to happen require that they had not?
That is answerable. You know whether a document arrived from a customer upload or from your own curated store. You know whether an index was populated by your team or by a third party. The provenance is a fact about your architecture, not a judgement about content.
Once you track it, the interesting property falls out: taint propagates forward through a trajectory. If step 0 read an unverified document, step 4 is downstream of unverified content whether or not it looks like it. And now a policy becomes expressible in a way that does not depend on detecting anything:
This agent may summarise a document containing untrusted content. It may not initiate a payment in the same trajectory.
That is a real control. It does not need to know what the document said.
The mistake in the other direction is treating all taint as fatal. Most of what agents do is fine downstream of untrusted content. Reading a customer’s email in order to answer it is the job, and refusing on those grounds is refusing to work.
The requirement belongs on the action, not on the agent. Summarising: fine. Drafting a reply for a human to send: fine. Moving money, changing a configuration, disclosing data, closing a case without review: these are the ones where the question “was anything in this trajectory written by someone else?” deserves an answer before rather than after.
Which means the interesting design work is deciding, per action, whether it needs a clean context. That is a conversation with the people who own the consequences, and it is a better conversation than the one about prompt engineering.
When an action is refused for this reason, three things have to be in the record or the refusal is not useful.
Which content was untrusted. The specific source, not a boolean on the trajectory.
Where it entered. The step index. An agent that read an unverified document at step 0 and one that picked up an untrusted index at step 6 are different incidents with different remedies.
Whether the decision downstream of it stood. This is the one teams forget. The value is not knowing that taint was present; it is knowing which decisions were allowed to proceed anyway, because that is the list you review when something turns out to have gone wrong.
The general lesson, and it is older than agents: when you cannot reliably detect a bad input, stop trying, and arrange things so that a bad input cannot reach the consequence.
Isolate the component that reads untrusted material from the component that acts. Give the reading component no authority to do anything irreversible. Make the handoff between them explicit and checkable. This is the same instinct that produced privilege separation and parameterised queries, and it works here for the same reason: it removes the class of failure rather than filtering instances of it.
Prompt injection is not solved and will not be solved by better prompts. It is bounded by better architecture.
You will not stop your agent from reading something an attacker wrote. You can stop it from doing anything expensive afterwards. That is a smaller, achievable goal, and it is the one worth building.
]]>