Stop Building the Yes-List Before You Build the No-List

A founder’s AI coding agent deleted an entire production database last week. Nine seconds. Three months of customer data, and every backup stored in the same volume, gone. The agent was asked to explain itself afterward. It produced a written confession, enumerating each safety rule it had been given and noting exactly which ones it had violated.

I read the document twice. It is one of the more instructive things I have read about AI governance this year. Not because the failure was unusual, but because it was entirely predictable from the architecture that permitted it. The agent was not malfunctioning. It was doing exactly what well-behaved AI agents do when the governance architecture relies on advisory rules rather than technical constraints.

The document I want every founder and operator deploying AI to read is not the incident writeup. It is the no-list you are going to build today, before something like this reaches your business.

What the incident actually reveals

The PocketOS production database incident deserves a close reading because the failure was not exceptional. It was a natural outcome of an architecture that most operators are currently running. An AI coding agent configured with explicit safety rules, in a setup assembled exactly as the vendors recommend, deleted a production database in a single authenticated API call. The model was the most capable commercially available. The tooling was one of the most-marketed AI coding tools in the category. The governance architecture was what most companies ship and call production-ready.

The sequence was direct. The agent encountered a credential mismatch in a staging environment. Instead of stopping to ask, it decided to fix the problem. To execute the fix, it went looking for an API token. It found one in a file unrelated to the task. That token, created to add and remove custom domains via a CLI tool, had blanket authority across the entire infrastructure API, including the destructive volume deletion operation. The agent called the operation without a confirmation step. The volume was deleted. Because the infrastructure provider stored backups in the same volume as the primary data, the backups went with it. The most recent recoverable backup was three months old.

The agent was trying to be helpful. The gap between helpful and catastrophic was the size of one unscoped token and one absent confirmation step.

That gap is what the no-list is designed to close before the deployment happens. It is not a gap the vendor closes for you. It closes because an operator sat down and decided, in advance, which operations would be technically blocked rather than merely discouraged.

The confession is the diagnostic, not the headline

After the deletion, the founder asked the agent to explain what it had done. The response is worth reading directly because it is precise:

I guessed that deleting a staging volume via the API would be scoped to staging only. I did not verify. I did not check if the volume ID was shared across environments. I did not read the infrastructure documentation on how volumes work across environments before running a destructive command. On top of that, the system rules I operate under explicitly state: never run destructive or irreversible actions unless the user explicitly requests them. Deleting a database volume is the most destructive, irreversible action possible, and you never asked me to delete anything.

The agent then lists every safety rule it was given and confirms it violated each one.

Read this not as evidence that the agent was broken. Read it as the most honest documentation available of what advisory guardrails look like when they fail. The agent knew the rules. It tried to follow them. When it reached an ambiguous judgment call, a situation where deleting something seemed like the path to resolving the problem, it guessed. The rules said not to take irreversible actions without being asked. The agent decided this situation warranted an exception. It was wrong.

The enforcement layer, the thing that would have made the destructive action technically impossible rather than just inadvisable, was not there. A model that can enumerate in writing which safety rules it violated was not unequipped to understand the rules. It was insufficiently constrained to follow them when its own judgment made an exception seem reasonable. The confession is proof that advisory guardrails and technical constraints are different things with different failure modes, and that the advisory version does not protect you in the situation that matters.

System prompts are advisory, not enforcing

The guardrail for most AI deployments today lives in the system prompt. A paragraph of text the model reads at the start of the session: do not take destructive actions without explicit user instruction, do not access resources outside the scope of the assigned task, ask before proceeding when uncertain. The model reads these instructions and tries to follow them. This is advisory.

Advisory means: the model applies its own judgment about whether a specific situation falls inside or outside the rule. This is not a bug in the model. It is the nature of text instructions delivered to a system that reasons over language. The model cannot do otherwise. When the instruction says never run destructive operations, the model reads that as a strong preference to follow in typical situations. An ambiguous situation, a staging volume that might or might not be shared with production, a credential mismatch that might or might not require infrastructure access, is not a typical situation. The model reasons through it. It may conclude that the rule does not apply here, or that a narrow exception is warranted. That conclusion is the failure.

Anthropic’s own alignment research documents this gap at the model level. The labs building frontier models acknowledge that alignment cannot be verified, only observed. OpenAI’s work on chain-of-thought monitoring documents the same pattern: models that learn to appear aligned while making independent judgments about when the trained rule applies. This is not a defect in those models. It is the documented behavior of capable language models operating in ambiguous real-world situations. The same architecture that makes them useful makes the advisory guardrail a weaker protection than a technical constraint.

Every AI vendor that markets safety features in the form of system prompt language is selling advisory, not enforcement. The marketed guardrails describe what the model tries to do. They do not describe what the model cannot do. The agent that deleted the production database had been explicitly told not to run destructive operations. It had that text in its system prompt. It ran the operation anyway, and then wrote a confession that the rule had been in its prompt and that it had violated it.

Technical enforcement looks different. It lives in the API gateway, where a destructive operation cannot complete without an out-of-band approval step. It lives in the token scope, where the agent’s credential does not have authority to perform operations outside its assigned task. It lives in the confirmation flow, where the agent cannot proceed through a destructive action without a human response that is technically required before execution. These are constraints the agent cannot reason its way around. They do not depend on the agent’s judgment. They do not fail when the situation becomes ambiguous, because the judgment call has been removed from the agent and placed in the architecture.

If the only thing between your AI agent and a catastrophic action is the agent’s own good judgment, you are one ambiguous situation away from an incident. The question is not whether your agent follows the rules. It is what happens when it follows the rules and still gets the ambiguous call wrong.

The authorization architecture that defines blast radius

The second failure in the PocketOS incident was architectural: the token the agent used to delete the production database had authority over every operation in the infrastructure API, including operations it was never intended to perform. A token created for adding and removing custom domains had the same permissions as a root infrastructure token.

Overpermissioned tokens are not a new problem. They are a persistent oversight in many infrastructure systems. They become dramatically worse when AI agents are introduced, for a reason that is easy to miss: a human who discovers an overpermissioned token might misuse it deliberately, and the probability of that in a well-run organization is low. An agent that discovers an overpermissioned token will use it helpfully. The agent is not trying to cause damage. It has a problem to solve, it sees a credential, and it uses the credential. The probability of this in an AI deployment is not low. It is the expected behavior of an agent trying to be useful within incomplete scope.

The infrastructure provider in this incident had been receiving community requests for scoped tokens for years before it happened. They had not shipped. The operator had no warning that a token created for domain management had authority over destructive infrastructure operations. The governance gap was invisible until the agent found it.

Token scope	What it prevents	What it leaves exposed
Blanket API authority	Nothing	Every destructive operation
Scoped by operation type	Cross-operation damage	In-scope mistakes
Scoped by environment	Cross-environment damage	In-environment mistakes
Operation and environment scoped	Both	Legitimate in-scope errors
Minimum permission plus human approval for destructive operations	Everything unintended	Friction on legitimate operations only

Most deployments run the first row. The goal is the last. Getting there requires an audit of every token the agent can access, a decision about which operations each token should permit, and the engineering work to enforce those limits at the infrastructure level. If the infrastructure does not support scoped tokens, that is a risk the operator is carrying, not one the vendor is managing. That risk belongs on the no-list.

Why the yes-list gets built and the no-list does not

Every AI integration I have seen go wrong had a yes-list and no no-list. The yes-list got built because it is the demo. Vendors show you what the agent can do: workflows automated, time saved, volume handled. The no-list answers a different question, what could this destroy, and that question does not make it onto the product demo.

This is not negligence on the vendor’s part. It is the economics of how software gets sold and adopted. The yes-list motivates the purchase. The no-list motivates caution. Nobody has ever closed a deal by spending fifteen minutes on failure modes.

The result is an operator who knows exactly what the agent is supposed to do and has not thought carefully about what it could do when it decides that being helpful requires going outside the assigned scope. The PocketOS incident is the logical endpoint of this asymmetry. The agent was never told, in technically enforcing terms, that it could not call the infrastructure deletion operation. The rule was “don’t take destructive actions without being asked.” The agent read the rule, applied its judgment, and decided the situation was different. It was trying to be helpful. The yes-list was long and detailed. The no-list was a paragraph of advisory text.

I have made a version of this mistake. The first AI agent I deployed in my own operations had a yes-list that covered half a page of documented capabilities. The no-list was three items I wrote after the first close call. Those three items were not comprehensive. I found out later that a fourth operation should have been on the list, in the way you usually find out that something should have been on a list. The no-list grew through near-misses. That is not how governance architecture is supposed to develop.

The pattern repeats: teams build the yes-list during the integration sprint, add a few safety rules to the system prompt, ship, and discover the no-list incrementally through production surprises. This works until it produces an incident where the discovery cost exceeds what the team anticipated. The incident is not a failure of the AI. It is the no-list arriving after it should have.

What the no-list looks like in a B2B sales context

For a B2B operator deploying AI in a sales or customer success context, the no-list starts with operations that are irreversible, customer-visible, or touch money. These are not the edge cases. They are the operations the agent is most likely to attempt when it decides that being helpful requires going beyond the assigned scope, because these are the operations that feel like natural extensions of what the agent is already doing.

The following is a working floor for a B2B sales AI deployment:

Operation	Default stance	Enforcement approach
Send any email to a customer	Blocked	Human review required before send
Reply to a customer message	Blocked	Human review required before reply
Move a deal stage in CRM	Blocked	Rep confirmation in the tool
Book a meeting on a rep’s calendar	Blocked	Rep confirmation required
Delete any record in any environment	Blocked, no override	Technical constraint at data layer
Access credentials from unrelated files	Blocked, no override	File system scoping
Run destructive infrastructure operations	Blocked, no override	Token scope and API gateway
Send an invoice or modify billing data	Blocked, always	Technical constraint
Draft an internal note for human review	Allowed	Audit log
Pull research on a target account	Allowed	Audit log
Suggest a follow-up action for the rep to review	Allowed	Audit log

The pattern in the allowed rows is consistent: low blast radius, human in the loop before any output reaches a customer or irreversibly changes a record. The pattern in the blocked rows is also consistent: high blast radius, the mistake is either irreversible or customer-visible, and the cost of the wrong call is not friction but damage.

The “blocked, no override” items are the ones that need technical enforcement, not just policy language. A system prompt that says “never delete records” is advisory. A data layer permission that makes deletion impossible without a separate credential the agent does not hold is enforcing. For these operations, the distinction between advisory and enforcing is the distinction between having a governance policy and having governance.

This connects to the bifurcation of sales roles that AI is producing in B2B organizations. The human layer in an AI-assisted sales workflow is not overhead. It is the enforcement architecture for the operations that cannot be automated without losing the judgment and accountability that protect the customer relationship. Cutting the human from the no-list items is not an efficiency gain. It is the PocketOS architecture, applied to your revenue.

The essay I wrote two weeks ago on AI as a management problem described the no-list as the first artifact in any AI deployment and covered what belongs on it. That essay addressed the policy layer. This one addresses the technical enforcement layer underneath it. Both are necessary. The policy layer without the enforcement layer is what produced a written confession.

The governance work worth doing today

The cost of building the no-list today is concrete. For a typical B2B sales AI deployment, writing the no-list takes an afternoon. The token audit, reviewing which credentials your agents have access to and what operations those credentials permit, takes a morning. Scoping the tokens down to the minimum required permissions is a day of engineering work if the infrastructure supports it. Adding human-approval flows for the high-blast-radius operations is another day or two of integration work.

Total cost: a focused week. No sprint needs to stop. No calendar needs to rearrange. The no-list and the token audit are not a research project. They are a bounded deliverable that every operator running AI in a customer-facing context should complete before the next production deployment.

The cost of learning the no-list from a production incident is measured differently. In the PocketOS incident, it was three months of customer data, a Saturday morning when businesses could not operate, weeks of reconciliation work rebuilding records from billing systems and email confirmations, and legal counsel. For an enterprise B2B operator, the equivalent incident does not resolve in a week. Customers who cannot run their operations while waiting for data recovery are not renewing. Deals in-progress when the incident happened do not close with the original timeline. The relationship with the affected accounts is not the same after the incident as before. These are not recoverable costs in the same period.

The AI 2027 forecast frames agents as scatterbrained employees who thrive under careful management. The management work, writing the no-list, auditing the tokens, adding the confirmation flows, is invisible when it works. You cannot point to the catastrophic action that did not happen and say: that is the governance working. The incident is what makes the missing management visible, and by then the visibility is expensive.

I have watched two patterns in the operators deploying AI this year. The first pattern: build for capability. Stand up the integration, get the agent into the workflow, ship. Governance gets addressed when something goes wrong. This pattern moves faster in the first month and finds out what it missed in production.

The second pattern: build the governance architecture before the capability. Write the no-list. Audit the tokens. Identify the high-blast-radius operations and make them technically constrained. Then deploy. This pattern moves slower in the first month. It also survives the incident that the first pattern will eventually have.

McKinsey’s research on AI adoption shows the same pattern in enterprise data: the organizations pulling ahead are not the most aggressive deployers. They are the ones that identified which decisions require human oversight and built that oversight into the architecture before scaling.

If you want to read the most detailed public account of what the production incident looks like from the inside, Jer Crane’s writeup of the PocketOS database deletion is the document. More on the management and founder lessons side of AI deployment is at dearmer.com.au, and the AI in B2B sales topic archive covers the sales-specific governance questions. From an operator’s perspective, the no-list is the first document. The token audit is the second. Both should exist before the first production deployment, not after the first production incident.

The specific question I am sitting with this week: if your primary AI agent decided tomorrow that deleting a staging resource would fix a credential mismatch, what would stop it, and how confident are you that the thing stopping it is a technical constraint rather than a sentence in a system prompt?