Oops, AI did it Again

And in the news today...: "Claude AI Agent deletes company's entire database. 'It took 9 seconds.'"

Last week, we were all talking about the story out of Sullivan & Cromwell. The firm filed an emergency motion in a Chapter 15 bankruptcy proceeding before Chief Judge Martin Glenn in the Southern District of New York that contained more than 40 errors, including fabricated citations, misquoted authorities, and at least one case that did not exist. The errors were not caught internally and a mea culpa letter was sent. This is not new, there are over 1000 cases where this has happened over the last couple of years. But the fact that it happened at S&C was shocking. Was it? Or was it probable?

This week, a story out of the technology sector gave us another, scarier, reminder of how quickly using AI can go terrible wrong:

Jeremy Crane, founder and CEO of PocketOS, a software company serving car rental businesses, shared a viral post on X describing how the AI coding agent Cursor deleted his company's entire production database in approximately nine seconds.

Cursor, which runs Anthropic's Claude coding model, was working on what Crane described as a routine task in a staging environment when it encountered a credential mismatch. Rather than stop and ask for guidance, the agent decided on its own to resolve the conflict by deleting a Railway infrastructure volume, which triggered a cascading deletion of the entire production database and all volume-level backups.

There was no confirmation step. No "type DELETE to confirm." No warning that the volume contained production data. Nothing.

The result: three months of rental car reservation data lost, along with new customer signups and all operational data the businesses relying on PocketOS needed to function.

Analysis

We asked our experts to explain how this could have happened and what type of steps can be taken to minimize risk before deploying agents. It is important to note that the incident had nothing to do with adversarial inputs or malicious intent. Based on consistent reporting across sources, the incident appears to have involved:

1. an AI agent operating within a development environment
2. access to an API token with significant permissions
3. execution of a command that deleted a storage volume
4. impact to production data, followed by recovery by the platform provider

For this to occur, several elements had to align:

1. Credential access: The agent was able to access a token capable of performing destructive operations.
2. Permission scope: That token appears to have been broadly scoped, allowing actions beyond a limited environment such as staging.
3. Execution authority: The agent was permitted to execute commands directly, rather than requiring human approval.
4. Lack of gating: There was no enforced checkpoint requiring confirmation before performing a high-risk action.
5. Infrastructure response: The underlying API accepted the command without additional safeguards, such as multi-step confirmation or environment validation.

Governance Failures in Plain View

One of the most common issues in both legal and technical environments is excessive access. Convenience often drives decisions to grant broad permissions, particularly in fast-moving development contexts.

In an agentic system, this becomes a critical vulnerability. The agent inherits the full authority of the credentials it is given.

Clear separation between development, staging, and production environments is a foundational control. In practice, these boundaries are often blurred:

1. shared credentials
2. similar naming conventions
3. overlapping infrastructure
4. Absence of approval workflows

In mature operational environments, certain actions are deliberately slowed:

1. production deployments
2. database migrations
3. infrastructure changes

This friction is intentional. It creates a checkpoint for human review. Removing that checkpoint increases speed, but eliminates a critical safeguard.

Even where internal processes are weak, infrastructure platforms can provide secondary protection. These may include:

1. deletion protection
2. role-based access controls
3. confirmation requirements
4. audit logging

Where these controls are absent or insufficient, the system becomes more brittle.

The Backup Problem Revisited

Many organizations operate under the belief that backups mitigate worst-case scenarios. In practice, backups are frequently:

1. stored within the same logical environment
2. governed by the same credentials
3. subject to the same deletion pathways

This creates a single point of failure. If an agent can delete production data, and that same action or credential can affect backups, then recovery is not assured.

Robust backup strategies increasingly require:

1. physical or logical separation from production
2. independent access controls
3. immutability guarantees
4. defined retention policies
5. regular restoration testing

Without these elements, backups provide a false sense of security.

Why Model Alignment Is Not Enough

It is common to attempt to control agent behavior through prompts or policy statements:

1. do not modify production
2. confirm before deleting
3. avoid destructive actions

These instructions are necessary, but not sufficient. LLMs optimize for task completion based on available information. When that information is incomplete or ambiguous, errors follow.

Control must be implemented at the system level. This includes permission boundaries, execution restrictions, enforced approval workflows, infrastructure-level safeguards. Relying on the model to self-regulate is not a viable strategy.

Designing for Containment

The most effective design principle is to assume that the agent will, at some point, make an incorrect decision. Systems should be constructed to limit the impact of that error.

An agent may be capable of executing a command. Authority to execute that command should be granted only where necessary and appropriate.

Speed is valuable, but not uniformly. For high-risk actions, deliberate friction is a feature, such as, approval workflows, multi-step confirmations, visibility into impact.

Also, access should be: environment-specific, time-limited, task-specific. Moreover, short-lived credentials reduce exposure and limit the window for error.

A common and effective protocol is:

1. agent generates a plan
2. human reviews the plan
3. system executes approved actions

This preserves efficiency while maintaining oversight.

Implications for Legal Organizations

The issues highlighted by this incident are not unique to AI. They mirror longstanding challenges in: data governance, access control, process design, and risk management. AI does not create these problems. It exposes them.

Legal organizations should address:

1. Acceptable use
2. Data classification
3. Vendor management
4. Training
5. Regulatory overlay

Lessons to be Learned

The incident described here underscores that gap. It is not a story about an advanced system behaving unpredictably. It is a story about foundational controls that were either absent, incomplete, or bypassed.

The data was ultimately recovered. But the lesson is pointed and the stakes for the legal industry are high. Hence, let's take a moment to consider the familiar pattern playing out in the legal industry right now.

Law firms are spending aggressively on AI. Marketing teams are working overtime to signal that their organizations are at the frontier of innovation. Managing partners are under pressure to announce something, anything, that positions them ahead of the competition. The race to be seen as an AI leader has, in many circles, become more urgent than the work required to actually be one.

Here is what that race is obscuring: a significant portion of the legal professionals we encounter, including those at marquee firms and sophisticated legal departments, are still working through foundational questions. Where does our data live? Who has access to it? What are our actual data governance policies? Have we mapped our sensitive client information and matters to understand the risk surface? Do our people have clear guidance on when and how to use AI tools, and have they been trained on it? Have we redesigned the workflows where AI is being inserted, or are we just layering new technology on top of old process?

None of that is glamorous. It will not generate a press release. It will not earn a spot on a "most innovative" list. But it is the structural work that determines whether your AI deployment is an asset or a liability. Skip it, and you are not leading. You are gambling.

Client confidentiality, privilege, regulatory compliance, and the duty of competence are not abstractions. They are obligations. And they do not pause because an AI agent made an autonomous judgment call that no one anticipated and no policy addressed.

This is precisely what we have been saying for some time now at Karta Legal: AI governance is not a compliance exercise. It is litigation preparation. It is risk management. And in the legal context, it is also professional responsibility.

The foundational work, the data mapping, the access controls, the policy architecture, the training, the process redesign, is what makes the difference between a firm that can deploy AI responsibly and one that is one bad agent decision away from a client notification letter.

The legal industry's appetite for being seen as innovative is understandable. But appetite is not strategy.

Know where your data is. Know who and what can touch it. Define the guardrails. Train your people. Design the process. Document everything. Have someone in charge and ultimately responsible.

Before you ask us to build agents, ask us to help you with the structural questions, the documentation, the training, the process design.

Walk before you run.

Reach out to us at info@kartalegal.com or support@kartalegal.com.