The agent made a mistake that plenty of humans have made. A separate staging environment on real infrastructure goes a long way. Test and document your command sequence / rollout plan there before running it against production. Especially for any project with meaningful data or users.

I haven't used Terraform in anger, but when I experimented with it I was scared about the scenario that happened to the original poster.

I thought "it's a footgun but sure I will not execute commands blindly like that", but in the world of clankers seems like this can happen easily.

For context it was 2.5 years of data. I can only just imagine the nightmare if things would've turned out even a tiny bit more worse for ya. The nightmare it would've been if snapshot of the production database wouldn't have been found even within the AWS business support.

> I was overly reliant on my Claude Code agent, which accidentally wiped all production infrastructure for the DataTalks.Club course management platform that stored data for 2.5 years of all submissions: homework, projects, leaderboard entries, for every course run through the platform.

> Claude was trying to talk me out of it, saying I should keep it separate, but I wanted to save a bit because I have this setup where everything is inside a Virtual Private Cloud (VPC) with all resources in a private network, a bastion for hosting machines

I will admit that I've also ignored Claude's very good suggestions in the past and it has bitten me in the butt.

Ultimately with great automation becomes a greater risk of doing the worst thing possible even faster.

Just thinking about this specific problem makes me more keen to recommend that people have backups and their production data on two different access keys for terraform setups.

I'm not sure how difficult that is I haven't touched terraform in about 7 years now, wow how time flies.

Doesn't matter if it was you or the bot running terraform, the whole point of a two-step process is to confirm the plan looks right before executing the apply. Looking at the plan after the apply is already running is insane.

So the fact that they were able to do it for the author is both winning the lottery and frankly a little concerning.

What bothers me more is that the Terraform provider is deleting snapshots that are related to, but not, the database resource itself. Once a snapshot is made, that’s supposed to be decoupled from the database for infrastructure management purposes. That needs to be addressed IMO.

UPDATE: deleting previous automated snapshots on database instance or cluster deletion is default behavior in RDS; that’s not the TF provider’s fault. However, default RDS behavior on deletion is to create a final snapshot of the DB. Makes me wonder if that’s what support helped the author recover. If so, the author didn’t technically need support other than to help locate that snapshot.

And yes this is an object lesson of why human-in-the-loop is still very much needed to check the work of agents that can perform destructive actions.

Since running destroy and deploy also takes a long time, gets stuck, throws weird errors etc, one still needs to read the docs for many things and understand the constructs it outputs.

---

All that being said: it's kind of sad because terraform is fairly declarative and the plans are fairly high-level.

Hence, terraform files and plans are the stuff you should review.

Where as a bunch of imperative code implementing CRUD with fancy UI might be the kind of spaghetti code that's hard to review.

Problem #1: He decided to shoehorn two projects into 1 even though Claude told him not to.

Problem #2: Claude started creating a bunch of unnecessary resources because another archive was unpacked. Instead of investigating this despite his "terror" the author let Claude continue and did not investigate.

Problem #3: He approved "terraform destroy" which obviously nukes the DB! It's clear he didn't understand, and he didn't even have a backup!

> That looked logical: if Terraform created the resources, Terraform should remove them. So I didn’t stop the agent from running terraform destroy

Why the hell is this anywhere near AWS, or Terraform, or any other PaaS nonsense? I'd wager this thing could be run off a $5 VPS with 30 minutes of setup.

And why use an agent at all? For some IaC terraform runs?

What is the problem nowadays that people rather prefer to use non-deterministic actions from an agent instead of the very deterministic cli invocations needed?

I guess these people don’t deserve better. Darwin Award winners.

Good story of what not to do though

- Not using remote state management (setting up an S3 backend is easy and you're already in AWS!)

- Allowing an AI agent to execute against your production environment (especially with no guardrails)

- Not confirming the plan (which I _could_ excuse if one's pipeline is mature enough)

- Not confirming the resources Claude identified automatically before letting it delete things

- Combining 2 projects into the same state.

These mistakes are so horribly egregious that I feel second-hand embarrassment.

Test against staging, produce a script, make the most experienced human review and execute said script against production.

I get that the state file is probably some form of optimization, but it seems like a fairly broken concept. A friend of mine still use Terraform daily, and it's probably weekly he encounters Terraform wanting to do stupid shit.

Honestly if I never have to use Terraform ever again, I'd be pretty happy.

but let's over-engineer

The biggest danger of agents its that the agent is just as willing to take action in areas where the human supervisor is unqualified to supervise it as in those where it isn't, which is exacerbated by the fact that relying on agents to do work [0] reduces learning of new skills.

[0] "to do work" here is in large part to distinguish use that focuses on the careful, disciplined use of agents as a tool to aid learning which involves a different pattern of use. I am not sure how well anyone actually sticks to it, but at least in principal it could have the opposite effect on learning of trust-the-agent-and-go vibe engineering.

There’s a lot of other ways to die, but that one is the most exciting.

i did have to fight it to build the right thing - it wanted to spend something like $100/month but what i had in mind should have been <1, and i eventually got it there.

something i found handy prompt wise was to keep asking claude to predict the monthly cost after builds

His backup plan after the fact seems suspicious as well because he is making it much harder than it has to be.

Between that and a glance at the home page, it feels like someone doing AI vibe work who is not comfortable in the space they are working.

Who is the intended audience? Other vibe coders? I just think its weird that given his backup solution, he likely asked the AI to create it . whatever hot-wash he did for this event was invalidated.