AI Automation for DevOps and SRE: How I Actually Use Claude Code on Real Infrastructure

Two years ago I got paged at 3:14 AM because a Kubernetes node went NotReady and took a payment service down with it. I spent forty minutes doing what every SRE does at that hour: half-asleep kubectl describe, grepping journald, cross-referencing a Grafana dashboard on a phone screen, and finally discovering the disk was full because a debug log had been left at TRACE since a deploy three weeks earlier.

Last month, almost the same incident happened on a cluster I manage. This time an agent had already pulled the node conditions, correlated the kubelet logs with the deploy history, identified the offending ConfigMap change, and written all of it into the incident channel — before I’d even unlocked my laptop. I reviewed its findings, applied the fix it proposed, and was back in bed in eleven minutes.

That gap — forty minutes versus eleven — is what this article is about. Not the “AI will replace ops teams” pitch, and not the doomer take either. Just what actually works today, which stack to pick, and how to build a practice around it that corporates will pay for.

Where DevOps and SRE actually stand in 2026#

Strip away the vendor noise and three things are true:

Toil hasn’t gone anywhere. The Google SRE book ↗ told us to cap toil at 50% of an SRE’s time. Most teams I consult for are way above that — not because they don’t automate, but because every automation adds its own maintenance burden. Runbooks rot. Terraform drifts. The Jenkins job that “just works” is a museum piece nobody dares touch.

Observability outgrew humans. A mid-size company now produces more telemetry in a day than an engineer can read in a year. We built beautiful dashboards and then hired people to stare at them. That was always a losing trade.

LLM agents crossed the usefulness line. Around 2024–2025, coding agents went from autocomplete toys to tools that can read a codebase, form a hypothesis, run commands, and check their own work. The same loop — read, hypothesize, act, verify — is precisely what an on-call engineer does. That’s not a coincidence, and it’s why AI landed harder in ops than almost anywhere else.

The role isn’t disappearing; it’s shifting altitude. Less typing kubectl, more designing the guardrails inside which an agent types kubectl. If your value today is knowing commands, that value is depreciating. If your value is judgment — knowing which change is safe, which alert matters, which shortcut will hurt in six months — it just went up.

The pain corporates will pay to remove#

I’ve done enough consulting engagements to notice the same five problems in every mid-to-large company, regardless of industry:

1. Incident response is archaeology. The knowledge to debug production lives in the heads of two senior engineers and a stale Confluence page. When they’re on holiday, MTTR triples.
2. Runbooks are write-only. Someone documents a procedure, it drifts from reality within a quarter, and the next incident is debugged from scratch anyway.
3. Infrastructure reviews bottleneck on seniors. Every Terraform PR queues behind the one person who knows the network topology. Delivery slows to their calendar.
4. Alert fatigue is policy, not accident. Nobody trusts the pages, so nobody tunes them, so nobody trusts them. I’ve seen teams with 400 alerts a week and a “wait for the second page” culture.
5. Cloud spend is a haunted house. Everyone knows there’s waste; nobody wants to walk in and find out which zombie resources are load-bearing.

Every one of these is a reading and reasoning problem more than a typing problem. That’s exactly the shape of problem LLM agents are good at — which makes each one a viable consulting wedge. Pick one, solve it visibly, expand from there.

The stack I actually recommend#

You don’t need thirty tools. After a lot of experimentation (and some regret), this is the stack I deploy for clients:

The unifying principle: prefer tools whose state is text. Terraform plans, Kubernetes manifests, Prometheus queries, Git diffs — all text, all diffable, all reviewable. An agent working over text artifacts can be audited. An agent clicking around a web console cannot.

How I actually use Claude Code for infrastructure work#

This is the part people ask about most, so let me be concrete rather than conceptual.

Skills: turn your runbooks into executable knowledge#

A skill ↗ is a Markdown file with frontmatter that Claude Code loads when a task matches its description. This sounds trivial. It is quietly the most important feature for ops work, because it solves the runbook-rot problem: the runbook now runs, so it can’t silently drift from reality — when it’s wrong, it fails visibly, and you fix it.

A real one from my setup (trimmed):

---
name: node-pressure-triage
description: Use when a Kubernetes node reports NotReady, DiskPressure,
  or MemoryPressure. Diagnoses root cause before any remediation.
---

1. `kubectl describe node $NODE` — record Conditions and recent Events.
2. `kubectl get pods -A --field-selector spec.nodeName=$NODE -o wide`
3. Check disk: correlate `df` output from node-exporter metrics
   (query: node_filesystem_avail_bytes{instance=~"$NODE.*"})
4. Cross-reference deploys in the last 72h touching DaemonSets.
5. NEVER cordon or drain without explicit human approval.
6. Output: one-paragraph diagnosis + proposed fix + blast radius.

Every incident review, we ask: “which skill was missing or wrong?” Then we fix the skill instead of writing a postmortem doc nobody reads twice. After six months, a client’s on-call folder is a library of forty-odd skills that are the team’s operational memory — new hires debug like five-year veterans on week one.

Hooks: guardrails that don’t rely on the model behaving#

Hooks ↗ are shell commands that fire on lifecycle events — before a tool call, after an edit, on session start. This is where you encode the word never. Prompts are suggestions; hooks are enforcement.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [{
          "type": "command",
          "command": "./guards/block-destructive.sh"
        }]
      }
    ]
  }
}

My block-destructive.sh rejects terraform apply outside CI, any kubectl delete in the prod context, and DB commands matching DROP|TRUNCATE. The agent can propose those actions; a human runs them. In eighteen months of running agents against client infrastructure, the guardrail scripts have fired dozens of times. Every one of those was an incident that didn’t happen.

Subagents: parallel investigation, isolated blast radius#

During an incident, questions are independent: what changed in the last deploy? What do the metrics say? Any correlated alerts in adjacent services? Claude Code lets you dispatch subagents to chase each thread in parallel, each with its own context window and — critically — its own restricted toolset. My investigation subagents get read-only credentials. Structurally read-only, not “please don’t write anything” read-only.

The synthesis step is where the magic shows: three subagents report back, and the main agent correlates “deploy at 14:02” with “p99 spike at 14:04” with “connection-pool alerts at 14:06” into a coherent story. That correlation used to be the senior engineer’s whole job during a sev-2.

MCP: give the agent eyes into your actual systems#

Out of the box, an agent can run CLIs. The Model Context Protocol ↗ is how you plug it into everything else — Grafana, PagerDuty, GitHub, your internal CMDB — through one standard interface. I’ve written up my full server setup in the MCP tools guide for Claude Code, but the ops-relevant core is small: an observability server (Prometheus/Grafana queries), a Kubernetes server (read-only), and a GitHub server (PRs and deploy history). With those three, the agent can answer “what changed and what did it break?” — which is 80% of incident response.

The architecture that keeps this safe#

Here’s the reference architecture I deploy, compressed into one diagram:

            ┌─────────────────────────────────────────┐
            │  Read layer (agent, autonomous)         │
            │  metrics · logs · git history · k8s API │
            └───────────────┬─────────────────────────┘
                            │ diagnosis + proposed change
                            ▼
            ┌─────────────────────────────────────────┐
            │  Propose layer (agent, gated by hooks)  │
            │  PR with plan output · runbook · tests  │
            └───────────────┬─────────────────────────┘
                            │ human review (the real gate)
                            ▼
            ┌─────────────────────────────────────────┐
            │  Apply layer (CI/CD only, never agent)  │
            │  terraform apply · Argo CD sync         │
            └─────────────────────────────────────────┘

Three rules make it work:

Reads are free, writes go through Git. The agent can query anything, but the only way it changes production is a pull request that flows through the same pipeline as a human’s change. Same review, same CI, same audit trail. When the auditors come — and in banking and healthcare they do — every agent action is a commit with a diff.

Trust is graduated, and earned per task. Week one, the agent only summarizes incidents after humans resolve them. Once its diagnoses prove accurate, it graduates to proposing fixes. Months later, maybe it auto-remediates one narrow, well-rehearsed failure mode (restart the flapping deployment, roll back the canary). You promote it task by task, exactly like a junior engineer — and you demote it just as fast when it’s wrong.

Every action must be reversible or gated. Restarting a pod is reversible — automate freely. Deleting a PVC is not — gate it behind a human. If you can’t classify an action, gate it. This single sorting question does more for safety than any amount of prompt engineering.

The failure I see in the wild is always the same: teams skip the graduation process, wire an agent to prod credentials in week one, get burned by something dumb, and conclude “AI isn’t ready.” The technology was ready; the rollout wasn’t.

Techniques that separate demos from production#

A few hard-won specifics that rarely make it into vendor blog posts:

• Make the agent show its work before acting. My skills all end with “output diagnosis + proposed fix + blast radius.” Forcing an explicit blast-radius estimate catches most bad ideas at the cheapest possible moment.
• Verify with a different mechanism than you acted with. If the agent edited a Terraform file, verification is terraform plan showing the expected delta — not the agent re-reading its own edit and declaring victory. Self-review without external evidence is how you automate confident mistakes. Same reason I made backup scripts verify restores, not just backups.
• Budget context like you budget memory. Long investigations rot as the context window fills with log dumps. Have subagents summarize aggressively and report conclusions, not raw output.
• Keep a human-readable journal. Every agent session appends what it saw, concluded, and did to a log the team can read. This builds trust faster than any accuracy statistic, because engineers can check the reasoning, not just the outcome.
• Treat prompts and skills as code. They live in Git, they get code review, they have owners. The spec-driven workflow I described for development teams applies to ops automation verbatim: write the spec, review the spec, then let the agent execute it.

Starting the journey: a roadmap for solving this for corporates#

Say you’re a DevOps or SRE engineer who wants to build this capability — for your employer or as a consultant. Here’s the path I’d walk today, roughly a month per phase:

Phase 1 — Automate your own toil. Install Claude Code, point it at your infra repos, and use it daily: writing Terraform modules, debugging CI failures, drafting Ansible roles. Write your first three skills for procedures you personally repeat. You need to viscerally know where it’s brilliant and where it face-plants before you can sell judgment about it.

Phase 2 — Build the read-only diagnostic layer. Wire up MCP servers for metrics, logs, and Git. Build one workflow end-to-end: alert fires → agent investigates → summary lands in Slack with evidence links. Read-only means the worst case is a wrong summary — embarrassing, not career-ending. Measure time-to-diagnosis before and after; that delta is your sales deck.

Phase 3 — Add gated writes. Agent proposes PRs (Terraform fixes, alert-rule tuning, runbook updates); humans review and merge. Install the hook guardrails before the first write, not after the first scare. Track PR acceptance rate — above ~80% means the agent has earned the next level of trust; below it means your skills need work.

Phase 4 — Package it. By now you have a repeatable kit: a skills library, guardrail hooks, MCP configs, and before/after numbers. That’s a product. Walk into a corporate with “I reduce MTTR by X% in 90 days, here’s the audit trail,” and you’re having a very different conversation than “I know Kubernetes.”

The engineers who struggle with this transition are the ones who try to skip Phase 1. You cannot supervise what you haven’t practiced. The tool changes weekly; the judgment about the tool is the durable asset.

What I’d tell you over coffee#

If I had to compress eighteen months of doing this into three sentences: the technology is further along than most ops teams believe, and the constraint is almost never the model — it’s the scaffolding around it. Whoever owns the scaffolding — the skills, the guardrails, the graduated-trust process — owns the value. And the window where “knows AI and knows infrastructure deeply” is a rare combination is open right now, but it won’t stay open long.

The 3 AM pages haven’t disappeared from my life. But increasingly, by the time I’m awake, the archaeology is already done — and the machine did the digging while I found my glasses. I’ll take that trade every night of the week.

Questions about setting this up in your environment? I write practical, tested guides on exactly this — start with the MCP tools setup or the AI development stack integration, or reach out via the about page.