They ran on virtual machines managed by configuration scripts, and a deploy took hours because a deploy meant touching machines. I owned the move for my services — building the infrastructure definitions and the deployment pipelines, with verification built into them rather than added afterwards.

Deploys went from hours to under three minutes. Rollbacks to under two.

I also built production clusters from nothing on three different clouds, including one region delivered end to end. Building a region from an empty account showed me how much of our setup was actually deliberate and how much had just been copied forward.

Releases ran across multiple clusters on different days, and active-active doubled the surface. Verification was manual — an engineer triggering jobs in sequence and reading each result by hand — and end to end a release took about seven hours, most of it verification and waiting.

I rebuilt it on our internal workflow engine, starting on the canary and dogfood clusters, replacing each manual step with a platform component or building what didn't exist yet.

Verification was the part I underestimated. Across that many clusters the cost compounds, and a slip there causes real problems — the step meant to catch mistakes was the one most prone to them at that scale. The wall was that it needed shell scripts running against pods, and the workflow engine couldn't do that. So I went into the pipeline runtime underneath it and extended it until it could.

The script checks process health, logs, metrics, ports and release versions, then parses the output: fail on what matters, flag what needs a look, pass when it's clean. It tells you what's wrong, not just that something is.

Parallel deploys were argued about on reliability grounds. My position was that active-active was already in place and the infrastructure supported it — one side down doesn't affect the side still serving. A failed release now rolls back in parallel too, so recovery got faster as a side effect.

Fifty-two minutes now, deploying and verifying together, and verification runs against every pod rather than one sample per cluster. That caught edge cases we'd been missing.

Other services have picked up the pattern since, and the verification script gets adapted and asked about. I'd rather build the thing the next service can pick up than the thing that only solves my problem.

The edge layer is where every request arrives. I led its replatform across the production fleet on two clouds — phased cutovers, automated verification and failover validation at each stage, no customer impact, and edge infrastructure cost on the largest cluster cut in half.

The reason it worked is a tool I wrote. The DNS zone has over a hundred thousand records, and moving traffic weight across it by hand is the kind of task where one wrong entry is a very bad afternoon. So: a Python CLI that traces the resolution chain with live health-check data overlaid, changes weights in bulk behind a preview-confirm-apply flow, and writes an audit line for every change. It became the standard tooling for cutovers and got packaged for DevOps teams elsewhere in the company. I mostly wanted to be able to see what I was about to do before I did it.

I also ran the cache-layer transition from Redis to Valkey as a documented process that became the team standard, and retired the legacy edge stack — DNS, certificates and monitoring cleaned up behind it. That part took longer than the replacement did.

Active-active was a fleet-wide programme; I ran it for the chat backend and edge services, across fourteen of fifteen production clusters. Both regions live, traffic weighted between them, failover wired into the traffic-orchestration console. Recovery time is near zero because there's nothing to recover to — the other side is already carrying its share.

Most of that work wasn't architecture. It was VPC peering, route tables, DNS resolution conflicts and scheduler behaviour. None of it was difficult to understand and all of it took a long time.

Certificates came out of the same work. I automated the TLS lifecycle for the edge, validating rotations on a node running the full production configuration but taking no live traffic, before anything real touched it.

The company moved off Grafana onto an internal metrics platform; I took my services across and built out what they needed — hundreds of dashboards, alerting sorted into severity tiers and tied to runbooks, and service level objectives defined with the developers rather than handed to them.

I've been on call since my third month and have run incident response for the messaging estate through hundreds of P0 and P1 events, alongside the NOC, development and platform teams. Every one turned into a change somewhere — a metric, a threshold, a runbook.

Most of the work was taking alerts away.

Instance right-sizing, storage-mode tuning across the document store and cache layers, and Kubernetes resource refinement, applied across the services I own. Per-cluster reductions between 29 and 77 percent, with six-figure annual savings, and I wrote the roadmap that keeps it from drifting back.

None of it was possible before the monitoring was good enough to show what the services actually used.