When Karpenter Quietly Deleted Our Online Presence

Let me tell you about the night I got paged for a SEV-1 outage that I had absolutely no idea how to fix, ended up recovering in 20 minutes by manually clicking around the AWS Console, and then stayed up until 3 AM installing the thing that should have prevented it in the first place.

This happened at a firm, where I had recently moved laterally to a Senior DevOps role. It was my on-call week on the platform team. The incident was a total outage of all internet assets - 100% of incoming requests returning Cloudflare Error 522 - “Connection Timed Out” - and it lasted 48 minutes, from 23:02 to 23:50 CEST.

The Setup We Inherited

To understand why this happened, you need a bit of context on the edge tier we were running at the time.

The firm was in the middle of in-housing its DevOps from a contractual agency based in Turkey HQd in London. As part of that transition, we inherited a cluster that the agency had set up, and - to put it gently - it was not bespoke.

The edge looked like this: 6 large nginx reverse-proxy pods (8 GiB RAM, 4 vCPU each) running in our EKS cluster in eu-central-1. These nginx pods were sitting in front of the frontend, terminating requests and forwarding them downstream. So far, so normal.

The weird part was how traffic actually reached those pods. There was an AWS Network Load Balancer (NLB) in front of them, and the nginx pod IPs were manually registered into the NLB’s target group as IP targets. Not via a Kubernetes Service of type LoadBalancer, not via any controller - just someone, at some point, typing IP addresses into the AWS Console and clicking “register”.

That meant there was no automated reconciliation between the Kubernetes side (where pods get rescheduled to new nodes with new IPs all the time) and the AWS side (where the target group still pointed at the old IPs). The two worlds were completely disconnected.

The piece that should have connected them - the AWS Load Balancer Controller (LBC) - had simply never been installed for this tier. I didn’t know that yet. I would find out at 2 AM.

The Cost Optimization That Started The Decay

A few weeks before the incident, we went on a cost-optimization drive. Cluster spend was high, nodes were underutilized, and the obvious levers were Karpenter and HPAs.

We deployed Karpenter with consolidationPolicy: WhenUnderutilized, and we put HPAs in front of the relevant workloads. The idea was simple: Karpenter watches for underutilized nodes, evicts the pods on them, reschedules those pods onto other nodes, and removes the empty ones. Sounds great on paper. It is great on paper. It is also great in practice - when the rest of your stack is wired up correctly.

Here’s what actually happened over the following two weeks.

Karpenter did its job. It found underutilized nodes, it consolidated them, and as part of that, it evicted nginx pods and rescheduled them onto new nodes. Every time a pod gets rescheduled in EKS with the AWS VPC CNI, it gets a new pod IP from the node’s subnet.

So the nginx pod that used to live at 10.0.8.12 and was registered in the NLB target group - it got evicted, rescheduled, and came back at, say, 10.0.18.45. The pod was healthy. The pod was ready to serve traffic. The pod was passing its probes. But the NLB target group still had 10.0.8.12 registered, and that IP didn’t belong to anything anymore.

The AWS VPC CNI had already released it.

So the NLB kept sending traffic at a ghost IP, the health check kept failing, and the target slowly flipped from healthy to unhealthy.

One by one. Over two weeks.

6 healthy → 5 → 4 → 3 → 2 → 1.

And here’s the part that really stings: no alert fired. We were not ingesting the NLB HealthyHostCount metric into or NewRelic stack. Nobody was looking at it. The decay was completely invisible to us. We had a two-week linear countdown to zero, in plain sight, and we missed every single step of it.

23:00 CEST - The Last Target Drops

At 23:00:14 CEST, Karpenter made its final consolidation move. It consolidated node ip-10-0-8-100.eu-central-1.compute.internal, which was hosting the nginx pod tied to the last live registered target in the NLB group.

The eviction log looked like this:

23:00:14 CEST - karpenter-6d5f4c8b9-q8z12 - NodeClaimConsolidated: Triggered consolidation for node ip-10-0-8-100.eu-central-1.compute.internal
23:00:20 CEST - kubelet - Stopping container nginx-edge-6

By 23:01:30 CEST, the final target had failed its health check. HealthyHostCount hit 0. The NLB now had nowhere to send traffic.

Cloudflare, which sits in front of the NLB, tried to open a TCP connection to the origin, got nothing back, and started surfacing Error 522 - “Connection Timed Out” for 100% of incoming requests to all our pages. From the outside world, the retail and internal admin facade were just gone.

Internally, though, everything looked fine. The nginx pods were 1/1 Running. The frontend pods were 1/1 Running. Probes were passing. CPU was under 5%. There was nothing in the application logs. The failure was strictly at the NLB target registration layer - a layer the application had no visibility into.

23:02 CEST - Diego Gets Paged

At 23:02 CEST, a New Relic synthetic browser check tried to load all our pages and failed. PagerDuty did its job and paged the frontend on-call.

That was Diego.

Frontend was the first paging target whenever all our pages was down - that’s just how the routing was set up, since the URL was “owned” by the frontend team. So Diego got the page, opened his laptop, and started doing what any reasonable frontend engineer would do: he looked at his own stuff first.

He walked through the usual frontend-side hypotheses. Recent feature-flag changes. Recent frontend deploys. Code diffs from the last few hours. Anything that could explain why the login page was suddenly returning a Cloudflare error.

Everything looked clean.

He spent about 30 minutes on this - 23:02 to 23:32 CEST - before concluding that this wasn’t a frontend problem and escalating to the platform on-call.

That was me.

23:32 CEST - I Join, And I’m Clueless

It was my on-call week on the platform team, so the escalation landed on me. I joined the call at 23:32 CEST.

And I want to be honest here: when I first joined, I had no idea what was going on.

I did what you’re supposed to do. I walked through the recent code changes. I checked ArgoCD sync status - everything was in sync, no drift, no failed syncs. I pulled the nginx pod logs - nothing weird, no errors, no restarts. The pods were 1/1 Running, probes passing, CPU basically idle.

From where I was sitting, everything looked perfect. The application was healthy. The cluster was healthy. ArgoCD was happy. And yet, the login page was returning a Cloudflare error to 100% of the world.

I was clueless. And that’s a weird place to be during a SEV-1, because you can’t fix something you can’t even locate.

~23:40 CEST - The 522 Saves Me

The breakthrough came from the error message itself.

Up until that point I had been thinking about this as an application problem - something in the code, something in the config, something in the cluster. But the error users were seeing was Cloudflare Error 522 - “Connection Timed Out”.

And 522 has a very specific meaning. It doesn’t mean the application returned an error. It doesn’t mean the application was slow. It means Cloudflare could not establish a TCP connection to the origin at all. The origin was unreachable.

That reframed the whole thing. The failure wasn’t in the app. The failure was somewhere between Cloudflare and the application - which, in our setup, meant the AWS layer. The NLB. The target group.

So I pivoted. I stopped poking at the cluster and opened the NLB page in the AWS Console.

And there it was. The target group was fully unhealthy. Every single target was sitting in unhealthy state, with the reason Target.FailedHealthChecks. Six targets, all dead, all pointing at IPs that didn’t belong to any pod anymore.

I had found it.

~23:42 CEST - The Manual Fix

Now I had a hypothesis: the targets were stale, the pods had moved, and the NLB had nowhere to send traffic. The fix, in theory, was simple - register the live nginx pod IPs into the target group.

So I grabbed one live nginx pod IP from the cluster and manually registered it into the NLB target group through the AWS Console.

It went initial → healthy within a few seconds.

I refreshed all our pages.

It loaded.

That was the moment the outage ended, technically, though I didn’t declare recovery yet, because one healthy target out of six is not what I’d call “restored”. But the page loaded, and that was enough proof that the hypothesis was right.

23:42 – 23:50 CEST - Registering The Rest

Over the next eight minutes, I manually registered the remaining five nginx pod IPs.

All six went healthy. Total time from me joining the incident to full recovery was about 20 minutes.

At 23:50 CEST, I declared recovery. all our pages were fully functional again, with six healthy targets behind the NLB.

I was exhausted. But I also couldn’t sleep. There’s a specific kind of wired that hits you after a SEV-1 - your body is tired, your brain is not, and you can’t shut off the question of why.

23:50 CEST – ~02:50 CEST - The Three Hours After

I couldn’t sleep, so I went online and started reading about the correct way to wire an NLB to a Kubernetes cluster. I knew the manual fix I had just done was a hack - clicking IPs into a Console at midnight is not a strategy, it’s a panic response - and I wanted to understand what the proper setup was supposed to look like.

That’s when I learned about the AWS Load Balancer Controller (LBC).

The LBC is the component that reconciles target group membership with pod lifecycle. When a pod gets rescheduled to a new IP, the LBC is the thing that’s supposed to notice and update the target group automatically. It does this by watching a Kubernetes CRD called TargetGroupBinding, which wires an existing AWS target group to a Kubernetes Service. As pods come and go, the LBC adds and removes their IPs from the target group on its own.

And then it clicked: the manual fix I had just performed at 23:42 was exactly the reconciliation the LBC is supposed to do on its own. That’s why my fix worked - I was doing, by hand, the job the controller was meant to be doing all along. And that’s also why the outage had happened in the first place - the controller wasn’t there, so nobody was doing that job, and the target group had been quietly rotting for two weeks.

I figured this would be a quick install. It was not.

What I thought was “install one Helm chart” turned into three hours of work, because the LBC needs a bunch of prerequisites before it can actually do anything useful. Here’s what I had to do, step by step.

1. Create the IAM policy for the LBC

The LBC needs permission to manage AWS load balancer resources - NLBs, ALBs, target groups, listeners, rules. So I had to create an IAM policy granting it those permissions. AWS publishes a recommended policy document for this; I grabbed the up-to-date one and created it in our account.

2. Set up IRSA (IAM Roles for Service Accounts)

The cluster was on EKS, so the correct auth path was IRSA - IAM Roles for Service Accounts. The idea is that you create an IAM role, attach the policy from step 1 to it, and then annotate the LBC’s Kubernetes ServiceAccount so that pods using that ServiceAccount can assume the IAM role via OIDC. This is way better than handing the controller static AWS access keys, which is the old-school way and a security smell.

So I had to:

• Make sure the cluster’s OIDC issuer URL was set up (it was).
• Create the IAM role with the right trust policy scoped to the LBC’s ServiceAccount.
• Attach the LBC IAM policy to that role.

3. Install the LBC Helm chart

With the IAM side sorted, I installed the LBC itself via its Helm chart - aws-load-balancer-controller - into the kube-system namespace, configured with the right cluster name, region, and the ServiceAccount annotated with the IRSA role ARN.

4. Create the TargetGroupBinding object

This is the CRD that actually wires things together. I created a TargetGroupBinding object that linked the existing NLB target group (by its ARN) to the nginx Kubernetes Service. Once that object existed, the LBC started watching it and reconciling the target group membership against the live nginx pod IPs.

~02:30 CEST - The Experiment

Only once all four pieces were in place could I run the experiment I had actually set out to run at the start of this whole detour: does the LBC actually reconcile on its own?

It was already ~02:30 CEST, also a very low-traffic moment for all our pages, so briefly dropping one healthy target would carry almost no user-visible risk. This was a production experiment, and I wanted the blast radius to be as close to zero as possible if I was wrong.

I deleted one of the nginx pods.

Watched it getting recreated.

And I watched the LBC logs.

Within seconds, the LBC noticed the target was unhealthy, looked up the live nginx pod IP, and registered the target back again automatically.

That auto-reconciliation was an immense revelation. It was the proof I needed that the controller was doing its job, and that the manual fix I had done at 23:42 was exactly what the LBC does on its own every time a pod moves. The thing that had been missing forever - the thing that had caused the outage - was now in place and working.

My nerves finally calmed down.

By ~02:50 CEST, three hours after I started the install, I decided to call it a day.

The Next Morning - Confirming The Root Cause

With the LBC already installed and proven overnight, the next morning I sat down and confirmed the full root cause properly. I had been too tired to dig into the metrics the night before; now, with daylight and a working controller behind me, I went and looked.

CloudWatch told the story

I opened CloudWatch metrics for the NLB and watched HealthyHostCount decay from 6 to 0 over the previous two weeks. You could see it step down, target by target - 6, then 5, then 4, then 3, then 2, then 1, then 0. Each step corresponded to a Karpenter consolidation event that had rescheduled an nginx pod to a new IP, leaving the old IP stale in the target group.

It was a perfect linear decay, in plain sight, on a metric we had never wired into our dashboards.

Karpenter events confirmed it

I cross-referenced Karpenter events over the same window and saw it repeatedly changing the nodes hosting the nginx pods. Every reschedule left the manual NLB target group pointing at stale IPs that the AWS VPC CNI had already released. The pattern was exactly what I had hypothesized at 23:40 the night before - but now I had the data to back it up.

The missing piece was exactly what I’d found at 2 AM

The piece that should have reconciled those targets automatically - the AWS Load Balancer Controller - had never been installed for this tier. The manual target registration I had done at ~23:42 was exactly the reconciliation the LBC performs automatically on pod reschedule.

Root cause confirmed. Three things had lined up to produce the outage:

1. Unmanaged edge routing. The agency-built edge used an AWS NLB pointing directly at static pod IPs, with no controller and no Service of type LoadBalancer to reconcile them. The LBC had simply never been installed for this tier.
2. Karpenter consolidation vs. static targets. Karpenter kept rescheduling nginx pods to new nodes with new IPs, and the manual NLB target group kept pointing at the old, released IPs.
3. Observability gap. NLB HealthyHostCount was not ingested into our NewRelic stack, so a two-week linear decay from 6 hosts to 1 went completely unnoticed until the final host dropped to 0.

The Long-Term Fix - Retiring The nginx Tier

With the root cause understood and the LBC in place, I went to the team that afternoon and made a case: rather than just bolting the LBC onto the legacy edge (NLB + 6x nginx pods), we should retire that architecture entirely.

The nginx tier was an artifact of how the agency had originally set things up. It was a reverse-proxy layer sitting between the ALB/NLB and the application pods, doing termination and forwarding. With the LBC installed, we didn’t need it anymore - the load balancer could talk directly to the application pods via their AWS VPC CNI pod IPs.

The team agreed, and we migrated to an ALB in IP mode. Here’s what that looked like.

1. ALB in IP mode

We replaced the NLB with an AWS Application Load Balancer (ALB) configured with alb.ingress.kubernetes.io/target-type: ip. In IP mode, the ALB targets application pods directly via their AWS VPC CNI pod IPs - no nginx termination tier in the middle, zero utility of kube-proxy, eliminating the extra network hops.

2. LBC-managed lifecycle

With the LBC installed, pod reschedules are now reconciled natively. When a pod moves to a new IP, the LBC updates the target group in under 5 seconds. The exact failure mode that took down all our pages for 48 minutes is now structurally impossible - there is no manual target group to rot, because nothing is manual anymore.

3. Decommissioning the nginx tier

We removed the six nginx reverse-proxy pods. Each was 8 GiB RAM and 4 vCPU, so removing all six freed up 48 GiB of memory and 24 vCPUs in cluster request overhead. That’s a meaningful chunk of capacity on an EKS cluster, and it also removed an entire network hop between the load balancer and the application.

4. Alerts so this never happens again

We added two alerts on the SRE side:

• HealthyHostCount < 2 - page immediately, regardless of tier.
• A 24-hour host-decay alert - if healthy host count is trending down over a day, page, even if it hasn’t hit zero yet.

What I Took Away From This

A few things stuck with me after this incident, and I think they’re worth saying out loud.

The error message is a clue, not just a symptom. I spent the first eight minutes of my involvement looking at the application, because that’s where my instincts pointed. The moment I actually read what the 522 meant - “Cloudflare could not establish a TCP connection to the origin” - the whole investigation pivoted. The error was telling me where to look. I just wasn’t listening to it yet.

Manual is not a configuration. It’s a countdown. The agency had set up the NLB target group by hand, and it had worked fine for as long as nothing moved.

And this setup I found over time is actually very prevalent. Large nginx pods or 3 large nodes. Target IPs manually configured.

Can’t really blame as we were the ones who introduced Karpenter breaking their assumptions. The manual setup became a ticking clock. We just didn’t know the clock was running.

If a metric exists in AWS and you’re not ingesting it, you’re flying blind. HealthyHostCount is a first-class NLB metric. The NLB, and the target group were all screaming at us for two weeks. We just had no pipeline listening. The observability gap was the real root cause - the unmanaged target group was the trigger, but the reason it ran for two weeks undetected was that we weren’t watching the one metric that would have told the story.

The fix you do at 2 AM is sometimes the most educational part of the incident. The 20-minute recovery was the urgent part. But the three hours I spent installing the LBC, setting up IRSA, creating the TargetGroupBinding, and running the experiment at 02:30 - that’s the part that actually prevented the next outage. The manual fix closed the incident. The controller install closed the root cause.

And finally - call it a day when you’ve proven the fix. I ran the experiment at 02:30 because I needed to see the LBC auto-reconcile with my own eyes before I could sleep. Once I saw it, I stopped. I didn’t keep refactoring, I didn’t keep optimizing, I didn’t start the ALB migration that night. I went to bed. The morning was for confirming the root cause, the afternoon was for the team conversation. Pacing matters at 2 AM.

Comments/feedback?

Leave your comments/feedback below. Have you ever been bitten by a manual AWS setup that quietly rotted under a Kubernetes cluster? What did you do to fix it? And did you fix the cause, or just the symptom?