If you run a Kubernetes cluster on Amazon Linux 2 and you tried to upgrade past EKS 1.32 this week, you already know how this story ends. Kubelet won’t start. Pods don’t schedule. Nodes go NotReady, drain themselves, and your Slack lights up. The reason is buried under three layers of release notes: Kubernetes 1.35 finally removed cgroup v1 support, and AL2 only does cgroup v1.
I’ve helped two teams climb out of this in the last two weeks. Both thought it was a config drift bug. Both ended up rebuilding their node groups from scratch. Here’s the actual playbook — the one that would have saved them a weekend.
What broke, and why nobody warned you cleanly
Kubernetes 1.35 ships with failCgroupV1: true as the new kubelet default (KEP-5573). That’s not a deprecation. It’s the kubelet refusing to start when it sees a v1 hierarchy at /sys/fs/cgroup. The flag still exists — you can flip it to false and limp along — but the cgroup v1 code paths inside the kubelet are gone in the next release. Everything you read on Reddit about “just override it” is a six-month rental, not a fix.
The bigger problem is timing. Amazon Linux 2 hits end of support on June 30, 2026, AWS stopped publishing EKS-optimized AL2 AMIs back on November 26, 2025, and EKS 1.32 was the last version that supported AL2 at all. If you’re running EKS today and you didn’t migrate node groups in the last six months, you are now firefighting on three fronts: the OS upgrade, the kubelet behavior change, and the containerd 1.x sunset that’s chasing you down the same hallway.
Containerd 2.0 is the supported branch going forward. The 1.x series shipped its last minor in late 2025 and only gets critical CVE fixes through this summer. If you’re on EKS AL2, you’re almost certainly on containerd 1.7. Moving to AL2023 gets you containerd 2.x at the same time, which sounds like a freebie until the registry mirror config breaks.
Step one: figure out which nodes you actually have on v1
Before anyone touches a node group, do an honest inventory. The single command that tells you the truth on a node is:
stat -fc %T /sys/fs/cgroupIf the output is cgroup2fs, you’re on v2 and the kubelet will boot fine. If it’s tmpfs, you’re on v1 (or the awkward hybrid mode, which counts as v1 for kubelet’s purposes). Anything else means the node is misconfigured in a way I haven’t seen, and you should investigate before doing anything else.
For a cluster-wide pass, this DaemonSet works without installing anything:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cgroup-audit
namespace: kube-system
spec:
selector:
matchLabels: { app: cgroup-audit }
template:
metadata:
labels: { app: cgroup-audit }
spec:
hostPID: true
containers:
- name: probe
image: busybox:1.36
command: ["sh", "-c"]
args:
- |
MODE=$(stat -fc %T /host/sys/fs/cgroup)
echo "$(hostname) cgroup=$MODE kernel=$(uname -r)"
sleep 3600
volumeMounts:
- { name: cgroup, mountPath: /host/sys/fs/cgroup, readOnly: true }
volumes:
- name: cgroup
hostPath: { path: /sys/fs/cgroup }Then kubectl logs -l app=cgroup-audit --tail=1 --prefix and pipe it into sort. You’ll get one line per node with the mode and kernel version. Anything reporting tmpfs plus kernel 5.10 is an AL2 node and your migration target.
I know this looks paranoid. Trust me — last month I watched a platform team confidently declare “we’re all on AL2023” because that’s what the launch template said. Three node groups were still pulling AL2 because someone had pinned the AMI ID two years ago for “stability.” The audit DaemonSet caught it. Their dashboards didn’t.
EKS-specific: the only path that works in June 2026
There is no in-place AL2 → AL2023 upgrade. AWS doesn’t support it, the kernel changes (5.10 → 6.1) are too invasive, and nobody at re:Post will help you if you try. You replace the node group. End of story.
The pattern that actually works, in order:
- Create a new managed node group in the same cluster, AMI type
AL2023_x86_64_STANDARD(orBOTTLEROCKET_x86_64if you’re going that way — more on this in a second), same instance types, same subnets, same labels and taints minus the OS-specific ones. - Cordon the old node group:
kubectl cordon -l eks.amazonaws.com/nodegroup=<old-ng>. - Drain in waves of two or three nodes, watching PDBs. If you have a workload with
minAvailable: 100%, fix the PDB before you start, not during. I’ve seen 4-hour drain stalls because someone wrote a PDB that mathematically cannot tolerate any disruption. - Once the old node group is empty, delete it.
This whole dance is normally boring. What makes it dangerous right now is that AL2023 changes how memory accounting works under cgroup v2: page cache counts toward container memory. A workload that ran at 70% memory on AL2 can suddenly look like it’s at 95% on AL2023, even though nothing about the process changed. You can either bump memory limits by 20-25% to absorb it, or accept that OOM kills will pick up if you don’t.
This is the part the AWS migration docs hand-wave. Go look at any workload with a memory.limit_in_bytes set tightly. It will need re-tuning. Plan a one-week shakedown on a non-prod cluster before you touch production.
Bottlerocket is a good answer if you don’t want to think about kernel upgrades again. It’s cgroup v2 native, ships with containerd 2.x, and the read-only root filesystem cuts off a whole category of node-level vulnerabilities. The cost is operational difference: SSM-only access, immutable host config, and no Docker socket. If your debugging muscle memory leans on kubectl exec plus host shells, you’ll feel the friction for a quarter.
GKE and AKS: less drama, but still real work
GKE container-optimized OS (COS) has been on cgroup v2 by default since 1.26 for new node pools. If your cluster is older than that and you let node pools auto-upgrade without rolling, you might still have v1 nodes. The detection script above works the same way. The fix is the same recreate-and-drain pattern, except you can usually let GKE node auto-provisioning handle it if you’ve enabled it.
Ubuntu node pools on GKE are the gotcha. Older Ubuntu LTS images stayed on hybrid cgroups longer than they needed to. If you’re on Ubuntu COS-equivalents from 2023 or earlier, audit and recreate.
AKS is the cleanest of the three because Azure pushed cgroup v2 default earlier on all the AKS-native images. The only AKS scenario where I’ve seen v1 in 2026 is on user-supplied VM images for compliance-bound clusters. If that’s you, you already know what you signed up for.
The failCgroupV1 escape hatch — when it’s safe, when it’s a trap
You can set failCgroupV1: false in your kubelet config and the node will boot on a cgroup v1 hierarchy. The kubelet log will scream a deprecation warning. Your cluster will work. For about ten weeks.
I think this is the right call in exactly one situation: you’ve got a regulated workload on an OS where the cgroup v2 kernel option isn’t approved yet (this happens with some hardened RHEL builds awaiting recertification), and you have a board-level sign-off on an exception. Otherwise, every day you spend on the override is a day you’re not building the muscle to actually migrate.
What you lose by staying on v1, even with the override:
- MemoryQoS doesn’t work. Burstable pods will misbehave under memory pressure.
- PSI (Pressure Stall Information) eviction signals are unavailable. Your node-pressure handling falls back to coarse heuristics.
- In-place pod resize assumes v2 semantics. The feature gate is on by default in 1.35, and it will silently misbehave.
- Swap on Linux support depends on v2 cgroup controllers. If you turned on swap to save money on RAM-heavy workloads, you’re stuck.
The override is a planning buffer, not a destination. Set a calendar reminder for the day you turn it off, and treat that date as immovable.
Containerd 1.x → 2.0: the registry mirror change that bites everyone
If your migration also moves you to containerd 2.x, there’s a quiet config breakage that has burned every team I’ve watched go through this. The registry mirror syntax in /etc/containerd/config.toml changed.
The old syntax (containerd 1.x):
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]The new syntax (containerd 2.x) splits config into hosts.toml files under /etc/containerd/certs.d/<host>/hosts.toml:
server = "https://registry-1.docker.io"
[host."https://my-mirror.internal"]
capabilities = ["pull", "resolve"]The old block is silently ignored. Your pulls don’t fail — they just bypass your mirror and go straight to the upstream registry. You don’t notice until your egress bill or your rate-limited Docker Hub account does. Audit your image pull telemetry after the migration. If pull counts to public registries spike, the mirror config didn’t carry over.
What 1.36 will break next, so you don’t do this twice
The reason to move now, even if your AL2 nodes happen to be running fine under an override, is that 1.36 is locking down further. Based on the sig-release discussion and the KEPs in flight:
- Several feature gates that are GA in 1.35 will be locked in 1.36, meaning you can’t disable them anymore. If you’ve been carrying any gate overrides as workarounds, audit them before the upgrade.
- KYAML becomes the default output for
kubectl get -o yaml. Most tooling handles it, but homegrown YAML diff tools that parse output may need adjustment. - Structured authentication config replaces the legacy webhook config for the apiserver. The migration is mechanical but easy to miss.
None of these are the kind of break that costs you a weekend on its own. But stacking them on top of a forced cgroup migration is what makes 1.36 a bad surprise for clusters that punted on 1.35.
A validation checklist before you call it done
Once the new node group is live and the old one is gone, run through this before you close the ticket:
kubectl get nodes -o wideand confirm every node reports the new kernel (6.1+ for AL2023).- Pick a workload with a tight memory limit and stress it. Watch
container_memory_working_set_bytesvs the limit. If you’re at 90% steady-state, bump the limit before page-cache pressure does it for you. - Run
kubectl auth can-i create pods/exec --as=system:serviceaccount:default:default. Cgroup v2 in combination with the 1.35 security defaults tightens what privileged pods can do via the host cgroup hierarchy. Some debugging pods that used to work via host PID will need explicit RBAC. - Re-verify image pull secrets. The containerd 2.x config rewrite can drop credential helper paths that were configured outside the main config file.
- Test an in-place pod resize if you use it. It’s a v2-only feature that silently no-ops on v1, so this is the first cluster where it actually works.
One thing to try this afternoon
If you’ve read this far and you’re not sure where your fleet stands, just run the audit DaemonSet on one cluster. Five minutes of work tells you whether you’re in the clear or you’ve got a migration to plan. The teams that lost a weekend to this weren’t unlucky — they were the ones who assumed the dashboards had it covered.
The deadline isn’t moving. Better to know now.
Sources: