AKS Fundamentals for IT Admins: Node Pools, CNI Networking, and Upgrade Strategy · CloudEngineerHub
Azure11 min read8 views
AKS Fundamentals for IT Admins: Node Pools, CNI Networking, and Upgrade Strategy
Run AKS like an IT admin: system, user, and spot node pools; kubenet vs Azure CNI Overlay vs Cilium; sequential upgrades with PDBs, max-surge, and maintenance windows.
C
CloudEngineerHub Team
Published: September 5, 2026· Last Updated: October 3, 2026·11 min read
AKS Fundamentals for IT Admins: Node Pools, CNI Networking, and Upgrade Strategy
Platform teams inherit AKS clusters the way they inherit file servers: someone else built it, and now it is yours. The parts of AKS that map to classic IT admin duties are node lifecycle, IP addressing, and upgrades — get these three right and the cluster stays boring.
Visual Walkthrough
Version skew rules and surge nodes make upgrades boring — on purpose.
Interactive FlowAKS Upgrade — Control Plane to Node Pools6/6
1
Plan skew compliance
Node pools must lag control plane by ≤1 minor version.
Networking plugin and cluster extensions align versions.
AKS Upgrade — Control Plane to Node Pools. 1. Plan skew compliance: Node pools must lag control plane by ≤1 minor version. — 2. Upgrade control plane: API server bumps to target minor with HA intact. — 3. Surge nodes spin up: max-surge creates extra capacity for drain dance. — 4. Cordon & drain pool: Pods evicted gracefully; PDBs prevent app outages. — 5. Validate workloads: Probes, HPA and app health confirmed post-swap. — 6. CNI & add-ons follow: Networking plugin and cluster extensions align versions.
C
CloudEngineerHub Team
Author
The CloudEngineerHub editorial team — Microsoft endpoint management and cloud engineers writing hands-on guides on Intune, Azure, Entra ID, Autopilot, Defender and PowerShell. Every article is reviewed against current Microsoft documentation.
Roles: Azure Kubernetes Service Cluster Admin Role (or Contributor) on the cluster resource group; cluster-admin inside Kubernetes for PDB and workload checks.
Tools: az CLI, Az PowerShell, and kubectl — plus a maintenance window agreed with workload owners before your first upgrade.
Quota headroom for max-surge extra nodes during upgrades; compute quota surprises are the most common upgrade-day failure.
A PDB review across workloads — the upgrade path drains nodes, and missing budgets make that eviction uncontrolled.
Container Insights (or your monitoring of choice) wired before the next upgrade so you can watch the cordon/drain live.
How AKS Works
AKS is a managed control plane plus your node pools. Microsoft operates the API server, scheduler, and etcd; you own node pools, networking, and everything running in pods. System pools run the cluster-critical pods (CoreDNS, metrics-server, tunnel components); user pools carry applications and scale independently; spot pools run evictable VMs for interruptible work. Networking is the second axis: pods either get their own private CIDR routed inside the cluster (kubenet/Overlay) or real VNet IPs (classic Azure CNI) — a decision that determines your subnet math forever. Upgrades are rolling by design: the control plane is upgraded first, then node pools replace nodes batch by batch with surge nodes absorbing load. Every failure mode in this article maps to one of those three responsibilities.
Node Pools: System, User, and Spot
Every cluster has node pools, and the discipline is separation. The system pool runs CoreDNS, metrics-server, and the tunnel components; keep it small and dedicated (three nodes spread across availability zones supports the 99.95% uptime SLA) and taint it so workloads cannot land there. User pools carry applications and scale independently. Spot pools (Spot priority VMs, evictable) are perfect for CI runners and batch jobs — and terrible for anything that cannot tolerate eviction.
Bash
1
2
3
4
5
6
7
az aks create -g rg-aks -n aks-prod --tier standard --zones123 \
--nodepool-name systempool --node-count3 \
--network-plugin azure --network-plugin-mode overlay --pod-cidr 10.244.0.0/16
az aks nodepool add -g rg-aks --cluster-name aks-prod -n userpool \
--mode User --node-count3 --zones123 --max-surge33%
az aks nodepool add -g rg-aks --cluster-name aks-prod -n spotpool \
--mode User --priority Spot --eviction-policy Delete --spot-max-price -1 --node-count2
The --max-surge value matters twice: at scale-out and during upgrades, where it controls how many extra nodes spin up per batch (33% keeps cost sane on large pools; 1 on tiny ones).
CNI Networking: Pick Your Pod IP Model
Four networking modes, one decision: do pods need IPs from your VNet?
kubenet — pods get virtual IPs routed via user-defined routes; cheapest on IP consumption, adds a routing hop, no pods-as-VNet-addresses.
Azure CNI — every pod gets a real VNet IP. Required when external systems (firewalls, on-prem ACLs) must address pods directly, but it burns node-count × max-pods addresses and exhausts subnets fast.
Azure CNI Overlay — nodes take VNet IPs while pods use a private CIDR (defaults of 10.244.0.0/16 for pods and 172.16.0.0/16 for services). No UDR hop, no IP exhaustion, and the recommended default for new clusters.
Azure CNI powered by Cilium — Overlay plus an eBPF data plane for better throughput and network policy enforcement, selected with --network-dataplane cilium.
The one planning rule people skip: the pod and service CIDRs must not overlap your VNet or anything you will peer with. Fixing an overlap later means rebuilding the cluster. Keep NSGs on node subnets regardless of mode — they still apply to node traffic.
Upgrade Strategy Without Weekend Outages
AKS supports a rolling window of recent Kubernetes minors, and upgrades must be sequential — 1.30 to 1.31 to 1.32, never skipping. Upgrade the control plane first, then node pools, so a bad version stops before it touches workloads:
Bash
1
2
3
4
az aks get-upgrades -g rg-aks -n aks-prod -o table
az aks upgrade -g rg-aks -n aks-prod --kubernetes-version 1.31.4 --control-plane-only
az aks nodepool upgrade -g rg-aks --cluster-name aks-prod -n userpool \
--kubernetes-version 1.31.4 --max-surge33%
The two levers that make upgrades boring: PodDisruptionBudgets (without minAvailable or maxUnavailable, the cordon-and-drain will happily evict every replica of your database) and max-surge (extra nodes mean zero capacity loss during rolling replacement). Turn on an auto-upgrade channel (patch, stable, or rapid) plus managed maintenance windows so upgrades land in approved hours instead of cron jobs — and schedule node image upgrades on the same cadence for OS patches.
Configuration Deep Dive
System pool sizing: three DS3_v2-class nodes across zones is plenty for CoreDNS and friends on mid-size clusters; resist putting workloads there even though the scheduler will let you until the taint is applied.
Spot discipline: pair spot pools with tolerations and eviction handling in workloads; anything with persistent state or strict latency stays on standard pools.
Overlay + Cilium: the default posture for new clusters — Cilium enforces Kubernetes NetworkPolicy in the eBPF data plane instead of iptables, which is both faster and the foundation for FQDN policies.
Maintenance windows: define them per-pool; the plannedMaintenance profile keeps auto-upgrades and node-image upgrades inside approved hours.
Quota math: 33% surge on a 12-node pool needs 4 extra VMs of quota at upgrade time — check quota before, not during.
Real-World Enterprise Scenario
A payments platform inherited a 1.26-era cluster: one node pool, no PDBs, kubenet, upgrades done by hand at 2am. The remediation ran across three sprints. Sprint one: created a dedicated system pool (3 nodes, zones, tainted) and moved workloads to a new user pool on Azure CNI Overlay — pod CIDR chosen after mapping every peered network. Sprint two: added PDBs (minAvailable: 2) to every stateless service and a max-surge of 33% on both pools, then rehearsed the upgrade on a staging cluster end to end. Sprint three: enrolled the cluster in the stable auto-upgrade channel with a maintenance window of Sunday 02:00–06:00, plus scheduled node-image upgrades. The first production upgrade after that ran unattended: control plane, then pools, 47 minutes, zero dropped requests, monitored via Container Insights. The cultural change mattered as much as the config — upgrades went from "nobody touches it" to "the channel decides, we review the changelog."
PowerShell Automation
PowerShell
1
2
3
4
5
6
7
8
9
10
11
12
13
Connect-AzAccount
$aks = Get-AzAksCluster -ResourceGroupName rg-aks -Name aks-prod
$aks | Select-Object Name, KubernetesVersion, @{n='Tier';e={$_.Sku.Tier}}, @{n='Plugin';e={$_.NetworkProfile.NetworkPlugin}}
# expected: aks-prod, 1.31.4, Standard, azure (networkPluginMode Overlay when configured)# Add a zoned user pool and wire kubectl to the cluster
New-AzAksNodePool -ResourceGroupName rg-aks -ClusterName aks-prod -Name userpool `
-Mode User -NodeCount3 -Zone'1','2','3' -VmSetType VirtualMachineScaleSets
Import-AzAksCredential -ResourceGroupName rg-aks -Name aks-prod
# expected: Merged 'aks-prod' as the current context in ~/.kube/config
kubectl get nodes -o wide # 6 nodes: 3 system, 3 user, all Ready across zones
Troubleshooting AKS Operations
Symptom
Likely Cause
Fix
Pods stuck Pending in classic Azure CNI
Subnet IP exhaustion (nodes × max-pods)
Move to Overlay, or expand the subnet / lower max-pods
Upgrade stuck draining a node
PDB with minAvailable = replica count
Set realistic budgets (maxUnavailable: 1); the drain waits for them
Nodes NotReady after network change
NSG on node subnet blocking required ports
Re-apply the AKS-required NSG rules; verify with Network Watcher
Spot nodes evicted mid-job
Spot capacity reclaimed — expected behavior
Retry/tolerations in workload; keep stateful work off spot pools
Control plane upgraded, pools not
Control-plane-only upgrade completed
Run node pool upgrades per pool with matching version
Common Errors
InsufficientSubnetSize — classic Azure CNI ran out of pod IPs; the durable fix is Overlay, not another subnet carve-out.
Quota exceeded during upgrade — surge nodes need compute quota; request it before the window.
Upgrade blocked: unsupported version skip — minors upgrade sequentially; run 1.30→1.31 before 1.31→1.32.
PDB prevents all evictions — a PDB stricter than the replica count wedges the drain; correct the budget and re-run.
Visual Walkthrough
Trace one packet through peering, UDRs and the security appliance.
Interactive FlowHub-Spoke Routing — Follow the Packet6/6
1
Spoke workload sends
VM forwards to its default route (0.0.0.0/0 → NVA).
route
2
UDR steers traffic
Route table points inter-spoke and egress via hub appliance.
transit
3
Peering transits hub
VNet peering carries the packet into the hub VNet.
inspect
4
Firewall/NVA inspects
Rules allow, log or drop — this is your chokepoint of truth.
egress
5
Egress or on-prem
Via S2S VPN / ExpressRoute to the destination.
return
6
Return path symmetrical
Asymmetric routing is the classic failure — keep routes paired.
Hub-Spoke Routing — Follow the Packet. 1. Spoke workload sends: VM forwards to its default route (0.0.0.0/0 → NVA). — 2. UDR steers traffic: Route table points inter-spoke and egress via hub appliance. — 3. Peering transits hub: VNet peering carries the packet into the hub VNet. — 4. Firewall/NVA inspects: Rules allow, log or drop — this is your chokepoint of truth. — 5. Egress or on-prem: Via S2S VPN / ExpressRoute to the destination. — 6. Return path symmetrical: Asymmetric routing is the classic failure — keep routes paired.
Best Practices
Separate system, user, and spot pools; taint the system pool and give it three nodes across zones.
Default to Azure CNI Overlay; use classic Azure CNI only when pods must be VNet-addressable.
Pod/service CIDRs must not overlap peered networks — plan addressing before cluster creation.
Upgrades are sequential by minor version; stage control plane, then pools, guarded by PDBs and max-surge.
Auto-upgrade channels plus maintenance windows beat manual upgrade rituals.
Security Considerations
NSGs stay on node subnets in every CNI mode; Kubernetes NetworkPolicy (ideally Cilium) adds pod-level isolation.
Enable authorized IP ranges (or private clusters) so the API server is not reachable from the whole internet.
Keep node-image upgrades on the same cadence as Kubernetes upgrades — OS CVEs do not wait for minors.
System pool taints protect cluster-critical pods from noisy or compromised workloads; verify with kubectl describe node.
Key Takeaways
Separate system, user, and spot pools; taint the system pool and give it three nodes across zones.
Default to Azure CNI Overlay; use classic Azure CNI only when pods must be VNet-addressable.
Pod/service CIDRs must not overlap peered networks — plan addressing before cluster creation.
Upgrades are sequential by minor version; stage control plane, then pools, guarded by PDBs and max-surge.
Auto-upgrade channels plus maintenance windows beat manual upgrade rituals.
FAQ
Q: Which CNI mode should a new cluster use?
A: Azure CNI Overlay — pods get a private CIDR, nodes stay VNet-addressable, and you avoid the IP exhaustion that plagues classic CNI. Add Cilium when you want eBPF network policy enforcement.
Q: Can I change the pod CIDR after creation?
A: Not without rebuilding. Pod and service CIDRs are cluster-lifetime decisions; that is why the overlap check against peered networks happens before creation.
Q: Why upgrade the control plane separately from node pools?
A: Staging. Control-plane-only first means API compatibility is verified while workloads keep running on the old version; a bad result stops at the control plane instead of rolling through your fleet.
Q: What makes an upgrade "boring"?
A: PDBs that permit one unavailable replica, max-surge nodes absorbing load, and a maintenance window — capacity stays constant and evictions are bounded by design.
Q: How do spot pools fail?
A: Loudly but politely: eviction warnings then reclamation. Workloads need tolerations and eviction handling; anything stateful or latency-sensitive belongs on standard pools.
Conclusion
AKS admin work reduces to three disciplines: separate your pools, choose the pod IP model before creation, and make upgrades a scheduled, budget-guarded rollout. Do those and the cluster stays boring — which, for a platform you inherited, is the highest compliment available.
Deepen the skills with hands-on practice on /mock, reuse the cluster scripts in /tools, and grab the Kubernetes-on-Azure bundles in /store.
Comments
0No comments yet — be the first to share your thoughts.