Kubernetes Right-Sizing Is an Operating Model, Not a Setting
The AWS Containers blog recently published a detailed account of how Salesforce right-sizes Kubernetes workloads across its Amazon EKS fleet. It is worth reading if you run containers in production. The scale is extreme: more than 1,400 clusters and over $5 million in reported cumulative savings. But the useful lessons are not about scale. They are about how you change resource settings on live systems without breaking them.
The article’s opening problem will be familiar to anyone who has looked at a cloud bill: “Clusters running at low utilization consume compute that is paid for but never used.” Most Australian organisations we talk to are not running a thousand clusters. Many are running five to twenty, provisioned during a migration and never revisited. The waste is proportionally the same.
The real problem is guessed numbers
Every container in Kubernetes declares how much CPU and memory it requests. The scheduler reserves that capacity whether the workload uses it or not. In practice, those numbers are usually a guess made when the service was first deployed, padded for safety and then copied into every environment.
Nobody owns them afterwards. Product teams don’t want the risk of reducing them. Platform teams don’t know the workloads well enough to touch them. So requests drift upward, nodes fill with reserved but idle capacity, and the cluster autoscaler adds more nodes.
The Salesforce piece frames the fix as moving from static configuration to continuous tuning. In our view, the more important shift is giving resource requests an owner and a feedback loop. The Vertical Pod Autoscaler (VPA) is one way to automate that loop, but the loop has to exist first.
Matching the scaling approach to the workload
A point the article makes well is that not every workload should scale the same way. Some services get more capacity by adding replicas. Others, such as leader-follower designs or anything where every pod does the same work, gain nothing from extra replicas and need bigger pods instead. Some need both.
A simple way to think about the decision:
graph TD
A[Workload] --> B{Do more replicas help?}
B -->|No| C[VPA only]
B -->|Yes| D{Does per-pod size grow with load?}
D -->|No| E[HPA only]
D -->|Yes| F[VPA and HPA on separate signals]The critical rule for the combined case is that the two autoscalers must not react to the same signal. If the Horizontal Pod Autoscaler scales on CPU and VPA also adjusts CPU, they will fight. The article’s Istio example avoids this by scaling replicas on the number of connected proxies and letting VPA handle CPU and memory per pod. That pattern generalises: find a business or application metric for horizontal scaling, such as queue depth, connections or requests per second, and leave resource sizing to VPA.
What engineering teams should take from it
A few practices stand out as broadly applicable, regardless of fleet size.
Start in observe-only mode. VPA’s “Off” mode produces recommendations without changing anything. Running it for a few weeks across your workloads is cheap and tells you exactly how far your requests are from reality. Often that report alone is enough to justify the work.
Let VPA manage requests, not limits. The article recommends restricting VPA to requests and leaving limits as declared. We agree. Limits are your safety net for spikes, and having them recalculated from ratios makes behaviour hard to reason about during an incident.
Set realistic expectations. As the source puts it, “The Vertical Pod Autoscaler (VPA) is intended for steady-state, long-term resource optimization rather than handling sudden workload traffic spikes.” VPA is a slow, considered adjustment. Spikes are the job of horizontal scaling and sensible limits.
Bound the recommendations. Without a maximum, VPA can recommend a pod larger than any node you have, leaving it stuck in Pending. Set ceilings that reflect your node sizes, and floors that cover startup bursts, which are easy to miss when recommendations are based on steady-state usage.
Treat evictions as a cost. Applying new requests has traditionally meant restarting pods. Disruption budgets, minimum replica counts and a less frequent updater interval keep that churn under control. Newer in-place resizing reduces the problem, but it’s worth confirming what your Kubernetes version actually supports before relying on it.
The rollout pattern matters as much as the configuration. A staged approach looks something like this:
stateDiagram-v2 [*] --> Observe Observe --> NonProd : recommendations reviewed NonProd --> LowRiskProd : bake time passed LowRiskProd --> CriticalProd : no regressions NonProd --> Static : rollback LowRiskProd --> Static : rollback CriticalProd --> Static : rollback Static --> Observe
The rollback path back to static values is the part teams tend to skip. It shouldn’t be. Being able to switch autoscaling off for one workload, quickly and without a redeploy, is what makes the rest of the rollout safe to attempt.
What this means if you’re buying or running the work
For engineering leaders and product owners, the takeaway is that right-sizing is a program of work with risk attached, not a configuration toggle. A vendor or internal team promising large savings by “turning on autoscaling” is skipping the part that protects production.
Questions we’d recommend asking before approving this kind of work:
- What is the baseline? How much of the cluster is reserved versus used today, and how will savings be measured afterwards? Without cost tracking you’re relying on anecdote.
- Who owns resource settings after the change? If the answer is “the autoscaler”, ask who reviews its behaviour and who gets paged when it misbehaves.
- How will it be rolled out? Look for observe-only first, non-production before production, low-risk services before critical ones, and defined bake times.
- How do we roll back? It should be fast, per workload and tested.
- Which workloads are excluded? Stateful systems, latency-sensitive services and anything with unusual startup behaviour may need different treatment.
It is also worth keeping the scale in perspective. The savings Salesforce reports come from a fleet far larger than most local organisations will operate. For a mid-sized platform, the dollar figure may be modest, while the reliability benefit of accurate requests (fewer out-of-memory kills, less CPU throttling, more predictable scheduling) can matter more. Build the business case on both.
Finally, don’t let cost become the only metric. A right-sizing effort that saves money but increases restarts or tail latency has simply moved the cost somewhere less visible. Track eviction rates, restart counts and latency alongside spend.
Where to start
If you do one thing this quarter, enable VPA in observe-only mode across a non-production cluster and compare its recommendations to your current requests. The gap will tell you whether this is a priority and roughly where the effort should go.
If you’d like a second opinion on your cluster sizing, autoscaling strategy or how to scope this work safely, you’re welcome to book a free 30-minute session. We’re happy to talk it through, whatever stage you’re at.
← All posts