How We Scaled Kubernetes to 10,000 Pods in Production

Scaling started as an org problem, not a Kubernetes problem

When people hear "10,000 pods," they usually jump straight to control plane tuning. That matters, but it was not our first bottleneck. Our first bottleneck was the way teams requested capacity, published services, and treated clusters as a dumping ground for every workload that happened to run in a container.

At around 1,500 pods, we could still absorb bad habits. At 4,000, those habits became incidents. By the time we crossed 7,000, every weak assumption showed up as either runaway cost, noisy-neighbor behavior, or deployment latency.

The lesson was simple: large-scale Kubernetes is mostly about creating predictable boundaries. Once we accepted that, the technical work became much clearer.

We stopped thinking in terms of one giant cluster

The early temptation was to keep adding nodes to a small number of regional clusters. That looked efficient on paper, but it created blast-radius problems we did not want. Cluster upgrades were stressful, maintenance windows became political, and scheduler contention affected unrelated teams.

We moved to a cell-based model:

• Shared platform clusters for internal tooling and control services
• Dedicated workload clusters per environment and business domain
• Specialized node pools for CPU-heavy, memory-heavy, and GPU workloads
• Hard quotas for namespaces so one team could not quietly consume the region

This did two things. First, it reduced the operational radius of failure. Second, it made capacity planning a lot more honest. Teams could see what they were actually consuming instead of hiding behind one huge regional budget.

Scheduling quality mattered more than raw cluster size

The default scheduler is excellent, but our workload mix needed more intent. Some services were latency-sensitive, some were batch-oriented, and some could tolerate interruption on spot instances. Treating all of them the same left money and performance on the table.

We tightened the scheduling model around three questions:

1. Does this workload need stable latency or cheap compute?
2. Does it depend on large stateful backends in a specific zone?
3. Is it safe to preempt or drain during regional balancing?

That led to stronger use of affinity, anti-affinity, topology spread constraints, taints, tolerations, and priority classes. We also enforced requests and limits with admission policies. Without that, "Kubernetes autoscaling" often becomes "Kubernetes guessing."

The result was not glamorous, but it was effective. Pods landed on the right class of node more consistently, and the scheduler had enough truthful information to make decent decisions under load.

Autoscaling only improved after we fed it better signals

Horizontal Pod Autoscaler works well if your signal maps cleanly to user demand. CPU alone did not work for us. Some services were bottlenecked by queue depth. Others were bottlenecked by connection pools, outbound API rate limits, or memory pressure.

We ended up with a layered scaling model:

• HPA for stateless services with stable request-driven traffic
• KEDA for event and queue-based workers
• Cluster Autoscaler for coarse infrastructure growth
• Buffer capacity by node pool so we could absorb bursts without waiting for new nodes

The critical change was operational, not exotic. Every service owner had to define the signal that best represented user-facing saturation. If a team could not explain why a metric predicted demand, we did not let it drive scaling.

That slowed down onboarding at first. It also prevented the worst kind of scaling failure: spending more money while staying just as slow.

Networking and service-to-service traffic became cost centers

At low scale, east-west traffic is easy to ignore. At production scale, it becomes a meaningful source of both latency and cost. Cross-zone calls looked harmless until we graphed them by service pair and saw how much chatty traffic we were pushing around the region.

We cut that down through a few boring but high-leverage changes:

• Co-locating latency-sensitive services with the data stores they hit most often
• Replacing broad internal load balancing with clearer service boundaries
• Enforcing timeouts and retries centrally instead of letting every team invent its own policy
• Measuring cross-zone traffic as a platform KPI, not just a finance report line item

We also improved how services exposed telemetry for connection pooling, queue depth, and request fan-out. Once those numbers were visible, teams made smarter architectural tradeoffs without much pushing from the platform group.

Reliability came from standardization, not heroics

By the time we were running 10,000 pods, the most valuable platform feature was consistency. Every service needed the same deployment contract, the same health-check semantics, the same rollback pattern, and the same observability baseline.

We standardized around:

• Progressive rollouts for every production service
• Startup, readiness, and liveness probes that reflected actual health
• Namespace-level policies for quotas, network controls, and secret handling
• Default dashboards for saturation, availability, errors, and deployment health

This reduced the amount of custom reasoning required during incidents. The platform became easier to operate because it was easier to understand.

The headline number mattered less than the operating model behind it

Running 10,000 pods sounds impressive, but that number by itself is not the goal. The goal is to keep shipping while the system grows. For us, success meant regional upgrades felt routine, on-call responders could trust their dashboards, and finance conversations became grounded in real workload behavior rather than guesswork.

The durable takeaways were these:

• Scale is easier when you design for boundaries early
• Scheduler inputs have to reflect reality or autoscaling becomes theater
• Network costs are architecture feedback, not just a billing artifact
• Platform standards reduce incidents faster than one-off optimizations

If you are heading toward similar scale, do not wait for Kubernetes to "break" before tightening the operating model. Most large-cluster pain is visible long before the pod count becomes dramatic.