The root cause was an accumulation of finished Jobs created by the import controller for each customer file, generating roughly nine hundred Jobs per day since early 2024 and leaving behind corresponding Pods.

Etcd grew to about 6.4 GB, turning list and resync operations into slow bottlenecks and causing timeouts rather than explicit errors.

The article advocates proactive governance of resource lifecycles to prevent performance degradation and outages from accumulating finished objects.

Mitigations included: every new Job now has a TTL of one hour, admissions policy requires TTL presence, and alerts fire when any resource type in a namespace exceeds 5,000 objects.

Deleting the backlog was resource-intensive, needing oldest-first deletions in batches of 500 over two days, followed by compaction and defragmentation of each etcd member.

The import controller created Jobs directly via the API rather than a CronJob; unlike CronJobs, these Jobs never auto-deleted, leading to about 340,000 Jobs and a similar number of Pods accumulating.

The core lesson is that Kubernetes objects function like database rows read by all components; high-rate object creation must be matched with same-day removal rules to avoid unbounded growth and platform strain.

A cluster-wide deploy issue began on a Thursday morning with all deploys failing due to timeouts in the rollout tool, even though pods were healthy and running.