![]() ![]() To safely enable resource quota enforcement, we look at historical resource usage, add 20% headroom on top of peak value, and set it as the initial value for resource quota for every project. In addition, we also rejected any pod specification without resource requests and limits in the CRD validation layer.Īnother challenge we overcame was to streamline quota management across teams and organizations. We achieved that by adding default resource requests and limits for all pods and containers in the CRD transformation layer. ![]() In Pinterest Kubernetes platform, workloads in different namespaces are owned by different teams for different projects, and platform users configure their workload via Pinterest CRD. One challenge we faced is that enforcing resource quota in every namespace implicitly requires all pods and containers to have resource requests and limits specified. It is key to have resource usage bounded in every namespace in order to ensure stability. As our previous incident mentioned, a surge of pod creation in a single namespace could overload kube-apiserver and cause cascading failure. Kubernetes already provides resource quotas management to ensure no namespace can request or occupy unbounded resources in most dimensions: pods, cpu, memory, etc. In addition, we had to temporarily scale up Kubernetes master nodes that host kube-apiserver and etcd to reduce resource constraint.įigure 3: Pinterest Kubernetes Platform Architecture (blue is in-house, green is open source) Governance Resource Quota Enforcement In order to mitigate the incident, we had to perform etcd operations like compacting old revisions, defragmenting excessive spaces, and disabling alarms to recover it. The burst of writes caused etcd to reach its total data size limit and start rejecting all write requests, and the platform lost availability in terms of workload management. The unbound retry from Kubelets resulted in a 7x jump on kube-apiserver load. The kube-apiserver started to first experience latency spikes and increased error rate, and then get Out of Memory (OOM) killed due to resource limit. In early 2020, one of our clusters experienced a sudden spike of pods creation (~3x above planned capacity), causing the cluster autocalor to bring up 900 nodes to accommodate the demand. As more business critical workloads onboard the Kubernetes platform, the expectations on platform reliability naturally rise to a new level. It requires the Kubernetes platform to be more scalable in order to catch up with the increasing load from workload management, pods scheduling and placement, and node allocation and deallocation. ![]() 2020 in a Short StoryĪs user adoption grows, the variety and number of workloads increases. Based on our user survey, the top three benefits shared by our users are reducing the burden of managing compute resources, better resource and failure isolation, and more flexible capacity management.īy the end of 2020, we orchestrated 35K+ pods with 2500+ nodes in our Kubernetes clusters - supporting a wide range of Pinterest businesses - and the organic growth is still rocket high. In general, Kubernetes platform users gave positive feedback. Since then, we have delivered many features to facilitate customer adoption, ensure reliability and scalability, and build up operational experience and best practices. It has been more than a year since we shared our Kubernetes Journey at Pinterest. Scaling Kubernetes with Assurance at PinterestĪnson Qian| Software Engineer, Cloud Runtime Introduction
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |