Datadog on Kubernetes Node Management

Datadog, the observability platform used by thousands of companies, runs on dozens of self-managed Kubernetes clusters in a multi-cloud environment, adding up to tens of thousands of nodes, or hundreds of thousands of pods. This infrastructure is used by a wide variety of engineering teams at Datadog, with different feature and capacity needs.

How do we make sure that tens of thousands of nodes, with very different specifications and on different clouds are healthy, updated with the latest security patches, and running an updated version of the kubelet and container runtime, without breaking applications or interrupting more than a thousand engineers that rely on this infrastructure for their daily job?

In this episode, Adrien Trouillaud, Engineering Manager and David Benque, Staff Software Engineer, both part of the Compute team, shared their strategies, lessons learned, and practical tips on how to successfully manage a huge fleet of Kubernetes nodes.

00:00 - Introduction
03:57 - Life of a Node
09:14 - Provisioning & Autoscaling
14:06 - Scaling Down
19:06 - Eviction API
34:00 - Draining
42:56 - Node Problem Detector
47:14 - Mass Eviction
52:04 - Q&A