Resolving Kubernetes Taint Issues on Master Nodes: A Guide for K3S Users

Encounter issues with Kubernetes taints on master nodes in your K3S cluster? This guide explores the problem and offers clear, actionable solutions.
---
This video is based on the question https://stackoverflow.com/q/72880188/ asked by the user 'Xhelliom' ( https://stackoverflow.com/u/13266581/ ) and on the answer https://stackoverflow.com/a/72887391/ provided by the user 'Xhelliom' ( https://stackoverflow.com/u/13266581/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: Kubernetes taint on master but no scheduling on worker node

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Understanding the Issue: Kubernetes Taint on Master Nodes

If you're using Kubernetes K3S and faced the frustrating scenario of pods failing to schedule due to taints on master nodes, you're not alone. The problem can be attributed to the way Kubernetes handles node taints and affinity. In this post, we will explore a specific case of taint issues, break down the concepts involved, and discuss how to properly configure your Kubernetes cluster to avoid such scheduling problems.

The Problem

In a K3S cluster, a user was faced with the following error while trying to deploy pods:

[[See Video to Reveal this Text or Code Snippet]]

Background Context

The user has four K3S servers: 3 nodes serve as control planes while 1 serves as a worker node.

Initially, all nodes had no taints, allowing pods to be scheduled freely.

The user desired to add taints to the master nodes to prevent pods from scheduling on them. They added the following taint:

[[See Video to Reveal this Text or Code Snippet]]

Following this change, an existing deployment failed to schedule, despite there being available worker nodes.

Solution: Breaking Down the Deployment Configuration

To understand why the deployment did not schedule, we need to examine the deployment configuration in detail as well as the labels applied to the nodes.

Step 1: Check Node Labels

The first step is to verify the configured labels and current state of the nodes with the following command:

[[See Video to Reveal this Text or Code Snippet]]

This command will help identify which labels are currently applied to each node. Upon running it, the output might show labels similar to this:

[[See Video to Reveal this Text or Code Snippet]]

Step 2: Identify the Root Cause

In this scenario, the existence of the label type=slow on baal-01 and baal-02 (the tainted control plane nodes) conflicts with the nodeSelector defined in the deployment. The deployment is configured to request pods on nodes with the label type=slow. However, since those nodes have a taint preventing scheduling, this results in no nodes being available to host the pod.

Step 3: Correcting the Deployment

To resolve this issue, you have a few options:

Adjust the Node Selector: Modify the nodeSelector in your deployment configuration to match the available worker nodes, or remove it entirely if it’s not necessary.

Revisit Node Assignment: Ensure that your deployments have alternative nodes that meet required conditions without conflicting with newly added taints.

Best Practices Moving Forward

Use Taints and Tolerations Wisely: Make sure that your pods can tolerate the taints on your willing nodes.

Monitor Node State Regularly: Use commands to frequently check the state and labels of your nodes to avoid unwanted scheduling issues.

Document Changes: Always document changes made to your cluster configuration for reference, ensuring that you can track and manage them easily.

Conclusion

Taints are a powerful feature in Kubernetes that can help prevent unwanted scheduling on specific nodes. However, understanding how they interact with node selectors and affinities is key to avoiding deployment issues. By carefully managing these configurations, you can ensure smooth and efficient operation of your K3S cluster.

By following the steps outlined above, you're well on your way to resolving any taint-related issues and ensuring your deployments run smoothly across your Kubernetes cluster.