MySphere Posts

When you delete a node using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods.

• To delete the node from the UPI installation, the node must be firstly drained and then marked unschedulable prior to deleting it:
  
  

$ oc adm cordon <node_name>
$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets
- Ensure also that there are no current jobs/cronjobs being ran or scheduled in this specific node as the draining does not take it into consideration.
- For Red Hat OpenShift Container Platform 4.7+, utilize the option `--delete-emptydir-data` in case `--delete-local-data` doesn't work. The `--delete-local-data` option is deprecated in favor of `--delete-emptydir-data`.

$ oc get node <node_name> -o yaml > backupnode.yaml

Before proceeding with deletion of the node, it needs to be under "power off" status:
$ oc delete node <node_name>

Although the node object is now deleted from the cluster, it can still rejoin the cluster after reboot or if the kubelet service is restarted. To permanently delete the node and all its data, you must decommission the node once it is in shutdown mode.

Once the node is deleted, it can be ready for a power-off activity, or if it is needed to rejoin the cluster, it could be possible to either restart the kubelet or create the yaml back:

$ oc create -f backupnode.yaml

In order to get the node back, it can also be back by restarting kubelet:

$ systemctl restart kubelet

If it is needed to destroy then all the data from the worker node to delete all the software installed, execute the following:

# nohup shred -n 25 -f -z /dev/[HDD]
This command will overwrite all data on /dev/[HDD] repeatedly, in order to make it harder for even very expensive hardware probing to recover the data. Command line parameter -z will overwrite this device with zeros at the end of cycle to re-write data 25 times (it can be overridden with -n [number]).

One should consider running this command from RescueCD.

In order to monitor the deletion of the node, get the kubelet live logs:

$ oc adm node-logs <node-name> -u kubelet

https://access.redhat.com/solutions/4976801

In the context of a Kubernetes cluster, CPU throttling still refers to the process of limiting the amount of CPU time a container or pod can use, but it’s slightly different than throttling within an individual CPU or device, as Kubernetes manages resources at the container level. Here’s a breakdown of how CPU throttling works in a Kubernetes environment:

1. CPU Resources in Kubernetes

Kubernetes allows you to specify how much CPU a container can request and how much it is allowed to consume. This is done through resource requests and limits:

• CPU Request: The minimum CPU resource that the container is guaranteed to have.
• CPU Limit: The maximum CPU resource the container can use.

Kubernetes uses CPU throttling to ensure that containers do not exceed their allocated CPU limits. If a container tries to use more CPU than it has been allocated (based on the CPU limit), Kubernetes will throttle the container’s CPU usage to prevent it from violating the resource limits.

2. How CPU Throttling Works in Kubernetes

• CPU Requests: When a container is scheduled on a node, Kubernetes ensures that the requested CPU is available to the container. If the node doesn’t have enough available CPU, the pod may not be scheduled.
• CPU Limits: If a container exceeds its CPU limit (i.e., tries to use more CPU than what is specified in the limit), Kubernetes throttles the container’s CPU usage. The system does this by applying CPU usage constraints (using mechanisms like CFS (Completely Fair Scheduler) in Linux) to ensure that the container doesn’t exceed its allocated CPU time.
• CFS Throttling: The CFS quota system controls how much CPU a container can use. If a container tries to use more CPU than its allocated limit, the kernel uses a mechanism called CFS throttling. Essentially, the Linux kernel will temporarily stop a container from using the CPU until it is within the allowed usage range.
• Exceeding Limits: If a container tries to use more CPU than its limit allows (e.g., 1 CPU core), Kubernetes will restrict or throttle the container, reducing its access to the CPU until it falls back within the limit.

3. Example: CPU Limits and Throttling

Suppose you define a pod in Kubernetes with the following resource configuration:

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: myimage
    resources:
      requests:
        cpu: "500m"      # 0.5 CPU core requested
      limits:
        cpu: "1000m"      # 1 CPU core max

• Request: The container is guaranteed 500 milli-CPU (or 0.5 CPU core).
• Limit: The container can burst up to 1000 milli-CPU (or 1 CPU core).

If the container tries to use more than 1 CPU core (e.g., if the workload spikes and tries to use 1.5 CPU cores), Kubernetes will throttle it back down to 1 core.

4. How Throttling Happens in Practice

• Within Node: If there are multiple containers on the same node and they exceed their CPU limits, the Linux kernel (via CFS) enforces throttling to ensure that no container exceeds its CPU limit. This can cause delays or latency in container performance, especially when several containers are competing for CPU on a node.
• Overcommitment: If a node is overcommitted (i.e., the sum of all container CPU limits exceeds the physical capacity of the node), Kubernetes will throttle the containers that try to exceed the available CPU capacity.

5. Monitoring CPU Throttling in Kubernetes

You can monitor CPU throttling in a Kubernetes cluster by observing certain metrics, such as:

• container_cpu_cfs_throttled_seconds_total: This metric in Prometheus shows how much time a container has been throttled by the kernel (CFS throttling).
• container_cpu_usage_seconds_total: This metric shows the total CPU usage by a container, which can help correlate throttling behavior with usage spikes.

You can query these metrics in Prometheus and Grafana to see if containers are being throttled and to identify performance bottlenecks.

6. What Happens When Throttling Occurs?

When CPU throttling happens, a container might experience:

• Increased Latency: Throttling limits the amount of CPU time available to the container, leading to increased response times.
• Reduced Performance: The container may be unable to process requests as quickly, affecting application performance.
• Delays in Processing: If the container cannot access enough CPU resources, jobs that require more compute power will queue up and take longer to complete.

7. How to Avoid CPU Throttling in Kubernetes

To avoid CPU throttling and ensure containers have the necessary resources:

• Proper Resource Allocation: Set appropriate CPU requests and limits. The request should reflect the expected CPU usage, while the limit should provide headroom for occasional spikes.
• Monitor Resource Usage: Use monitoring tools like Prometheus, Grafana, and Kubernetes metrics server to observe resource usage and throttling events.
• Avoid Overcommitment: Ensure that the sum of the CPU requests of all containers on a node doesn’t exceed the total CPU capacity of that node.
• Horizontal Scaling: If a container regularly hits its CPU limits, consider scaling the application horizontally by adding more pods to distribute the load.

8. Conclusion

In a Kubernetes cluster, CPU throttling is primarily a mechanism to enforce resource limits and ensure that each container gets its fair share of CPU time, preventing any single container from monopolizing resources. While this helps maintain system stability and prevent resource exhaustion, it can result in performance degradation if a container is constantly throttled. Proper resource allocation and monitoring are key to avoiding excessive throttling and ensuring efficient operation of your Kubernetes workloads.