Category: <span>Uncategorized</span>

Kubernetes has long been a cornerstone for managing containerized workloads, and its continuous evolution keeps it at the forefront of cloud-native technologies. One of the exciting advancements in recent releases is the enhancement of startup scaling capabilities, particularly through features like Kube Startup CPU Boost and dynamic resource scaling. In this blog post, we’ll dive into what startup scaling is, how it works, and why it’s a significant addition for Kubernetes users looking to optimize application performance during startup.

What is Startup Scaling in Kubernetes?

Startup scaling refers to the ability to dynamically allocate additional resources, such as CPU, to pods during their initialization phase to accelerate startup times. This is particularly useful for applications that require significant resources during their boot process but may not need those resources once they’re running steadily. By providing a temporary resource boost, Kubernetes ensures faster deployment and improved responsiveness without over-provisioning resources long-term.

The concept of startup scaling ties closely with Kubernetes’ broader autoscaling capabilities, including Horizontal Pod Autoscaling (HPA) and Vertical Pod Autoscaling (VPA). However, startup scaling specifically addresses the transient needs of applications during their startup phase, a critical period for performance-sensitive workloads.

Key Features of Startup Scaling

One of the standout implementations of startup scaling is the Kube Startup CPU Boost, introduced as an open-source operator in Kubernetes 1.28 and further refined in subsequent releases. Here’s how it works:

  • Dynamic Resource Allocation: Kube Startup CPU Boost temporarily increases CPU resources for pods during their startup phase. Once the pod is fully initialized, the operator scales down the resources to their normal levels, optimizing resource utilization.
  • No Pod Restarts: Unlike traditional vertical scaling, which might require pod restarts to adjust resources, this feature leverages in-place resource resizing, a capability introduced in Kubernetes 1.27 and graduated to beta in 1.33. This ensures zero downtime during resource adjustments.
  • Targeted Use Cases: Startup scaling is ideal for applications with heavy initialization processes, such as machine learning workloads, databases, or complex microservices that perform significant computations or data loading during startup.

How Does Kube Startup CPU Boost Work?

The Kube Startup CPU Boost operator monitors pods and applies a predefined CPU boost policy during their startup phase. Here’s a simplified workflow:

  1. Pod Creation: When a pod is created, the operator identifies it as a candidate for CPU boost based on configured policies (e.g., specific labels or annotations).
  2. Resource Adjustment: The operator temporarily increases the pod’s CPU allocation (requests and/or limits) to speed up initialization.
  3. Monitoring and Scaling Down: Once the pod reaches a stable state (determined by readiness probes or a timeout), the operator reduces the CPU allocation back to its baseline, ensuring efficient resource usage.
  4. In-Place Resizing: Leveraging the in-place pod vertical scaling feature, these adjustments occur without restarting the pod, maintaining application availability.

This process is seamless and integrates with Kubernetes’ existing autoscaling mechanisms, making it a natural fit for clusters already using HPA or VPA.

Benefits of Startup Scaling

The introduction of startup scaling, particularly through Kube Startup CPU Boost, brings several advantages:

  • Faster Application Startup: By allocating more CPU during initialization, applications launch quicker, reducing latency for end-users.
  • Resource Efficiency: Temporary boosts prevent over-provisioning, ensuring resources are only allocated when needed.
  • Improved User Experience: Faster startup times are critical for user-facing applications, where delays can impact satisfaction.
  • Support for Resource-Intensive Workloads: AI/ML applications, databases, and other compute-heavy workloads benefit significantly from this feature.
  • No Downtime: In-place resource resizing ensures that scaling operations don’t disrupt running applications.

Getting Started with Startup Scaling

To leverage startup scaling in your Kubernetes cluster, you’ll need to:

  1. Enable the InPlacePodVerticalScaling Feature Gate: This is enabled by default in Kubernetes 1.33, allowing in-place resource resizing. Verify your cluster version and configuration to ensure compatibility.
  2. Install the Kube Startup CPU Boost Operator: This open-source operator can be deployed via a Helm chart or directly from its GitHub repository. Configure it with policies that match your workload requirements.
  3. Configure Pod Annotations: Use annotations to specify which pods should receive a CPU boost and define the boost parameters (e.g., duration or resource limits).
  4. Monitor and Optimize: Use Kubernetes monitoring tools like Prometheus or Grafana to track the impact of startup scaling on your application performance and resource usage.

Best Practices

  • Test in a Staging Environment: Before enabling startup scaling in production, test it in a non-critical environment to understand its impact on your workloads.
  • Combine with Autoscaling: Use startup scaling alongside HPA and VPA for a comprehensive scaling strategy that handles both startup and runtime demands.
  • Monitor Resource Usage: Ensure your cluster has sufficient resources to handle temporary boosts, especially in multi-tenant environments.
  • Fine-Tune Boost Policies: Adjust boost duration and resource limits based on your application’s startup behavior to avoid over- or under-provisioning.

What’s Next for Startup Scaling?

As Kubernetes continues to evolve, we can expect further refinements to startup scaling. The graduation of in-place pod vertical scaling to beta in Kubernetes 1.33 is a promising step, and future releases may bring this feature to stable status. Additionally, enhancements to the Kube Startup CPU Boost operator could include more granular control over boost policies or integration with other resource types, such as memory or GPU.

Conclusion

Startup scaling, exemplified by Kube Startup CPU Boost, is a powerful addition to Kubernetes’ scaling arsenal. By addressing the unique resource needs of applications during their startup phase, it enables faster deployments, better resource efficiency, and improved user experiences. Whether you’re running AI/ML workloads, databases, or microservices, this feature can help optimize your Kubernetes cluster for performance and cost.

To learn more, check out the official Kubernetes documentation or explore the Kube Startup CPU Boost project on GitHub. Start experimenting with startup scaling today and see how it can transform your application deployments

Uncategorized

Released on February 24, 2025, Red Hat OpenShift 4.18 is here, bringing a fresh wave of enhancements to the Kubernetes-powered hybrid cloud platform. Whether you’re a developer, a sysadmin, or an IT decision-maker, this update has something to pique your interest—think beefed-up security, slick virtualization upgrades, and tools to make your clusters easier to monitor and manage. Let’s dive into what’s new and why OpenShift 4.18 might just be the upgrade your team’s been waiting for.

Security That Packs a Punch

In today’s world, keeping sensitive data safe is non-negotiable, and OpenShift 4.18 steps up to the plate. One standout feature is the Secrets Store CSI Driver Operator, now fully available after being a tech preview since 4.14. This nifty tool lets your workloads tap into external secrets managers—like Azure Key Vault, Google Cloud Secret Manager, or HashiCorp Vault—without leaving sensitive credentials lying around on your cluster. Secrets are mounted as ephemeral volumes, meaning the cluster stays blissfully unaware of the juicy details. Pair this with OpenShift GitOps or Pipelines, and you’ve got a secure, streamlined way to handle credentials across your apps. It’s a game-changer for teams juggling compliance and agility.

Virtualization Gets a Glow-Up

If you’re running virtual machines alongside containers, OpenShift 4.18’s virtualization updates will catch your eye. Built on Kubernetes 1.31 and CRI-O 1.31, this release supercharges OpenShift Virtualization. A big win?User Defined Networks (UDNs) are now generally available, giving you custom networking options—Layer 2, Layer 3, or localnet—for your pods and VMs via OVN-Kubernetes. It’s all about flexibility.

Bare metal fans, rejoice: 4.18 expands support across Google Cloud (C3, C4, C4A, N4 machines) and Oracle Cloud Infrastructure, with deployment options via Assisted or Agent-based Installers. Plus, the Migration Toolkit for Virtualization (MTV) gets smarter with user-defined PVC names and optimized migration schedules, making VM transfers faster and less of a headache. Whether you’re hybrid cloud-curious or all-in, these updates make managing VMs smoother than ever.

Observability: See It All, Fix It Fast

Ever wished you could get a clearer picture of what’s happening in your cluster? OpenShift 4.18 delivers with the Cluster Observability Operator (COO) 1.0.0, now GA. This unifies metrics, logs, and traces into one tidy package, complete with dashboards, a troubleshooting UI, and distributed tracing. Add in multi-namespace Prometheus alerts and GPU accelerator metrics, and you’ve got a toolkit to spot issues before they spiral. It’s like giving your cluster a superpower: total visibility.

Developers, This One’s for You

The developer experience in 4.18 is all about small wins that add up. The OpenShift Console now boasts colored Tekton Pipeline logs (because who doesn’t love a little flair?), one-click YAML imports via OpenShift Lightspeed, and a YAML editor with dark/light mode support—perfect for late-night coding sessions. There’s also the OpenShift CLI Manager Operator (tech preview), which simplifies CLI management in disconnected environments. These tweaks might not scream “revolutionary,” but they’ll make your day-to-day a little smoother.

Under the Hood: Core Platform Goodies

OpenShift 4.18 swaps out RunC for Crun as the default container runtime (don’t worry, RunC’s still an option), aligning with OCI standards for a leaner, meaner runtime. Single-node clusters can now auto-recover after a shutdown—great for edge setups—and high-availability cloud clusters can snooze for up to 90 days without breaking a sweat. It’s the kind of reliability that keeps operations humming.

Lifecycle and Availability

Red Hat backs 4.18 with at least 6 months of full support (or 90 days after 4.19 drops, whichever’s longer), followed by maintenance support until August 2026. As an even-numbered release, it might snag Extended Update Support (EUS), stretching its lifecycle to 24 months. You can deploy it anywhere—on-prem, AWS, Azure, Google Cloud, Oracle Cloud, or bare metal—starting now.

Why It Matters

OpenShift 4.18 isn’t about reinventing the wheel; it’s about making the wheel spin better. From tighter security to sharper observability and a friendlier developer experience, it’s a release that listens to what users need in 2025: tools that work hard so you don’t have to. Whether you’re modernizing apps, managing VMs, or scaling across clouds, 4.18 has your back.

Ready to explore? Check out the [official OpenShift 4.18 Release Notes](https://docs.openshift.com) for the full scoop, and let us know what you think in the comments. What feature are you most excited to try?

Uncategorized

I was installing the IBM MQ Server on a linux box yesterday. I got the error bellow:

[root@gateway MQServer]# dnf install rpm-build
Updating Subscription Management repositories.
Instana 14 B/s | 20 B 00:01
Errors during downloading metadata for repository ‘instana-agent’:

Status code: 401 for https://_:[email protected]/agent/rpm/generic/x86_64/repodata/repomd.xml (IP: 34.253.98.124)
Error: Failed to download metadata for repo ‘instana-agent’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Solution:

Delete the Instana Agent repo: rm /etc/yum.repos.d/Instana-Agent.repo
Clear dnf cache: dnf clean all
run dnf install rpm-build again

Uncategorized

“I have lots of photo files. Since 2006, when I purchased my first digital camera, the number of photos has grown quickly, and after getting an iPhone, the number of photos exploded.

With the high number of photos, the number of backups grew as well.

I decided to organize all backups and create folders using the format YYYY-MM from the metadata of the photo files.”

Bellow the python script. The script runs on macos:

import os
import shutil
import datetime
import logging
import tkinter as tk
from tkinter import filedialog
from PIL import Image, ExifTags
import pillow_heif
import piexif
import struct

# Setup logging
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s")

ATOM_HEADER_SIZE = 8
EPOCH_ADJUSTER = 2082844800  # Difference between Unix and QuickTime epoch

def get_file_date(file_path):
    try:
        if file_path.lower().endswith(".heic") and pillow_heif.is_supported(file_path):
            heif_file = pillow_heif.open_heif(file_path, convert_hdr_to_8bit=False)
            exif_data = heif_file.info.get("exif")
            if exif_data:
                exif_dict = piexif.load(exif_data)
                date_str = exif_dict["Exif"].get(piexif.ExifIFD.DateTimeOriginal)
                if date_str:
                    return datetime.datetime.strptime(date_str.decode("utf-8"), "%Y:%m:%d %H:%M:%S")
        
        elif file_path.lower().endswith((".jpg", ".jpeg")):
            with Image.open(file_path) as img:
                exif_data = img.getexif()
                if exif_data:
                    exif_dict = {ExifTags.TAGS.get(tag, tag): value for tag, value in exif_data.items()}
                    logging.debug(f"EXIF metadata for {file_path}: {exif_dict}")
                    
                    if "DateTimeOriginal" in exif_dict:
                        date_str = exif_dict["DateTimeOriginal"]
                    elif "DateTime" in exif_dict:
                        date_str = exif_dict["DateTime"]
                    else:
                        date_str = None
                        logging.warning(f"No DateTimeOriginal or DateTime found for {file_path}")
                    
                    if date_str:
                        try:
                            logging.debug(f"Extracted date string from EXIF: {date_str}")
                            return datetime.datetime.strptime(date_str, "%Y:%m:%d %H:%M:%S")
                        except ValueError as ve:
                            logging.error(f"Error parsing date for {file_path}: {ve}")
                    else:
                        logging.warning(f"DateTime metadata missing or unreadable for {file_path}")
                else:
                    logging.warning(f"No EXIF metadata found for {file_path}")
    
    except Exception as e:
        logging.error(f"Error extracting date from {file_path}: {e}")
    
    # If metadata exists but could not be parsed, use file birth time (creation date on macOS)
    file_stats = os.stat(file_path)
    file_birth_time = file_stats.st_birthtime
    logging.debug(f"Using file birth time for {file_path}: {datetime.datetime.fromtimestamp(file_birth_time)}")
    return datetime.datetime.fromtimestamp(file_birth_time)

def move_files_to_folders(source_folder):
    for filename in os.listdir(source_folder):
        file_path = os.path.join(source_folder, filename)
        if filename.lower().endswith((".jpg", ".jpeg", ".heic", ".mov")):
            date_taken = get_file_date(file_path)
            if date_taken:
                folder_name = date_taken.strftime("%Y-%m")
            else:
                logging.warning(f"Could not determine date for {file_path}, using 'unknown' folder.")
                folder_name = "unknown"
            
            dest_folder = os.path.join(source_folder, folder_name)
            os.makedirs(dest_folder, exist_ok=True)
            
            dest_file_path = os.path.join(dest_folder, filename)
            count = 1
            while os.path.exists(dest_file_path):
                name, ext = os.path.splitext(filename)
                dest_file_path = os.path.join(dest_folder, f"{name}_{count}{ext}")
                count += 1
            
            shutil.move(file_path, dest_file_path)
            logging.info(f"Moved {filename} to {dest_folder}")

if __name__ == "__main__":
    root = tk.Tk()
    root.withdraw()
    folder_selected = filedialog.askdirectory(title="Select the folder containing files")
    if folder_selected:
        move_files_to_folders(folder_selected)
        logging.info("File organization complete.")
    else:
        logging.warning("No folder selected.")

Uncategorized

I set up an OpenShift 4.16 cluster using UPI on top of VMware. The cluster has 3 Masters, 3 Worker Nodes, and 3 InfraNodes. The infra nodes were necessary to install IBM Storage Fusion.

After the setup, I needed to create a load balancer in front of the OpenShift cluster. There are several options, and one of them is HAProxy.

I just installed an RHEL 9 server, added 3 ips to the network card and setup the haproxy.

Prerequisites

  • A system running RHEL 9
  • Root or sudo privileges
  • A basic understanding of networking and load balancing

Step 1: Install HAProxy

First, update your system packages:

sudo dnf update -y

Then, install HAProxy using the package manager:

sudo dnf install haproxy -y

Verify the installation:

haproxy -v

Step 2: Configure HAProxy

The main configuration file for HAProxy is located at /etc/haproxy/haproxy.cfg. Open the file in a text editor:

sudo nano /etc/haproxy/haproxy.cfg

The configuration bellow was used for my cluster. Change the IP adresses to match

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
    #ssl-default-bind-ciphers PROFILE=SYSTEM
    #ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    tcp
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------

frontend api
    bind 192.168.252.171:6443
    default_backend controlplaneapi

frontend apiinternal
    bind 192.168.252.171:22623
    bind 192.168.252.171:22624
    default_backend controlplaneapiinternal

frontend secure
    bind 192.168.252.170:443
    default_backend secure

frontend insecure
    bind 192.168.252.170:80
    default_backend insecure

#---------------------------------------------------------------------
# static backend
#---------------------------------------------------------------------

backend controlplaneapi
    balance source
    server master-01  192.168.252.5:6443 check
    server master-02  192.168.252.6:6443 check
    server master-03  192.168.252.7:6443 check


backend controlplaneapiinternal
    balance source
    server master-01  192.168.252.5:22623 check
    server master-02  192.168.252.6:22623 check
    server master-03  192.168.252.7:22623 check
    server master-01  192.168.252.5:22624 check
    server master-02  192.168.252.6:22624 check
    server master-03  192.168.252.7:22624 check

backend secure
    balance source
    server worker-01  192.168.252.8:443 check
    server worker-02  192.168.252.9:443 check
    server worker-03  192.168.252.10:443 check
    server  worker-04   192.168.252.11:443 check
    server  worker-05   192.168.252.12:443 check
    server  worker-06   192.168.252.13:443 check

backend insecure
    balance roundrobin
    server worker-01  192.168.252.8:80 check
    server worker-02  192.168.252.9:80 check
    server worker-03  192.168.252.10:80 check
    server worker-04   192.168.252.11:80 check
    server worker-05   192.168.252.12:80 check
    server worker-06  192.168.252.13:80 check

Uncategorized

When you delete a node using the CLI, the node object is deleted in Kubernetes, but the pods that exist on the node are not deleted. Any bare pods not backed by a replication controller become inaccessible to OpenShift Container Platform. Pods backed by replication controllers are rescheduled to other available nodes. You must delete local manifest pods.

  • To delete the node from the UPI installation, the node must be firstly drained and then marked unschedulable prior to deleting it:

$ oc adm cordon <node_name>
$ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets
- Ensure also that there are no current jobs/cronjobs being ran or scheduled in this specific node as the draining does not take it into consideration.
- For Red Hat OpenShift Container Platform 4.7+, utilize the option `--delete-emptydir-data` in case `--delete-local-data` doesn't work. The `--delete-local-data` option is deprecated in favor of `--delete-emptydir-data`.

$ oc get node <node_name> -o yaml > backupnode.yaml

Before proceeding with deletion of the node, it needs to be under "power off" status:
$ oc delete node <node_name>

Although the node object is now deleted from the cluster, it can still rejoin the cluster after reboot or if the kubelet service is restarted. To permanently delete the node and all its data, you must decommission the node once it is in shutdown mode.

Once the node is deleted, it can be ready for a power-off activity, or if it is needed to rejoin the cluster, it could be possible to either restart the kubelet or create the yaml back:

$ oc create -f backupnode.yaml

In order to get the node back, it can also be back by restarting kubelet:

$ systemctl restart kubelet

If it is needed to destroy then all the data from the worker node to delete all the software installed, execute the following:

# nohup shred -n 25 -f -z /dev/[HDD]
This command will overwrite all data on /dev/[HDD] repeatedly, in order to make it harder for even very expensive hardware probing to recover the data. Command line parameter -z will overwrite this device with zeros at the end of cycle to re-write data 25 times (it can be overridden with -n [number]).

One should consider running this command from RescueCD.

In order to monitor the deletion of the node, get the kubelet live logs:

$ oc adm node-logs <node-name> -u kubelet

https://access.redhat.com/solutions/4976801

Uncategorized

Applying a specific node selector to all infrastructure components will guarantee that they will be scheduled on nodes with that label. See more details on node selectors in placing pods on specific nodes using node selectors, and about node labels in understanding how to update labels on nodes.

Our node label and matching selector for infrastructure components will be node-role.kubernetes.io/infra: "".

To prevent other workloads from also being scheduled on those infrastructure nodes, we need one of two solutions:

  • Apply a taint to the infrastructure nodes and tolerations to the desired infrastructure workloads.
    OR
  • Apply a completely separate label to your other nodes and matching node selector to your other workloads such that they are mutually exclusive from infrastructure nodes.

TIP: To ensure High Availability (HA) each cluster should have three Infrastructure nodes, ideally across availability zones. See more details about rebooting nodes running critical infrastructure.

TIP: Review the infrastructure node sizing suggestions

By default all nodes except for masters will be labeled with node-role.kubernetes.io/worker: "". We will be adding node-role.kubernetes.io/infra: "" to infrastructure nodes.

However, if you want to remove the existing worker role from your infra nodes, you will need an MCP to ensure that all the nodes upgrade correctly. This is because the worker MCP is responsible for updating and upgrading the nodes, and it finds them by looking for this node-role label. If you remove the label, you must have a MachineConfigPool that can find your infra nodes by the infra node-role label instead. Previously this was not the case and removing the worker label could have caused issues in OCP <= 4.3.

This infra MCP definition below will find all MachineConfigs labeled both “worker” and “infra” and it will apply them to any Machines or Nodes that have the “infra” role label. In this manner, you will ensure that your infra nodes can upgrade without the “worker” role label.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""

If you are not using the MachineSet API to manage your nodes, labels and taints are applied manually to each node:

Label it:

oc label node <node-name> node-role.kubernetes.io/infra=
oc label node <node-name> node-role.kubernetes.io=infra

Taint it:

oc adm taint nodes -l node-role.kubernetes.io/infra node-role.kubernetes.io/infra=reserved:NoSchedule node-role.kubernetes.io/infra=reserved:NoExecute

openshift Uncategorized

In the context of a Kubernetes cluster, CPU throttling still refers to the process of limiting the amount of CPU time a container or pod can use, but it’s slightly different than throttling within an individual CPU or device, as Kubernetes manages resources at the container level. Here’s a breakdown of how CPU throttling works in a Kubernetes environment:

1. CPU Resources in Kubernetes

Kubernetes allows you to specify how much CPU a container can request and how much it is allowed to consume. This is done through resource requests and limits:

  • CPU Request: The minimum CPU resource that the container is guaranteed to have.
  • CPU Limit: The maximum CPU resource the container can use.

Kubernetes uses CPU throttling to ensure that containers do not exceed their allocated CPU limits. If a container tries to use more CPU than it has been allocated (based on the CPU limit), Kubernetes will throttle the container’s CPU usage to prevent it from violating the resource limits.

2. How CPU Throttling Works in Kubernetes

  • CPU Requests: When a container is scheduled on a node, Kubernetes ensures that the requested CPU is available to the container. If the node doesn’t have enough available CPU, the pod may not be scheduled.
  • CPU Limits: If a container exceeds its CPU limit (i.e., tries to use more CPU than what is specified in the limit), Kubernetes throttles the container’s CPU usage. The system does this by applying CPU usage constraints (using mechanisms like CFS (Completely Fair Scheduler) in Linux) to ensure that the container doesn’t exceed its allocated CPU time.
  • CFS Throttling: The CFS quota system controls how much CPU a container can use. If a container tries to use more CPU than its allocated limit, the kernel uses a mechanism called CFS throttling. Essentially, the Linux kernel will temporarily stop a container from using the CPU until it is within the allowed usage range.
  • Exceeding Limits: If a container tries to use more CPU than its limit allows (e.g., 1 CPU core), Kubernetes will restrict or throttle the container, reducing its access to the CPU until it falls back within the limit.

3. Example: CPU Limits and Throttling

Suppose you define a pod in Kubernetes with the following resource configuration:

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: myimage
    resources:
      requests:
        cpu: "500m"      # 0.5 CPU core requested
      limits:
        cpu: "1000m"      # 1 CPU core max
  • Request: The container is guaranteed 500 milli-CPU (or 0.5 CPU core).
  • Limit: The container can burst up to 1000 milli-CPU (or 1 CPU core).

If the container tries to use more than 1 CPU core (e.g., if the workload spikes and tries to use 1.5 CPU cores), Kubernetes will throttle it back down to 1 core.

4. How Throttling Happens in Practice

  • Within Node: If there are multiple containers on the same node and they exceed their CPU limits, the Linux kernel (via CFS) enforces throttling to ensure that no container exceeds its CPU limit. This can cause delays or latency in container performance, especially when several containers are competing for CPU on a node.
  • Overcommitment: If a node is overcommitted (i.e., the sum of all container CPU limits exceeds the physical capacity of the node), Kubernetes will throttle the containers that try to exceed the available CPU capacity.

5. Monitoring CPU Throttling in Kubernetes

You can monitor CPU throttling in a Kubernetes cluster by observing certain metrics, such as:

  • container_cpu_cfs_throttled_seconds_total: This metric in Prometheus shows how much time a container has been throttled by the kernel (CFS throttling).
  • container_cpu_usage_seconds_total: This metric shows the total CPU usage by a container, which can help correlate throttling behavior with usage spikes.

You can query these metrics in Prometheus and Grafana to see if containers are being throttled and to identify performance bottlenecks.

6. What Happens When Throttling Occurs?

When CPU throttling happens, a container might experience:

  • Increased Latency: Throttling limits the amount of CPU time available to the container, leading to increased response times.
  • Reduced Performance: The container may be unable to process requests as quickly, affecting application performance.
  • Delays in Processing: If the container cannot access enough CPU resources, jobs that require more compute power will queue up and take longer to complete.

7. How to Avoid CPU Throttling in Kubernetes

To avoid CPU throttling and ensure containers have the necessary resources:

  • Proper Resource Allocation: Set appropriate CPU requests and limits. The request should reflect the expected CPU usage, while the limit should provide headroom for occasional spikes.
  • Monitor Resource Usage: Use monitoring tools like Prometheus, Grafana, and Kubernetes metrics server to observe resource usage and throttling events.
  • Avoid Overcommitment: Ensure that the sum of the CPU requests of all containers on a node doesn’t exceed the total CPU capacity of that node.
  • Horizontal Scaling: If a container regularly hits its CPU limits, consider scaling the application horizontally by adding more pods to distribute the load.

8. Conclusion

In a Kubernetes cluster, CPU throttling is primarily a mechanism to enforce resource limits and ensure that each container gets its fair share of CPU time, preventing any single container from monopolizing resources. While this helps maintain system stability and prevent resource exhaustion, it can result in performance degradation if a container is constantly throttled. Proper resource allocation and monitoring are key to avoiding excessive throttling and ensuring efficient operation of your Kubernetes workloads.

Uncategorized

Since June 2022 the Red Hat OpenShift operator index images (redhat/redhat/operator-index) have been served from registry.redhat.io using Quay.io as the backend. OpenShift itself already needs access to the Quay.io registry and CDN hosts as explained in its installation instructions, and so this change required no action from customers at that time.

We are extending this to all Red Hat container images. This allows customers to benefit from the high availability of the Quay.io registry while simplifying the way Red Hat delivers container images and paving the way for future enhancements.

More informaion

Uncategorized

I started a channel on youtube. The first video series will be about software that helps beginners to start using linux and after this series i will start to talk about my jorney learning new things like OpenShift, IBM Cloud, Cloud Paks and related IBM technologies.
I will record the videos in Brazilian Portuguese first and maybe i will create videos in English.
There is no much content about the things i will record in my native language and not everyone here in Brazil are able listen videos in English.
There is another blog too https://www.chuvisco.net.br were you can view the transcription of the video.

Uncategorized