Kubernetes HPA Testing with Minikube and K6

This documentation outlines the process of testing Kubernetes Horizontal Pod Autoscalers (HPA) in a Minikube environment. We’ll use stress simulations and the K6 load testing tool to evaluate the scaling behavior of a PHP application.

Prerequisites

Minikube installed and running.
kubectl CLI tool configured for Minikube.
K6 installed for load testing.
Basic knowledge of Kubernetes deployments, services, and HPA.

Setup Steps

1. Deployment Configuration

Create a Kubernetes deployment for the stress-testing PHP application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress-php-deployment
  labels:
    app: stress-php
spec:
  replicas: 2
  selector:
    matchLabels:
      app: stress-php
  template:
    metadata:
      labels:
        app: stress-php
    spec:
      containers:
      - name: stress-php-container
        image: noscopev6/web-stress-php:v0.2
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: "100m"
            memory: "200Mi"
          limits:
            cpu: "200m"
            memory: "400Mi"
        livenessProbe:
          httpGet:
            path: /index.php
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 20
          timeoutSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /index.php
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 3
        startupProbe:
          httpGet:
            path: /index.php
            port: 80
          initialDelaySeconds: 20
          periodSeconds: 30
          timeoutSeconds: 5
          failureThreshold: 5

Apply the deployment:

kubectl apply -f deployment.yaml

2. Service Configuration

Expose the application using a NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: stress-php-service
spec:
  selector:
    app: stress-php
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30000  # Minikube NodePort range is 30000-32767
  type: NodePort
  # - protocol: TCP
  #   port: 80
  #   targetPort: 80
  # type: ClusterIP

Apply the service:

kubectl apply -f service.yaml

3. Horizontal Pod Autoscaler

Configure an HPA to scale pods based on CPU and memory utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: stress-php-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: stress-php-deployment
  minReplicas: 2
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50  # Target 50% CPU usage
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 50  # Target 70% Memory usage

Apply the HPA:

kubectl apply -f hpa.yaml

Stress Testing the Application

0. Health Check URL

Use the following PHP script to simulate HealthCheck tasks:

<?php
echo "PHP + NGINX is working!";

1. CPU Stress Test

Use the following PHP script to simulate CPU-intensive tasks:

<?php
// cpu_stress.php
ini_set('max_execution_time', 0); // Disable time limit for stress test

echo "Starting CPU stress test...\n";

// Simulate CPU-intensive task by running a loop that performs complex calculations
for ($i = 0; $i < 10000000; $i++) {
    $x = sqrt($i * rand(1, 100));
}

echo "CPU stress test completed.";
?>

Place this script in your application directory and access it via the service NodePort.

2. Memory Stress Test

Use the following PHP script to simulate memory-intensive tasks:

<?php
session_start();

ini_set('max_execution_time', 0);  // Disable time limit for stress test
ini_set('memory_limit', '-1');     // Set memory limit to unlimited

// Check if session variable 'memory' exists, if not, initialize it
if (!isset($_SESSION['memory'])) {
    $_SESSION['memory'] = 0;
}

echo "Starting memory stress test...\n";

// Simulate memory-intensive task by creating large arrays
$memoryHog = [];

// Increase memory usage each time the page is hit
$memoryIncrement = 1024 * 1024;  // 1MB per hit
$memoryTarget = $_SESSION['memory'] + $memoryIncrement;

while ($_SESSION['memory'] < $memoryTarget) {
    // Add large arrays to memory
    $memoryHog[] = str_repeat('A', 1024 * 1024); // Add 1MB strings
    $_SESSION['memory'] += $memoryIncrement;  // Track total memory used
    echo "Allocated " . $_SESSION['memory'] / (1024 * 1024) . "MB of memory...\n";
    flush();  // Flush the output to the browser immediately
    sleep(1);  // Sleep to prevent a complete hang
}

echo "Memory stress test completed.";
$_SESSION['memory'] = 0;  // Reset memory after completion
?>

3. Load Testing with K6

Use K6 to simulate traffic and trigger the HPA:

Example K6 Script for CPU scaling and memory:

CPU scaling

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  // Replace with a simple endpoint in your Laravel app
  http.get('http://192.168.49.2:30000/cpu_stress.php');
  sleep(1); // Pause for 1 second before the next request
}

Memory scaling

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  // Replace with a simple endpoint in your Laravel app
  http.get('http://192.168.49.2:30000/memory_stress.php');
  sleep(1); // Pause for 1 second before the next request
}

Replace <NodeIP> with the Minikube IP address:

minikube ip

Run the K6 test:

k6 run --vus 100 --duration 300s cpu-k6.js # vus is total no of virtual users
k6 run --vus 100 --duration 300s memory-k6.js # vus is total no of virtual users

Observing HPA Behavior

Monitor the pods:
kubectl get pods -w
Check the HPA status:
kubectl get hpa

Check the metrics status:

 kubectl top pod
 kubectl get deploy 
 kubectl top node

You should see the HPA scaling pods up and down based on the simulated traffic and resource utilization.

This testing demonstrates how Kubernetes HPA can dynamically scale your application to handle fluctuating workloads efficiently. Using Minikube and K6 provides a lightweight and effective environment for experimenting with these features.

Proof of Concepts

AWS S3 Storage

ElasticSearch POCs

Kibana Documentation

Code Examples

K8s HPA Testing

JSON Logger

K8s Log Generator

K8s Node Stress

Info

Kubernetes HPA Testing with Minikube and K6

Prerequisites

Setup Steps

1. Deployment Configuration

2. Service Configuration

3. Horizontal Pod Autoscaler

Stress Testing the Application

0. Health Check URL

1. CPU Stress Test

2. Memory Stress Test

3. Load Testing with K6

Example K6 Script for CPU scaling and memory:

Observing HPA Behavior

​Kubernetes HPA Testing with Minikube and K6

​Prerequisites

​Setup Steps

​1. Deployment Configuration

​2. Service Configuration

​3. Horizontal Pod Autoscaler

​Stress Testing the Application

​0. Health Check URL

​1. CPU Stress Test

​2. Memory Stress Test

​3. Load Testing with K6

​Example K6 Script for CPU scaling and memory:

​Observing HPA Behavior

Kubernetes HPA Testing with Minikube and K6

Prerequisites

Setup Steps

1. Deployment Configuration

2. Service Configuration

3. Horizontal Pod Autoscaler

Stress Testing the Application

0. Health Check URL

1. CPU Stress Test

2. Memory Stress Test

3. Load Testing with K6

Example K6 Script for CPU scaling and memory:

Observing HPA Behavior