Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.ahmadraza.in/llms.txt

Use this file to discover all available pages before exploring further.

​
🐳 Kafka KRaft (Raft-based) Setup with Docker Compose 🐳

This guide demonstrates how to set up a KRaft-based Apache Kafka broker (no Zookeeper needed!) along with Kafka UI for topic/consumer inspection.

​
📦 Docker Compose: Kafka KRaft + Kafka UI

services:
  kafka:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"  # Enable auto topic creation default
      KAFKA_NUM_PARTITIONS: 5

    ports:
      - "9094:9094"
    healthcheck:
      test: [
        "CMD-SHELL",
        "./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"
      ]
      interval: 10s
      timeout: 10s
      retries: 5

## This is kafka UI service to manager topics and consumers
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8088:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
    depends_on:
      kafka:
        condition: service_healthy

​
🔍 Key Kafka KRaft Config Explanation

VariableDescription
KAFKA_PROCESS_ROLESEnables both broker and controller in single node (Raft mode).
KAFKA_LISTENERSDeclares three listener endpoints: PLAINTEXT, CONTROLLER, and external client EXTERNAL.
KAFKA_ADVERTISED_LISTENERSMakes Kafka reachable from other containers (broker:9092) and host system (localhost:9094).
KAFKA_CONTROLLER_QUORUM_VOTERSDefines Raft controller quorum (use format: <node_id>@<host>:<port>).
KAFKA_CONTROLLER_LISTENER_NAMESIdentifies which listener handles controller communication (Raft).
KAFKA_TRANSACTION_STATE_LOG_*Required even for a single-broker Kafka with transactions enabled.

​
🚀 Usage Commands

​
🟢 Start the Kafka Cluster

docker-compose up -d
Kafka will be exposed on:
  • localhost:9094 → External client access
  • localhost:8088 → Kafka UI Dashboard

​
🔴 Stop and Clean Up

docker-compose down

​
🧪 Working with Kafka Topics (No Zookeeper)

​
✅ Create Topics

docker exec -it broker kafka-topics.sh \
  --create --topic test-topic \
  --bootstrap-server localhost:9092 \
  --partitions 1 --replication-factor 1

​
📋 List Topics

docker exec -it broker kafka-topics.sh \
  --list --bootstrap-server localhost:9092

​
📥 Consume from a Topic

docker exec -it broker kafka-console-consumer.sh \
  --topic test-topic --from-beginning \
  --bootstrap-server localhost:9092

​
📤 Produce to a Topic

docker exec -it broker kafka-console-producer.sh \
  --topic test-topic --bootstrap-server localhost:9092

​
🌐 Kafka UI Dashboard

Access the Kafka dashboard at: 👉 http://localhost:8088
You can:
  • View topics
  • Inspect partitions and messages
  • See consumer groups
  • Produce/consume messages directly from UI

​
⚠️ Notes

  • No Zookeeper: This setup uses Kafka KRaft mode, available from Apache Kafka 3.x+, eliminating Zookeeper dependency.
  • Multiple Brokers: For multi-node Raft quorum, adjust KAFKA_CONTROLLER_QUORUM_VOTERS accordingly.
  • Production Security: Add SSL and authentication configs before exposing Kafka publicly.

​
📚 Summary

ActionCommand
Start Clusterdocker-compose up -d
Stop Clusterdocker-compose down
Create Topickafka-topics.sh --create ...
List Topicskafka-topics.sh --list ...
Consume Logskafka-console-consumer.sh --topic ...
Kafka Dashboard UIhttp://localhost:8088