🐳 Kafka KRaft (Raft-based) Setup with Docker Compose 🐳

This guide demonstrates how to set up a KRaft-based Apache Kafka broker (no Zookeeper needed!) along with Kafka UI for topic/consumer inspection.

📦 Docker Compose: Kafka KRaft + Kafka UI

services:
  kafka:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"  # Enable auto topic creation default
      KAFKA_NUM_PARTITIONS: 5
 
    ports:
      - "9094:9094"
    healthcheck:
      test: [
        "CMD-SHELL",
        "./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"
      ]
      interval: 10s
      timeout: 10s
      retries: 5
 
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8088:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
    depends_on:
      kafka:
        condition: service_healthy

🔍 Key Kafka KRaft Config Explanation

Variable	Description
`KAFKA_PROCESS_ROLES`	Enables both `broker` and `controller` in single node (Raft mode).
`KAFKA_LISTENERS`	Declares three listener endpoints: `PLAINTEXT`, `CONTROLLER`, and external client `EXTERNAL`.
`KAFKA_ADVERTISED_LISTENERS`	Makes Kafka reachable from other containers (`broker:9092`) and host system (`localhost:9094`).
`KAFKA_CONTROLLER_QUORUM_VOTERS`	Defines Raft controller quorum (use format: `<node_id>@<host>:<port>`).
`KAFKA_CONTROLLER_LISTENER_NAMES`	Identifies which listener handles controller communication (Raft).
`KAFKA_TRANSACTION_STATE_LOG_*`	Required even for a single-broker Kafka with transactions enabled.

🚀 Usage Commands

🟢 Start the Kafka Cluster

docker-compose up -d

Kafka will be exposed on:

localhost:9094 → External client access
localhost:8088 → Kafka UI Dashboard

🔴 Stop and Clean Up

docker-compose down

🧪 Working with Kafka Topics (No Zookeeper)

✅ Create Topics

docker exec -it broker kafka-topics.sh \
  --create --topic test-topic \
  --bootstrap-server localhost:9092 \
  --partitions 1 --replication-factor 1

📋 List Topics

docker exec -it broker kafka-topics.sh \
  --list --bootstrap-server localhost:9092

📥 Consume from a Topic

docker exec -it broker kafka-console-consumer.sh \
  --topic test-topic --from-beginning \
  --bootstrap-server localhost:9092

📤 Produce to a Topic

docker exec -it broker kafka-console-producer.sh \
  --topic test-topic --bootstrap-server localhost:9092

🌐 Kafka UI Dashboard

Access the Kafka dashboard at: 👉 http://localhost:8088 (opens in a new tab)

You can:

View topics
Inspect partitions and messages
See consumer groups
Produce/consume messages directly from UI

⚠️ Notes

No Zookeeper: This setup uses Kafka KRaft mode, available from Apache Kafka 3.x+, eliminating Zookeeper dependency.
Multiple Brokers: For multi-node Raft quorum, adjust KAFKA_CONTROLLER_QUORUM_VOTERS accordingly.
Production Security: Add SSL and authentication configs before exposing Kafka publicly.

📚 Summary

Action	Command
Start Cluster	`docker-compose up -d`
Stop Cluster	`docker-compose down`
Create Topic	`kafka-topics.sh --create ...`
List Topics	`kafka-topics.sh --list ...`
Consume Logs	`kafka-console-consumer.sh --topic ...`
Kafka Dashboard UI	`http://localhost:8088`

Kafka Logger Ksetup Zookeeper