> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ahmadraza.in/llms.txt
> Use this file to discover all available pages before exploring further.

# KafkaSetup RaftProtocol

# 🐳 **Kafka KRaft (Raft-based) Setup with Docker Compose** 🐳

This guide demonstrates how to set up a **KRaft-based Apache Kafka broker** (no Zookeeper needed!) along with **Kafka UI** for topic/consumer inspection.

***

## 📦 **Docker Compose: Kafka KRaft + Kafka UI**

```yaml theme={null}
services:
  kafka:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"  # Enable auto topic creation default
      KAFKA_NUM_PARTITIONS: 5

    ports:
      - "9094:9094"
    healthcheck:
      test: [
        "CMD-SHELL",
        "./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1"
      ]
      interval: 10s
      timeout: 10s
      retries: 5

## This is kafka UI service to manager topics and consumers
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8088:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
    depends_on:
      kafka:
        condition: service_healthy
```

***

## 🔍 **Key Kafka KRaft Config Explanation**

| Variable                          | Description                                                                                     |
| --------------------------------- | ----------------------------------------------------------------------------------------------- |
| `KAFKA_PROCESS_ROLES`             | Enables both `broker` and `controller` in single node (Raft mode).                              |
| `KAFKA_LISTENERS`                 | Declares three listener endpoints: `PLAINTEXT`, `CONTROLLER`, and external client `EXTERNAL`.   |
| `KAFKA_ADVERTISED_LISTENERS`      | Makes Kafka reachable from other containers (`broker:9092`) and host system (`localhost:9094`). |
| `KAFKA_CONTROLLER_QUORUM_VOTERS`  | Defines Raft controller quorum (use format: `<node_id>@<host>:<port>`).                         |
| `KAFKA_CONTROLLER_LISTENER_NAMES` | Identifies which listener handles controller communication (Raft).                              |
| `KAFKA_TRANSACTION_STATE_LOG_*`   | Required even for a single-broker Kafka with transactions enabled.                              |

***

## 🚀 **Usage Commands**

### 🟢 Start the Kafka Cluster

```bash theme={null}
docker-compose up -d
```

> Kafka will be exposed on:

* `localhost:9094` → External client access
* `localhost:8088` → Kafka UI Dashboard

### 🔴 Stop and Clean Up

```bash theme={null}
docker-compose down
```

***

## 🧪 **Working with Kafka Topics (No Zookeeper)**

### ✅ Create Topics

```bash theme={null}
docker exec -it broker kafka-topics.sh \
  --create --topic test-topic \
  --bootstrap-server localhost:9092 \
  --partitions 1 --replication-factor 1
```

### 📋 List Topics

```bash theme={null}
docker exec -it broker kafka-topics.sh \
  --list --bootstrap-server localhost:9092
```

### 📥 Consume from a Topic

```bash theme={null}
docker exec -it broker kafka-console-consumer.sh \
  --topic test-topic --from-beginning \
  --bootstrap-server localhost:9092
```

### 📤 Produce to a Topic

```bash theme={null}
docker exec -it broker kafka-console-producer.sh \
  --topic test-topic --bootstrap-server localhost:9092
```

***

## 🌐 **Kafka UI Dashboard**

> Access the Kafka dashboard at:
> 👉 [http://localhost:8088](http://localhost:8088)

You can:

* View topics
* Inspect partitions and messages
* See consumer groups
* Produce/consume messages directly from UI

***

## ⚠️ Notes

* **No Zookeeper**: This setup uses Kafka **KRaft mode**, available from Apache Kafka 3.x+, eliminating Zookeeper dependency.
* **Multiple Brokers**: For multi-node Raft quorum, adjust `KAFKA_CONTROLLER_QUORUM_VOTERS` accordingly.
* **Production Security**: Add SSL and authentication configs before exposing Kafka publicly.

***

## 📚 Summary

| Action             | Command                                 |
| ------------------ | --------------------------------------- |
| Start Cluster      | `docker-compose up -d`                  |
| Stop Cluster       | `docker-compose down`                   |
| Create Topic       | `kafka-topics.sh --create ...`          |
| List Topics        | `kafka-topics.sh --list ...`            |
| Consume Logs       | `kafka-console-consumer.sh --topic ...` |
| Kafka Dashboard UI | `http://localhost:8088`                 |

***