> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ahmadraza.in/llms.txt
> Use this file to discover all available pages before exploring further.

# KafkaAsDockerCompose

# 🐳 **Docker Compose Setup `NGINX` Configuration for Kafka and Zookeeper** 🐳

This `docker-compose.yml` file sets up a Kafka and Zookeeper cluster using Confluent's Docker images. It configures the necessary environment variables and port mappings for the services.

## **Docker Compose File**

```yaml theme={null}
version: '3.8'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.3
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.4.3
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      # KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092  # use only if you are connecting other conatiner to this kafka 
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"  # Enable auto topic creation default
      KAFKA_NUM_PARTITIONS: 5
    # Command to create a default topic when the container starts
    # command: [bash, -c, "echo 'auto.create.topics.enable=true' >> /etc/kafka/server.properties && /etc/confluent/docker/run"]
```

## **Configuration Details**

### **Zookeeper Service**

* **`image`**: `confluentinc/cp-zookeeper:latest`
  * The Docker image for Zookeeper provided by Confluent.

* **`environment`**:
  * **`ZOOKEEPER_CLIENT_PORT`**: `2181`
    * Port on which Zookeeper listens for client connections.
  * **`ZOOKEEPER_TICK_TIME`**: `2000`
    * The basic time unit in milliseconds used by Zookeeper.

* **`ports`**:
  * **`"2181:2181"`**
    * Maps port 2181 on the host to port 2181 in the container.

### **Kafka Service**

* **`image`**: `confluentinc/cp-kafka:latest`
  * The Docker image for Kafka provided by Confluent.

* **`depends_on`**:
  * **`zookeeper`**
    * Ensures that the Kafka service starts only after the Zookeeper service is up.

* **`ports`**:
  * **`"9092:9092"`**
    * Maps port 9092 on the host to port 9092 in the container.

* **`environment`**:
  * **`KAFKA_BROKER_ID`**: `1`
    * Unique identifier for the Kafka broker.
  * **`KAFKA_ZOOKEEPER_CONNECT`**: `zookeeper:2181`
    * Connection string for Zookeeper.
  * **`KAFKA_ADVERTISED_LISTENERS`**: `PLAINTEXT://kafka:9092`
    * Advertised address for Kafka brokers that clients use to connect.
  * **`KAFKA_LISTENER_SECURITY_PROTOCOL_MAP`**: `PLAINTEXT:PLAINTEXT`
    * Security protocol mapping for listeners.
  * **`KAFKA_INTER_BROKER_LISTENER_NAME`**: `PLAINTEXT`
    * Listener name for inter-broker communication.
  * **`KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR`**: `1`
    * Replication factor for the offsets topic.
  * **`KAFKA_AUTO_CREATE_TOPICS_ENABLE`**: `"true"`
    * Enable automatic creation of topics when they are referenced.

* **`command`** (Commented Out):
  * The command section is commented out but can be used to append configuration settings or run additional scripts when the container starts.

## **Usage**

1. **Start Services**:
   ```bash theme={null}
   docker-compose up -d
   ```

2. **Stop Services**:
   ```bash theme={null}
   docker-compose down
   ```

## **Notes**

* **Network Configuration**: Ensure that your Docker network settings allow communication between the Kafka and Zookeeper containers.
* **Kafka Topics**: By default, Kafka topics will be auto-created due to the `KAFKA_AUTO_CREATE_TOPICS_ENABLE` setting. Adjust this setting as needed based on your use case.

***

# **NGINX Configuration for Kafka Reverse Proxy**

This NGINX configuration sets up a reverse proxy for a Kafka server. It routes incoming HTTP requests to the Kafka server running on `localhost:9092`.

## **NGINX Configuration File**

```nginx theme={null}
server {
    server_name your-domain.com;

    location / {
        proxy_pass http://localhost:9092;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

## **Configuration Details**

### **`server` Block**

* **`server_name`**:
  * Specifies the domain name or IP address that this server block should respond to. Replace `your-domain.com` with your actual domain or server IP.

### **`location /` Block**

* **`proxy_pass`**:
  * Forwards incoming requests to the specified backend server. In this case, it routes requests to `http://localhost:9092`, where your Kafka server is running.

* **`proxy_set_header` Directives**:
  * **`Host`**: Sets the `Host` header in the request to the original host header from the client.
  * **`X-Real-IP`**: Sets the `X-Real-IP` header with the IP address of the client making the request.
  * **`X-Forwarded-For`**: Sets the `X-Forwarded-For` header to include the client's IP address, helping maintain the client's IP address through the proxy chain.
  * **`X-Forwarded-Proto`**: Sets the `X-Forwarded-Proto` header to indicate the protocol used by the client (HTTP or HTTPS).

## **Usage**

1. **Install NGINX**:
   Make sure NGINX is installed on your server. You can install it using your package manager. For example, on Ubuntu:
   ```bash theme={null}
   sudo apt-get update
   sudo apt-get install nginx
   ```

2. **Save Configuration**:
   Save the above configuration in an NGINX configuration file. Typically, this file is located at `/etc/nginx/sites-available/your-config-file` or `/etc/nginx/nginx.conf`.

3. **Create Symbolic Link**:
   Create a symbolic link to enable the configuration:
   ```bash theme={null}
   sudo ln -s /etc/nginx/sites-available/your-config-file /etc/nginx/sites-enabled/
   ```

4. **Test Configuration**:
   Test the NGINX configuration to ensure there are no syntax errors:
   ```bash theme={null}
   sudo nginx -t
   ```

5. **Reload NGINX**:
   Apply the new configuration by reloading NGINX:
   ```bash theme={null}
   sudo systemctl reload nginx
   ```

## **Notes**

* **Security**: Ensure your Kafka server is properly secured and not exposed to the public internet unless absolutely necessary. Consider using authentication and encryption (e.g., SSL/TLS) for secure communication.
* **Testing**: Verify that the proxy is working correctly by accessing your domain and checking if the requests are being forwarded to the Kafka server.

> This configuration allows you to use NGINX as a reverse proxy for your Kafka server, making it accessible through a specified domain and handling request forwarding and header management.

***

# 🐳 **Working with Kafka Topics in Docker** 🐳

This guide will help you manage Kafka topics and consume logs within a Docker environment.

***

## **1. Accessing the Kafka Container**

To start, you need to access the Kafka container's shell:

```bash theme={null}
docker exec -it <kafka_container_id> /bin/bash
```

Replace `<kafka_container_id>` with your Kafka container ID or name.

***

## **2. Creating Kafka Topics**

Once inside the Kafka container, you can create new topics using the `kafka-topics` command:

```bash theme={null}
kafka-topics --create --topic project1_logs --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-topics --create --topic project2_logs --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
kafka-topics --create --topic project3_logs --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
```

In these commands:

* `--topic` specifies the topic name.
* `--bootstrap-server` specifies the Kafka broker address.
* `--partitions` defines the number of partitions.
* `--replication-factor` sets the replication factor.

***

## **3. Listing Kafka Topics**

To view a list of all Kafka topics:

```bash theme={null}
kafka-topics --list --bootstrap-server localhost:9092
```

***

## **4. Consuming Logs from a Kafka Topic**

To see logs from a specific Kafka topic, use the following command:

```bash theme={null}
docker exec <kafka-container-id> kafka-console-consumer --topic project1_logs --bootstrap-server localhost:9092 --from-beginning
```

Alternatively, if you're using Docker Compose:

```bash theme={null}
docker-compose exec kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic project1_logs --from-beginning
```

In these commands:

* `--topic` specifies the topic you want to consume logs from.
* `--from-beginning` tells Kafka to start reading from the beginning of the topic.

***

## **5. Kafka Logger Python Script**

To create logs in a Kafka topic using a Python script: [View](/DEVOPS/Logging/Kafka-Zookerper/Kafka-logger)

***

## **6. Kafka Consumer Python Script**

To consume logs from a Kafka topic using another Python script: [View](/DEVOPS/Logging/Kafka-Zookerper/Kafka-consumer)

***

## **Summary**

* **Access Kafka Container**: Use `docker exec -it <kafka_container_id> /bin/bash`.
* **Create Topics**: Use `kafka-topics --create`.
* **List Topics**: Use `kafka-topics --list`.
* **Consume Logs**: Use `kafka-console-consumer`.
* **Python Scripts**: Use `kafka-logger.py` for producing and `consumer.py` for consuming logs.

***

> This documentation should assist you in effectively managing Kafka topics within your Docker environment.