> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ahmadraza.in/llms.txt
> Use this file to discover all available pages before exploring further.

# KafkaSelfHost VM

# Kafka Setup Guide

This guide provides instructions for setting up Apache Kafka (version 3.7.0) on your system. The setup includes necessary directories, binaries, configurations, and libraries for running Kafka and its components effectively.

# Running Apache Kafka and Zookeeper on Your Server

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Installing Java](#installing-java)
3. [Installing Zookeeper](#installing-zookeeper)
4. [Installing Kafka](#installing-kafka)
5. [Configuring Zookeeper](#configuring-zookeeper)
6. [Configuring Kafka](#configuring-kafka)
7. [Setting Up Systemd Services](#setting-up-systemd-services)
8. [Starting Services](#starting-services)
9. [Verifying the Installation](#verifying-the-installation)
10. [Troubleshooting](#troubleshooting)

## Prerequisites

* A server running a Linux-based operating system (e.g., Ubuntu, CentOS).
* Access to the terminal with `sudo` privileges.
* Basic understanding of terminal commands.

## Installing Java

Apache Kafka requires Java to run. Make sure you have Java installed:

1. **Update your package index:**

   ```bash theme={null}
   sudo apt update
   ```

2. **Install OpenJDK:**

   ```bash theme={null}
   sudo apt install openjdk-17-jdk
   ```

3. **Verify the installation:**

   ```bash theme={null}
   java -version
   ```

## Installing Zookeeper

Zookeeper is required for managing Kafka brokers. Here’s how to install it:

1. **Download Zookeeper:**

   ```bash theme={null}
   wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
   ```

2. **Extract the downloaded file:**

   ```bash theme={null}
   tar -xzf apache-zookeeper-3.8.0-bin.tar.gz
   ```

3. **Move to `/opt`:**

   ```bash theme={null}
   sudo mv apache-zookeeper-3.8.0-bin /opt/zookeeper
   ```

4. **Create a configuration file:**

   ```bash theme={null}
   sudo nano /opt/zookeeper/conf/zoo.cfg
   ```

   Add the following configuration:

   ```ini theme={null}
   tickTime=2000
   dataDir=/var/lib/zookeeper
   clientPort=2181
   maxClientCnxns=60
   ```

5. **Create the data directory:**

   ```bash theme={null}
   sudo mkdir /var/lib/zookeeper
   ```

## Installing Kafka

1. **Download Kafka:**

   ```bash theme={null}
   wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
   ```

2. **Extract the downloaded file:**

   ```bash theme={null}
   tar -xzf kafka_2.13-3.4.0.tgz
   ```

3. **Move to `/opt`:**

   ```bash theme={null}
   sudo mv kafka_2.13-3.4.0 /opt/kafka
   ```

## Configuring Zookeeper

Make sure Zookeeper is configured correctly. The configuration file was created earlier. You may edit it if needed.

## Configuring Kafka

1. **Create a configuration file for Kafka:**

   ```bash theme={null}
   sudo nano /opt/kafka/config/server.properties
   ```

   Add or modify the following configurations:

   ```ini theme={null}
   broker.id=0
   listeners=PLAINTEXT://:9092
   log.dirs=/var/lib/kafka/logs
   zookeeper.connect=localhost:2181
   ```

2. **Create the logs directory:**

   ```bash theme={null}
   sudo mkdir -p /var/lib/kafka/logs
   ```

## Setting Up Systemd Services

### Create a Zookeeper Service

1. **Create the service file:**

   ```bash theme={null}
   sudo nano /etc/systemd/system/zookeeper.service
   ```

2. **Add the following content:**

   ```ini theme={null}
   [Unit]
   Description=Apache Zookeeper
   After=network.target

   [Service]
   User=your_username
   ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
   ExecStop=/opt/zookeeper/bin/zkServer.sh stop
   Restart=on-failure

   [Install]
   WantedBy=multi-user.target
   ```

### Create a Kafka Service

1. **Create the service file:**

   ```bash theme={null}
   sudo nano /etc/systemd/system/kafka.service
   ```

2. **Add the following content:**

   ```ini theme={null}
   [Unit]
   Description=Apache Kafka
   After=zookeeper.service

   [Service]
   User=your_username
   ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
   ExecStop=/opt/kafka/bin/kafka-server-stop.sh
   Restart=on-failure

   [Install]
   WantedBy=multi-user.target
   ```

## Starting Services

1. **Reload systemd to recognize the new services:**

   ```bash theme={null}
   sudo systemctl daemon-reload
   ```

2. **Start Zookeeper:**

   ```bash theme={null}
   sudo systemctl start zookeeper
   ```

3. **Start Kafka:**

   ```bash theme={null}
   sudo systemctl start kafka
   ```

4. **Enable both services to start on boot:**

   ```bash theme={null}
   sudo systemctl enable zookeeper
   sudo systemctl enable kafka
   ```

## Verifying the Installation

1. **Check the status of Zookeeper:**

   ```bash theme={null}
   sudo systemctl status zookeeper
   ```

2. **Check the status of Kafka:**

   ```bash theme={null}
   sudo systemctl status kafka
   ```

3. **Create a test topic:**

   ```bash theme={null}
   /opt/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
   ```

4. **List topics to confirm:**

   ```bash theme={null}
   /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
   ```

## Troubleshooting

* **Check logs** if Kafka or Zookeeper fail to start:
  * Zookeeper logs: `/opt/zookeeper/logs/zookeeper.out`
  * Kafka logs: `/opt/kafka/logs/server.log`
* **Ensure the correct version of Java** is installed.
* **Verify that no other services** are using the same ports (2181 for Zookeeper, 9092 for Kafka).

## Conclusion

You have successfully installed and configured Apache Kafka and Zookeeper on your server. Both services are set up to run automatically at startup, ensuring reliable operation. You can now start producing and consuming messages using Kafka.

***

## Directory Structure

The following is the directory structure for the Kafka installation:

```
kafka_2.12-3.7.0
├── bin                  # Contains scripts to run Kafka and Zookeeper
├── config               # Configuration files for Kafka and Zookeeper
├── kafka-consumer.py    # Python consumer script
├── kafka-logger.py      # Python logger script
└── libs                 # Required libraries and dependencies
```

### Key Directories and Files

* **bin/**: Scripts to start, stop, and manage Kafka and Zookeeper instances.
* **config/**: Contains properties files for configuring Kafka and its components.
* **libs/**: Contains all the necessary jar files required for Kafka to function.

***

## Prerequisites

Before installing Kafka, ensure that you have the following prerequisites:

* **Java**: Kafka requires Java 8 or later. Ensure that it is installed and the `JAVA_HOME` environment variable is set correctly.
* **Zookeeper**: Kafka uses Zookeeper for cluster management. It can be run separately or within Kafka.

***

## Installation Steps

1. **Download Kafka**:
   * Download the Kafka binary from the [Apache Kafka website](https://kafka.apache.org/downloads).
   * Extract the downloaded tar.gz file to your desired installation directory.

2. **Verify Installation**:
   * Navigate to the Kafka directory:
     ```bash theme={null}
     cd kafka_2.12-3.7.0
     ```
   * Note : Tested Kafka Version is `kafka_2.12-3.7.0` ( [https://archive.apache.org/dist/kafka/3.7.0/kafka\_2.12-3.7.0.tgz](https://archive.apache.org/dist/kafka/3.7.0/kafka_2.12-3.7.0.tgz) )
   * Check if the `bin` directory contains all necessary scripts.

***

## Configuration Files

The configuration files are located in the `config/` directory. Key files include:

* **server.properties**: Configuration for Kafka brokers.
* **zookeeper.properties**: Configuration for Zookeeper.
* **connect-distributed.properties**: Configuration for running Kafka Connect in distributed mode.

### Example of `server.properties`

```properties theme={null}
# Basic Kafka server configuration
broker.id=0
listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
```

### Example of `zookeeper.properties`

```properties theme={null}
# Basic Zookeeper configuration
tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181
```

***

## Starting Kafka

1. **Start Zookeeper**:
   * Use the following command to start Zookeeper:
     ```bash theme={null}
     bin/zookeeper-server-start.sh config/zookeeper.properties
     ```

2. **Start Kafka Server**:
   * Once Zookeeper is running, start the Kafka server:
     ```bash theme={null}
     bin/kafka-server-start.sh config/server.properties
     ```

***

## Using Kafka Command-Line Tools

Kafka provides various command-line tools located in the `bin/` directory. Here are some commonly used commands:

* **Create a Topic**:
  ```bash theme={null}
  bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
  ```

* **List Topics**:
  ```bash theme={null}
  bin/kafka-topics.sh --list --bootstrap-server localhost:9092
  ```

* **Produce Messages**:
  ```bash theme={null}
  bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092
  ```

* **Consume Messages**:
  ```bash theme={null}
  bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092
  ```

```bash theme={null}
cd /var/www/devops/kafka-setup/kafka_2.12-3.7.0/

# run zookerper
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/zookeeper-server-start.sh config/zookeeper.properties

# run kafka server oon 9092
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-server-start.sh config/server.properties

# run localer terminal and create a topic 
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-topics.sh --create --topic kafka-logs-topic1 --bootstrap-server localhost:9092

# show logs in terminal 
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka-logs-topic1 --from-beginning

# test server logs , mamnually create a log in kafka to check working 
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-console-producer.sh --topic kafka-logs-topic1 --bootstrap-server localhost:9092
```

***

## Python Integration

For Python integration, the `kafka-consumer.py` and `kafka-logger.py` scripts can be used. These scripts demonstrate how to interact with Kafka using Python.

### Example of Using `kafka-consumer.py`

Make sure you have the `kafka-python` library installed:

```bash theme={null}
pip install kafka-python
```

Run the consumer script:

```bash theme={null}
python kafka-consumer.py
```

***

## Conclusion

This guide provides a basic setup for Apache Kafka, covering installation, configuration, and usage of command-line tools. For advanced configurations and deployments, refer to the [official Kafka documentation](https://kafka.apache.org/documentation/).

> For troubleshooting, please check the logs located in the `/tmp/kafka-logs` directory or adjust the logging settings in the `log4j.properties` file found in the `config/` directory.