​

Kafka Setup Guide

​

Running Apache Kafka and Zookeeper on Your Server

​

Table of Contents

1. Prerequisites
2. Installing Java
3. Installing Zookeeper
4. Installing Kafka
5. Configuring Zookeeper
6. Configuring Kafka
7. Setting Up Systemd Services
8. Starting Services
9. Verifying the Installation
10. Troubleshooting

​

Prerequisites

• A server running a Linux-based operating system (e.g., Ubuntu, CentOS).
• Access to the terminal with sudo privileges.
• Basic understanding of terminal commands.

​

Installing Java

1. Update your package index:
   sudo apt update
   
2. Install OpenJDK:
   sudo apt install openjdk-17-jdk
   
3. Verify the installation:
   java -version
   

​

Installing Zookeeper

1. Download Zookeeper:
   wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
   
2. Extract the downloaded file:
   tar -xzf apache-zookeeper-3.8.0-bin.tar.gz
   
3. Move to /opt:
   sudo mv apache-zookeeper-3.8.0-bin /opt/zookeeper
   
4. Create a configuration file:
   sudo nano /opt/zookeeper/conf/zoo.cfg
   
   Add the following configuration:
   tickTime=2000
   dataDir=/var/lib/zookeeper
   clientPort=2181
   maxClientCnxns=60
   
5. Create the data directory:
   sudo mkdir /var/lib/zookeeper
   

​

Installing Kafka

1. Download Kafka:
   wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
   
2. Extract the downloaded file:
   tar -xzf kafka_2.13-3.4.0.tgz
   
3. Move to /opt:
   sudo mv kafka_2.13-3.4.0 /opt/kafka
   

​

Configuring Zookeeper

​

Configuring Kafka

1. Create a configuration file for Kafka:
   sudo nano /opt/kafka/config/server.properties
   
   Add or modify the following configurations:
   broker.id=0
   listeners=PLAINTEXT://:9092
   log.dirs=/var/lib/kafka/logs
   zookeeper.connect=localhost:2181
   
2. Create the logs directory:
   sudo mkdir -p /var/lib/kafka/logs
   

​

Setting Up Systemd Services

​

Create a Zookeeper Service

1. Create the service file:
   sudo nano /etc/systemd/system/zookeeper.service
   
2. Add the following content:
   [Unit]
   Description=Apache Zookeeper
   After=network.target
   
   [Service]
   User=your_username
   ExecStart=/opt/zookeeper/bin/zkServer.sh start /opt/zookeeper/conf/zoo.cfg
   ExecStop=/opt/zookeeper/bin/zkServer.sh stop
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target
   

​

Create a Kafka Service

1. Create the service file:
   sudo nano /etc/systemd/system/kafka.service
   
2. Add the following content:
   [Unit]
   Description=Apache Kafka
   After=zookeeper.service
   
   [Service]
   User=your_username
   ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
   ExecStop=/opt/kafka/bin/kafka-server-stop.sh
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target
   

​

Starting Services

1. Reload systemd to recognize the new services:
   sudo systemctl daemon-reload
   
2. Start Zookeeper:
   sudo systemctl start zookeeper
   
3. Start Kafka:
   sudo systemctl start kafka
   
4. Enable both services to start on boot:
   sudo systemctl enable zookeeper
   sudo systemctl enable kafka
   

​

Verifying the Installation

1. Check the status of Zookeeper:
   sudo systemctl status zookeeper
   
2. Check the status of Kafka:
   sudo systemctl status kafka
   
3. Create a test topic:
   /opt/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
   
4. List topics to confirm:
   /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server localhost:9092
   

​

Troubleshooting

• Check logs if Kafka or Zookeeper fail to start:
  • Zookeeper logs: /opt/zookeeper/logs/zookeeper.out
  • Kafka logs: /opt/kafka/logs/server.log
• Ensure the correct version of Java is installed.
• Verify that no other services are using the same ports (2181 for Zookeeper, 9092 for Kafka).

​

Conclusion

​

Directory Structure

kafka_2.12-3.7.0
├── bin                  # Contains scripts to run Kafka and Zookeeper
├── config               # Configuration files for Kafka and Zookeeper
├── kafka-consumer.py    # Python consumer script
├── kafka-logger.py      # Python logger script
└── libs                 # Required libraries and dependencies

​

Key Directories and Files

• bin/: Scripts to start, stop, and manage Kafka and Zookeeper instances.
• config/: Contains properties files for configuring Kafka and its components.
• libs/: Contains all the necessary jar files required for Kafka to function.

​

Prerequisites

• Java: Kafka requires Java 8 or later. Ensure that it is installed and the JAVA_HOME environment variable is set correctly.
• Zookeeper: Kafka uses Zookeeper for cluster management. It can be run separately or within Kafka.

​

Installation Steps

1. Download Kafka:
   • Download the Kafka binary from the Apache Kafka website.
   • Extract the downloaded tar.gz file to your desired installation directory.
2. Verify Installation:
   • Navigate to the Kafka directory:
     cd kafka_2.12-3.7.0
     
   • Note : Tested Kafka Version is kafka_2.12-3.7.0 ( https://archive.apache.org/dist/kafka/3.7.0/kafka_2.12-3.7.0.tgz )
   • Check if the bin directory contains all necessary scripts.

​

Configuration Files

• server.properties: Configuration for Kafka brokers.
• zookeeper.properties: Configuration for Zookeeper.
• connect-distributed.properties: Configuration for running Kafka Connect in distributed mode.

​

Example of server.properties

# Basic Kafka server configuration
broker.id=0
listeners=PLAINTEXT://localhost:9092
log.dirs=/tmp/kafka-logs
num.partitions=1

​

Example of zookeeper.properties

# Basic Zookeeper configuration
tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181

​

Starting Kafka

1. Start Zookeeper:
   • Use the following command to start Zookeeper:
     bin/zookeeper-server-start.sh config/zookeeper.properties
     
2. Start Kafka Server:
   • Once Zookeeper is running, start the Kafka server:
     bin/kafka-server-start.sh config/server.properties
     

​

Using Kafka Command-Line Tools

• Create a Topic:
  bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
  
• List Topics:
  bin/kafka-topics.sh --list --bootstrap-server localhost:9092
  
• Produce Messages:
  bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092
  
• Consume Messages:
  bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092
  

cd /var/www/devops/kafka-setup/kafka_2.12-3.7.0/

# run zookerper
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/zookeeper-server-start.sh config/zookeeper.properties

# run kafka server oon 9092
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-server-start.sh config/server.properties

# run localer terminal and create a topic 
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-topics.sh --create --topic kafka-logs-topic1 --bootstrap-server localhost:9092

# show logs in terminal 
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic kafka-logs-topic1 --from-beginning

# test server logs , mamnually create a log in kafka to check working 
/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-console-producer.sh --topic kafka-logs-topic1 --bootstrap-server localhost:9092

​

Python Integration

​

Example of Using kafka-consumer.py

pip install kafka-python

python kafka-consumer.py

​

Conclusion

For troubleshooting, please check the logs located in the /tmp/kafka-logs directory or adjust the logging settings in the log4j.properties file found in the config/ directory.