Install and Setup Zookeeper + Kafka `Ubuntu`

install php8.2-rdkafka extenstion
install open jdk 8 or 11

sudo apt-get install openjdk-8-jre

download the kafka tar file & extract the tar

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz
tar -xvf kafka_2.12-3.7.0.tgz

visit the dir and run this command to setup kafka locally run on port 9092

cd /var/www/devops/kafka-setup/kafka_2.12-3.7.0/

run zookerper

bin/zookeeper-server-start.sh config/zookeeper.properties

run kafka server oon 9092

bin/kafka-server-start.sh config/server.properties

-run localer terminal and create a topic

bin/kafka-topics.sh --create --topic hdfc-upi-logs --bootstrap-server localhost:9092

show logs in terminal

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hdfc-upi-logs --from-beginning

test server logs , mamnually create a log in kafka to check working

bin/kafka-console-producer.sh --topic hdfc-upi-logs --bootstrap-server localhost:9092

DOCS :

https://kafka.apache.org/downloads (opens in a new tab) https://kafka.apache.org/quickstart (opens in a new tab)

Kafka Public Hosted Service Setup

################################################################################################################## To run Kafka locally on your EC2 instance and make it accessible globally, you can follow these steps:

Install Java and Download Kafka: Make sure you have Java installed and download the Kafka tar file as you've done.
Configure Kafka: Update the server.properties file to make Kafka accessible over the public hostname.
Run Kafka and Zookeeper as Services: To avoid the problem of Kafka and Zookeeper stopping when you close the terminal, run them as services.
Open Ports in Security Group: Ensure the necessary ports are open in your EC2 instance's security group to allow external access.

Here’s a step-by-step guide:

Step 1: Install Java and Download Kafka

You've already done this part, so no changes needed here.

Step 2: Configure Kafka

Modify the server.properties file to include your public IP or DNS name.

Open the server.properties file:

nano /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/server.properties

Add or modify the following lines:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://<your-public-hostname>:9092

Replace <your-public-hostname> with your EC2 instance’s public DNS or IP address.

Step 3: Run Kafka and Zookeeper as Services

Create systemd service files for Zookeeper and Kafka.

Zookeeper Service: Create a file named zookeeper.service:

sudo nano /etc/systemd/system/zookeeper.service

Add the following content:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
After=network.target
 
[Service]
Type=simple
ExecStart=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/zookeeper-server-start.sh /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/zookeeper.properties
ExecStop=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/zookeeper-server-stop.sh
Restart=on-abnormal
 
[Install]
WantedBy=multi-user.target

Kafka Service: Create a file named kafka.service:

sudo nano /etc/systemd/system/kafka.service

Add the following content:

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service
 
[Service]
Type=simple
ExecStart=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-server-start.sh /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/server.properties
ExecStop=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-server-stop.sh
Restart=on-abnormal
 
[Install]
WantedBy=multi-user.target

Step 4: Enable and Start Services

Enable and start the services:

sudo systemctl enable zookeeper
sudo systemctl start zookeeper
 
sudo systemctl enable kafka
sudo systemctl start kafka

Step 5: Open Ports in Security Group

Make sure that your EC2 instance’s security group allows inbound traffic on the required ports (default is 9092 for Kafka and 2181 for Zookeeper).

Go to the EC2 Management Console.
Select your instance and click on the Security Group associated with it.
Edit the inbound rules to add rules for port 9092 (Kafka) and 2181 (Zookeeper).

Example:

Type: Custom TCP Rule
Protocol: TCP
Port Range: 9092
Source: 0.0.0.0/0 (or restrict to specific IP addresses)

Repeat for port 2181.

Verify Installation

You can verify that Kafka and Zookeeper are running:

sudo systemctl status zookeeper
sudo systemctl status kafka

This setup will keep Kafka and Zookeeper running even if you close the terminal, and they will be accessible via your EC2 instance’s public IP or DNS.

###########################################################################3 run as nginx reverse proxy server on the specified host ###########################################################################

To set up Kafka to be accessible through an Nginx reverse proxy, you can follow these steps:

Install Nginx: Install Nginx on your EC2 instance.
Configure Nginx: Set up Nginx as a reverse proxy for Kafka.
Update Kafka Configuration: Adjust Kafka configuration to work properly with Nginx.
Open Ports in Security Group: Ensure the necessary ports are open in your EC2 instance's security group.

Step 1: Install Nginx

First, install Nginx on your EC2 instance if it's not already installed:

sudo apt update
sudo apt install nginx

Step 2: Configure Nginx

Create a new Nginx configuration file for Kafka:

sudo nano /etc/nginx/sites-available/kafka

Add the following configuration to this file:

server {
    listen 80;
    server_name <your-public-hostname>;
 
    location / {
        proxy_pass http://localhost:9092;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace <your-public-hostname> with your EC2 instance’s public DNS or IP address.

Enable the new configuration by creating a symbolic link to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/kafka /etc/nginx/sites-enabled/

Test the Nginx configuration:

sudo nginx -t

If the test is successful, restart Nginx:

sudo systemctl restart nginx

Step 3: Update Kafka Configuration

Modify the server.properties file to ensure Kafka binds to the correct address and advertises the correct hostname:

Open the server.properties file:

nano /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/server.properties

Update the following lines:

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://<your-public-hostname>:80

Replace <your-public-hostname> with your EC2 instance’s public DNS or IP address.

Restart Kafka to apply the changes:

sudo systemctl restart kafka

Step 4: Open Ports in Security Group

Ensure that your EC2 instance’s security group allows inbound traffic on port 80.

Go to the EC2 Management Console.
Select your instance and click on the Security Group associated with it.
Edit the inbound rules to add a rule for port 80.

Example:

Type: HTTP
Protocol: TCP
Port Range: 80
Source: 0.0.0.0/0 (or restrict to specific IP addresses)

Verification

You can verify that Kafka is accessible via Nginx by using a Kafka client to connect to your EC2 instance’s public hostname on port 80. For example:

kafka-console-consumer.sh --bootstrap-server <your-public-hostname>:80 --topic hdfc-upi-logs --from-beginning

This setup ensures that Kafka traffic is proxied through Nginx, making it accessible without directly exposing port 9092.

OTHER METHODS

##########################################################

install php`8.2`-rdkafka extenstion

install open jdk 8 or 11

sudo apt-get install openjdk-8-jre

download the kafka tar file

extract the tar

wget https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz (opens in a new tab) tar -xvf kafka_2.12-3.7.0.tgz

visit the dir and run this command to setup kafka locally run on port 9092

cd /var/www/devops/kafka-setup/kafka_2.12-3.7.0/

run zookerper

bin/zookeeper-server-start.sh config/zookeeper.properties

run kafka server oon 9092

bin/kafka-server-start.sh config/server.properties

run localer terminal and create a topic

bin/kafka-topics.sh --create --topic hdfc-upi-logs --bootstrap-server localhost:9092

show logs in terminal

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic hdfc-upi-logs --from-beginning

test server logs , mamnually create a log in kafka to check working

bin/kafka-console-producer.sh --topic hdfc-upi-logs --bootstrap-server localhost:9092

DOCS :

https://kafka.apache.org/downloads (opens in a new tab) https://kafka.apache.org/quickstart (opens in a new tab)

Kafka Public Hosted Service Setup

Install Java and Download Kafka: Make sure you have Java installed and download the Kafka tar file as you've done.
Configure Kafka: Update the server.properties file to make Kafka accessible over the public hostname.
Run Kafka and Zookeeper as Services: To avoid the problem of Kafka and Zookeeper stopping when you close the terminal, run them as services.
Open Ports in Security Group: Ensure the necessary ports are open in your EC2 instance's security group to allow external access.

Here’s a step-by-step guide:

Step 1: Install Java and Download Kafka

You've already done this part, so no changes needed here.

Step 2: Configure Kafka

Modify the server.properties file to include your public IP or DNS name.

Open the server.properties file:

nano /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/server.properties

Add or modify the following lines:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://<your-public-hostname>:9092

Replace <your-public-hostname> with your EC2 instance’s public DNS or IP address.

Step 3: Run Kafka and Zookeeper as Services

Create systemd service files for Zookeeper and Kafka.

Zookeeper Service: Create a file named zookeeper.service:

sudo nano /etc/systemd/system/zookeeper.service

Add the following content:

[Unit]
Description=Apache Zookeeper server
Documentation=http://zookeeper.apache.org
After=network.target
 
[Service]
Type=simple
ExecStart=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/zookeeper-server-start.sh /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/zookeeper.properties
ExecStop=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/zookeeper-server-stop.sh
Restart=on-abnormal
 
[Install]
WantedBy=multi-user.target

Kafka Service: Create a file named kafka.service:

sudo nano /etc/systemd/system/kafka.service

Add the following content:

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service
 
[Service]
Type=simple
ExecStart=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-server-start.sh /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/server.properties
ExecStop=/var/www/devops/kafka-setup/kafka_2.12-3.7.0/bin/kafka-server-stop.sh
Restart=on-abnormal
 
[Install]
WantedBy=multi-user.target

Step 4: Enable and Start Services

Enable and start the services:

sudo systemctl enable zookeeper
sudo systemctl start zookeeper
 
sudo systemctl enable kafka
sudo systemctl start kafka

Step 5: Open Ports in Security Group

Make sure that your EC2 instance’s security group allows inbound traffic on the required ports (default is 9092 for Kafka and 2181 for Zookeeper).

Go to the EC2 Management Console.
Select your instance and click on the Security Group associated with it.
Edit the inbound rules to add rules for port 9092 (Kafka) and 2181 (Zookeeper).

Example:

Type: Custom TCP Rule
Protocol: TCP
Port Range: 9092
Source: 0.0.0.0/0 (or restrict to specific IP addresses)

Repeat for port 2181.

Verify Installation

You can verify that Kafka and Zookeeper are running:

sudo systemctl status zookeeper
sudo systemctl status kafka

This setup will keep Kafka and Zookeeper running even if you close the terminal, and they will be accessible via your EC2 instance’s public IP or DNS.

To set up Kafka to be accessible through an Nginx reverse proxy, you can follow these steps:

Install Nginx: Install Nginx on your EC2 instance.
Configure Nginx: Set up Nginx as a reverse proxy for Kafka.
Update Kafka Configuration: Adjust Kafka configuration to work properly with Nginx.
Open Ports in Security Group: Ensure the necessary ports are open in your EC2 instance's security group.

Step 1: Install Nginx

First, install Nginx on your EC2 instance if it's not already installed:

sudo apt update
sudo apt install nginx

Step 2: Configure Nginx

Create a new Nginx configuration file for Kafka:

sudo nano /etc/nginx/sites-available/kafka

Add the following configuration to this file:

server {
    listen 80;
    server_name <your-public-hostname>;
 
    location / {
        proxy_pass http://localhost:9092;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace <your-public-hostname> with your EC2 instance’s public DNS or IP address.

Enable the new configuration by creating a symbolic link to the sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/kafka /etc/nginx/sites-enabled/

Test the Nginx configuration:

sudo nginx -t

If the test is successful, restart Nginx:

sudo systemctl restart nginx

Step 3: Update Kafka Configuration

Modify the server.properties file to ensure Kafka binds to the correct address and advertises the correct hostname:

Open the server.properties file:

nano /var/www/devops/kafka-setup/kafka_2.12-3.7.0/config/server.properties

Update the following lines:

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://<your-public-hostname>:80

Replace <your-public-hostname> with your EC2 instance’s public DNS or IP address.

Restart Kafka to apply the changes:

sudo systemctl restart kafka

Step 4: Open Ports in Security Group

Ensure that your EC2 instance’s security group allows inbound traffic on port 80.

Go to the EC2 Management Console.
Select your instance and click on the Security Group associated with it.
Edit the inbound rules to add a rule for port 80.

Example:

Type: HTTP
Protocol: TCP
Port Range: 80
Source: 0.0.0.0/0 (or restrict to specific IP addresses)

Verification

You can verify that Kafka is accessible via Nginx by using a Kafka client to connect to your EC2 instance’s public hostname on port 80. For example:

kafka-console-consumer.sh --bootstrap-server <your-public-hostname>:80 --topic hdfc-upi-logs --from-beginning

This setup ensures that Kafka traffic is proxied through Nginx, making it accessible without directly exposing port 9092.

Goaccess Logging Tool

Install and Setup Zookeeper + Kafka Ubuntu

DOCS :

Kafka Public Hosted Service Setup

Step 1: Install Java and Download Kafka

Step 2: Configure Kafka

Step 3: Run Kafka and Zookeeper as Services

Step 4: Enable and Start Services

Step 5: Open Ports in Security Group

Verify Installation

Step 1: Install Nginx

Step 2: Configure Nginx

Step 3: Update Kafka Configuration

Step 4: Open Ports in Security Group

Verification

OTHER METHODS

install php8.2-rdkafka extenstion

install open jdk 8 or 11

download the kafka tar file

extract the tar

visit the dir and run this command to setup kafka locally run on port 9092

run zookerper

run kafka server oon 9092

run localer terminal and create a topic

show logs in terminal

test server logs , mamnually create a log in kafka to check working

DOCS :

Kafka Public Hosted Service Setup

Step 1: Install Java and Download Kafka

Step 2: Configure Kafka

Step 3: Run Kafka and Zookeeper as Services

Step 4: Enable and Start Services

Step 5: Open Ports in Security Group

Verify Installation

Step 1: Install Nginx

Step 2: Configure Nginx

Step 3: Update Kafka Configuration

Step 4: Open Ports in Security Group

Verification

Install and Setup Zookeeper + Kafka `Ubuntu`

install php`8.2`-rdkafka extenstion