Python Script to Send Log Messages to Kafka

This Python script demonstrates how to send log messages to a Kafka topic using the kafka-python library. The script initializes a Kafka producer, serializes log messages to JSON, and sends them to a specified Kafka topic.

Python Script

from kafka import KafkaProducer
import json
 
# Kafka configuration
# KAFKA_BROKER = 'localhost:9092'
KAFKA_BROKER = 'kafka.example.in:9092'
 
KAFKA_TOPIC = 'project1logs'
# KAFKA_TOPIC = 'djangologs'
 
# Initialize the Kafka producer
producer = KafkaProducer(
    bootstrap_servers=KAFKA_BROKER,
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)
 
# Sample log message
log_message = {
    'level': 'INFO',
    'message': 'This is a sample log message for project1.',
    'timestamp': '2024-07-10T12:00:00'
}
 
# Send the log message to Kafka
producer.send(KAFKA_TOPIC, log_message)
producer.flush()
producer.close()
 
print(f"Log sent to Kafka topic {KAFKA_TOPIC}")

Configuration Details

Kafka Configuration

• KAFKA_BROKER:

  • The address of the Kafka broker. Change this to the appropriate address of your Kafka instance.
  • Example: kafka.example.in:9092

• KAFKA_TOPIC:

  • The Kafka topic to which the log message will be sent. Modify this based on your use case.
  • Example: 'project1logs'

Producer Initialization

• KafkaProducer:
  • bootstrap_servers: Address of the Kafka broker(s).
  • value_serializer: Function to serialize the log message into JSON format. The lambda v: json.dumps(v).encode('utf-8') converts the log message into a JSON string and encodes it to UTF-8.

Log Message

• log_message:
  • A sample log message with fields such as level, message, and timestamp. Adjust this message based on your logging needs.

Sending the Message

• producer.send:
  • Sends the log message to the specified Kafka topic.
• producer.flush:
  • Ensures all messages are sent before closing the producer.
• producer.close:
  • Closes the producer connection.

Usage

1. Install Dependencies: Make sure you have the kafka-python library installed. You can install it using pip:

   pip install kafka-python

2. Run the Script: Execute the Python script to send a log message to Kafka:

   python send_log.py

3. Verify: Check your Kafka topic to ensure that the log message has been sent and is available.

Notes

• Error Handling: For a production script, consider adding error handling and logging to manage any issues with connecting to Kafka or sending messages.
• Security: If your Kafka broker requires authentication or uses SSL, you’ll need to add additional configuration to the KafkaProducer.

This script provides a basic example of sending log messages to Kafka. Adjust the configuration and message format as needed for your specific use case.

Setup Venv :

The error you're encountering is due to PEP 668, which prevents pip from modifying system-managed Python environments (like the one installed via apt). This is a safety feature in newer Debian/Ubuntu systems.

To resolve this cleanly and safely, create and use a virtual environment.

✅ Recommended Steps (Safe & Isolated):

# Step 1: Install Python virtual environment tools (if not already)
sudo apt install python3-venv python3-full -y
 
# Step 2: Create a virtual environment in your project directory
cd /var/www/devops/kafka-set
python3 -m venv .venv
 
# Step 3: Activate the virtual environment
source .venv/bin/activate
 
# Step 4: Install your required package (e.g., kafka-python)
pip install kafka-python

Once done, you can use Python and pip within this virtual environment safely:

python your_script.py  # This will use the venv’s interpreter

🧼 To Deactivate:

deactivate