Skip to main content

​
Python Script to Send Log Messages to Kafka

This Python script demonstrates how to send log messages to a Kafka topic using the kafka-python library. The script initializes a Kafka producer, serializes log messages to JSON, and sends them to a specified Kafka topic.

​
Python Script

from kafka import KafkaProducer
import json

# Kafka configuration
# KAFKA_BROKER = 'localhost:9092'
KAFKA_BROKER = 'kafka.example.in:9092'

KAFKA_TOPIC = 'project1logs'
# KAFKA_TOPIC = 'djangologs'

# Initialize the Kafka producer
producer = KafkaProducer(
    bootstrap_servers=KAFKA_BROKER,
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Sample log message
log_message = {
    'level': 'INFO',
    'message': 'This is a sample log message for project1.',
    'timestamp': '2024-07-10T12:00:00'
}

# Send the log message to Kafka
producer.send(KAFKA_TOPIC, log_message)
producer.flush()
producer.close()

print(f"Log sent to Kafka topic {KAFKA_TOPIC}")

​
Configuration Details

​
Kafka Configuration

  • KAFKA_BROKER:
    • The address of the Kafka broker. Change this to the appropriate address of your Kafka instance.
    • Example: kafka.example.in:9092
  • KAFKA_TOPIC:
    • The Kafka topic to which the log message will be sent. Modify this based on your use case.
    • Example: 'project1logs'

​
Producer Initialization

  • KafkaProducer:
    • bootstrap_servers: Address of the Kafka broker(s).
    • value_serializer: Function to serialize the log message into JSON format. The lambda v: json.dumps(v).encode('utf-8') converts the log message into a JSON string and encodes it to UTF-8.

​
Log Message

  • log_message:
    • A sample log message with fields such as level, message, and timestamp. Adjust this message based on your logging needs.

​
Sending the Message

  • producer.send:
    • Sends the log message to the specified Kafka topic.
  • producer.flush:
    • Ensures all messages are sent before closing the producer.
  • producer.close:
    • Closes the producer connection.

​
Usage

  1. Install Dependencies: Make sure you have the kafka-python library installed. You can install it using pip: 
    pip install kafka-python
  2. Run the Script: Execute the Python script to send a log message to Kafka: 
    python send_log.py
  3. Verify: Check your Kafka topic to ensure that the log message has been sent and is available.

​
Notes

  • Error Handling: For a production script, consider adding error handling and logging to manage any issues with connecting to Kafka or sending messages.
  • Security: If your Kafka broker requires authentication or uses SSL, you’ll need to add additional configuration to the KafkaProducer.
This script provides a basic example of sending log messages to Kafka. Adjust the configuration and message format as needed for your specific use case.

​
Setup Venv :

The error you’re encountering is due to PEP 668, which prevents pip from modifying system-managed Python environments (like the one installed via apt). This is a safety feature in newer Debian/Ubuntu systems. To resolve this cleanly and safely, create and use a virtual environment.

​
✅ Recommended Steps (Safe & Isolated):

# Step 1: Install Python virtual environment tools (if not already)
sudo apt install python3-venv python3-full -y

# Step 2: Create a virtual environment in your project directory
cd /var/www/devops/kafka-set
python3 -m venv .venv

# Step 3: Activate the virtual environment
source .venv/bin/activate

# Step 4: Install your required package (e.g., kafka-python)
pip install kafka-python
Once done, you can use Python and pip within this virtual environment safely: 
python your_script.py  # This will use the venv’s interpreter

​
🧼 To Deactivate:

deactivate