​

Python Script to Send Log Messages to Kafka

​

Python Script

from kafka import KafkaProducer
import json

# Kafka configuration
# KAFKA_BROKER = 'localhost:9092'
KAFKA_BROKER = 'kafka.example.in:9092'

KAFKA_TOPIC = 'project1logs'
# KAFKA_TOPIC = 'djangologs'

# Initialize the Kafka producer
producer = KafkaProducer(
    bootstrap_servers=KAFKA_BROKER,
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Sample log message
log_message = {
    'level': 'INFO',
    'message': 'This is a sample log message for project1.',
    'timestamp': '2024-07-10T12:00:00'
}

# Send the log message to Kafka
producer.send(KAFKA_TOPIC, log_message)
producer.flush()
producer.close()

print(f"Log sent to Kafka topic {KAFKA_TOPIC}")

​

Configuration Details

​

Kafka Configuration

• KAFKA_BROKER:
  • The address of the Kafka broker. Change this to the appropriate address of your Kafka instance.
  • Example: kafka.example.in:9092
• KAFKA_TOPIC:
  • The Kafka topic to which the log message will be sent. Modify this based on your use case.
  • Example: 'project1logs'

​

Producer Initialization

• KafkaProducer:
  • bootstrap_servers: Address of the Kafka broker(s).
  • value_serializer: Function to serialize the log message into JSON format. The lambda v: json.dumps(v).encode('utf-8') converts the log message into a JSON string and encodes it to UTF-8.

​

Log Message

• log_message:
  • A sample log message with fields such as level, message, and timestamp. Adjust this message based on your logging needs.

​

Sending the Message

• producer.send:
  • Sends the log message to the specified Kafka topic.
• producer.flush:
  • Ensures all messages are sent before closing the producer.
• producer.close:
  • Closes the producer connection.

​

Usage

1. Install Dependencies: Make sure you have the kafka-python library installed. You can install it using pip:
   pip install kafka-python
   
2. Run the Script: Execute the Python script to send a log message to Kafka:
   python send_log.py
   
3. Verify: Check your Kafka topic to ensure that the log message has been sent and is available.

​

Notes

• Error Handling: For a production script, consider adding error handling and logging to manage any issues with connecting to Kafka or sending messages.
• Security: If your Kafka broker requires authentication or uses SSL, you’ll need to add additional configuration to the KafkaProducer.

This script provides a basic example of sending log messages to Kafka. Adjust the configuration and message format as needed for your specific use case.

​

Setup Venv :

​

✅ Recommended Steps (Safe & Isolated):

# Step 1: Install Python virtual environment tools (if not already)
sudo apt install python3-venv python3-full -y

# Step 2: Create a virtual environment in your project directory
cd /var/www/devops/kafka-set
python3 -m venv .venv

# Step 3: Activate the virtual environment
source .venv/bin/activate

# Step 4: Install your required package (e.g., kafka-python)
pip install kafka-python

python your_script.py  # This will use the venv’s interpreter

​

🧼 To Deactivate:

deactivate