​

Setting Up Loki and Promtail with Self-Signed Certificates for Secure Communication

This guide explains how to set up Loki and Promtail using HTTPS with self-signed certificates for secure communication. Follow the steps below for certificate generation, configuration, and integration.

---

​

1. Generate Self-Signed Certificates

1. Generate a Private Key:
   openssl genrsa -out loki.key 2048
   
2. Generate a Certificate Signing Request (CSR):
   openssl req -new -key loki.key -out loki.csr -subj "/C=US/ST=State/L=City/O=Organization/OU=Unit/CN=localhost"
   
3. Generate a Self-Signed Certificate:
   openssl x509 -req -days 3650 -in loki.csr -signkey loki.key -out loki.crt
   
   • This certificate is valid for 10 years.
4. Place the Certificates in Secure Locations:
   • Private Key: /etc/ssl/private/loki.key
   • Certificate: /etc/ssl/certs/loki.crt

---

​

2. Configure Loki

1. Edit the Loki Configuration File: Update the Loki configuration (loki-config.yaml) with the following:
   server:
     http_listen_port: 3100
     http_tls_config:
       cert_file: /etc/ssl/certs/loki.crt
       key_file: /etc/ssl/private/loki.key
   
2. Start Loki:
   ./loki-linux-amd64 --config.file=loki-config.yaml
   

---

​

3. Configure Promtail

1. Prepare the Certificate for Promtail: Copy the loki.crt to the Promtail server and place it in a directory, e.g., /etc/ssl/certs/loki.crt.
2. Edit the Promtail Configuration File: Update the Promtail configuration (promtail-config.yaml) with the following:
   clients:
     - url: https://localhost:3100/loki/api/v1/push
       tls_config:
         ca_file: /etc/ssl/certs/loki.crt
         insecure_skip_verify: false
   positions:
     filename: /tmp/positions.yaml
   scrape_configs:
     - job_name: nginx-logs
       static_configs:
         - targets:
             - localhost
           labels:
             job: nginx
             __path__: /var/log/nginx/*.log
   
3. Start Promtail:
   ./promtail-linux-amd64 --config.file=promtail-config.yaml
   

---

​

4. Verify Secure Communication

1. Test Loki with HTTPS: Run the following curl command to verify Loki is serving HTTPS traffic:
   curl -v --cacert /etc/ssl/certs/loki.crt https://localhost:3100/ready
   
2. Check Promtail Logs: Ensure Promtail is successfully sending logs to Loki. Look for logs like:
   level=info msg="Batch sent successfully" tenant=""
   

---

​

5. Integrate Loki with Grafana

1. Access Grafana: Open Grafana in a browser (e.g., http://<grafana-ip>:3000).
2. Add Loki as a Data Source:
   • Navigate to Configuration > Data Sources > Add Data Source.
   • Select Loki.
   • Configure the following:
     • URL: https://<loki-server>:3100
     • TLS Settings:
       • CA Certificate: Paste the contents of loki.crt.
       • Skip TLS Certificate Validation: Leave unchecked.
3. Save & Test: Ensure Grafana successfully connects to Loki.

---

​

6. Automate Certificate Renewal (Optional)

For production use, automate the certificate renewal process and update the certificates without downtime. Use tools like cron to schedule renewal and reload services.

---

​

7. Troubleshooting

​

Common Issues:

• Certificate Verification Failed: Ensure the ca_file paths are correct in Promtail and Grafana.
• Connection Refused: Verify Loki is running on HTTPS and accessible on the specified port.
• Logs Not Pushing: Check Promtail logs for connectivity or configuration issues.

For more visit my blog: docs.ahmadraza.in