Skip to main content

​
Configure Logging and Monitoring in EKS/Kind/Minikube with External Prometheus and Loki Service

​
Logs Collection via Promtail and Push to External Loki

​
Step 1: Add the Grafana Helm repository

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm show values grafana/promtail > promtail-values.yaml

​
Step 2: Edit Custom Endpoint Push

Open the promtail-values.yaml file and configure the Loki API endpoint. 
nano promtail-values.yaml
Update the Loki URL under the clients section: 
config:
  clients:
    - url: http://12.34.56.78:3100/loki/api/v1/push  # Update the Loki API endpoint

​
Step 3: Apply Changes

Install Promtail using the modified promtail-values.yaml file: 
helm install promtail grafana/promtail -f promtail-values.yaml --namespace monitoring --create-namespace
Check the status of Promtail pods: 
k get pods -n monitoring
View Promtail logs: 
kubectl logs -n monitoring -l app.kubernetes.io/name=promtail
Check the status of Promtail pods again: 
k get pods -n monitoring

​
Metrics Collection and Sending to k*s_prometheus-server and Then External Prometheus Server

​
Step 1: Install Prometheus

Update the Helm repositories and install Prometheus in the monitoring namespace: 
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm search repo prometheus-community
helm install prometheus prometheus-community/prometheus --namespace monitoring --create-namespace
k get pods -n monitoring

​
Step 2: Export Prometheus Values

Export the Prometheus default values to customize them: 
helm show values prometheus-community/prometheus > prom-values.yaml

​
Step 3: Edit Remote Write Endpoint for Custom Push

Open the prom-values.yaml file and edit the remote write endpoint for the external Prometheus server. 
nano prom-values.yaml
Modify the remote_write section: 
remoteWrite:
  - url: "http://<external-prometheus-ip>:9090/api/v1/write"
Modify the Persistent storage section: 
server:
  enabled: true
  persistence:
    enabled: false  # Disable persistent storage
Modify the Persistent storage section: 
alertmanager:
  enabled: true
  persistence:
    enabled: false  # Disable persistence to avoid using PersistentVolumes
If Required kubectl delete statefulset prometheus-alertmanager -n monitoring
Note: Ensure that the external Prometheus has the flag --web.enable-remote-write-receiver enabled to accept connections from the EKS Prometheus.

​
Step 4: Apply Changes

Install or upgrade Prometheus with the modified configuration: 
helm upgrade --install prometheus prometheus-community/prometheus -f prom-values.yaml --namespace monitoring
Check the status of Prometheus pods: 
k get pods -n monitoring
kubectl describe pod <prometheus-server-pod-name> -n monitoring

​
Imp : Grafana Dash ID’s for Kubernetes : 15760 / 15757

​
Step 5: if prometheus pod is in error state due to PV and PVC

Create a PV and PVC with storage class in any one of the nodes which use the local storage to storage the data or you can use the AWS ebs if needed in you enviroment. Provide the Node Name 
# cat persistent-volumes.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-server-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  local:
    path: /mnt/data/prometheus  # Path on the node verify the permissions of this dir 
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - <NODE-NAME> # Node name where PV have to be create in local storage /mnt/data/prometheus

# cat persistent-volume-claim.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-server
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: prometheus-storage  # Use the StorageClass if needed

# cat storage-class.yaml 
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: prometheus-storage
provisioner: kubernetes.io/no-provisioner  # This is for local storage (not cloud)
volumeBindingMode: WaitForFirstConsumer
This setup allows you to collect logs via Promtail and metrics via Prometheus in your EKS cluster, pushing them to external Loki and Prometheus services for centralized monitoring.