CloudWatch Alarms → Slack Integration Reference

📊 Architecture Overview

┌─────────────────┐
│  CloudWatch     │
│  Alarms         │
│  (RDS/ALB)      │
└────────┬────────┘
         │
         │ 1. Alarm Triggers
         │    (State: OK → ALARM)
         ▼
┌─────────────────┐
│   SNS Topic     │
│  (PROD_Default_ │
│   CloudWatch_   │
│  Alarms_Topic)  │
└────────┬────────┘
         │
         │ 2. Publishes Message
         │    (JSON Format)
         ▼
┌─────────────────┐
│ Lambda Function │
│ (CloudWatch-    │
│  Alarms-To-     │
│  Slack)         │
└────────┬────────┘
         │
         │ 3. Formats & Sends
         │    (HTTP POST)
         ▼
┌─────────────────┐
│  Slack Channel  │
│  (via Webhook)  │
└─────────────────┘

🔄 Step-by-Step Flow

Step 1: CloudWatch Alarm Triggers

When: CPU > 80% for 15 minutes (3 consecutive 5-min periods)
What: CloudWatch changes alarm state from OK → ALARM
Action: Sends notification to SNS Topic

Example Alarm Configuration:

resource "aws_cloudwatch_metric_alarm" "rds_cpu" {
  alarm_name     = "CW-RDS-AppName-prod-rds-db-CPUUtilization"
  alarm_actions  = [var.sns_topic_arn]  # ← Sends to SNS
  # ...
}

SNS Topic ARN: arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic

SNS Message Format (JSON):

{
  "Records": [
    {
      "Sns": {
        "Type": "Notification",
        "MessageId": "abc-123-xyz",
        "TopicArn": "arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic",
        "Subject": "ALARM: CW-RDS-AppName-prod-rds-db-CPUUtilization",
        "Message": "{...CloudWatch alarm details...}",
        "Timestamp": "2026-01-12T10:30:00.000Z"
      }
    }
  ]
}

SNS has a subscription:

resource "aws_sns_topic_subscription" "lambda_subscription" {
  topic_arn = var.sns_topic_arn
  protocol  = "lambda"
  endpoint  = aws_lambda_function.slack_notifier.arn
}

This tells SNS: “Whenever you receive a message, invoke this Lambda function”

Step 3: Lambda Function Processes

Function: CloudWatch-Alarms-To-Slack
Runtime: Python 3.11
Timeout: 30 seconds

Lambda receives the SNS event and:

Extracts the CloudWatch alarm details from SNS message
Parses alarm information (name, state, reason, metrics)
Formats into a beautiful Slack message with colors and emojis
Sends HTTP POST to Slack webhook URL

Lambda Code Flow:

def lambda_handler(event, context):
    # 1. Parse SNS message
    sns_message = json.loads(event['Records'][0]['Sns']['Message'])
    
    # 2. Extract alarm details
    alarm_name = sns_message['AlarmName']
    new_state = sns_message['NewStateValue']
    
    # 3. Format Slack message
    slack_message = format_slack_message(sns_message)
    
    # 4. Send to Slack
    response = requests.post(SLACK_WEBHOOK_URL, json=slack_message)
    
    return {'statusCode': 200}

Step 4: Slack Receives Notification

Slack Message Format:

🚨 CloudWatch Alarm: ALARM

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📌 Alarm Name:
   CW-RDS-AppName-prod-rds-db-CPUUtilization

🔄 State Change:
   OK → ALARM

📝 Reason:
   Threshold Crossed: 3 datapoints [85.0, 87.2, 89.5] 
   were greater than the threshold (80.0)

📊 Metric Details:
   • Metric: CPUUtilization
   • Namespace: AWS/RDS
   • Threshold: > 80.0
   • DBInstance: AppName-prod-rds-db

🌍 Region: ap-south-1
⏰ Time: 2026-01-12 10:30:00 UTC

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

🔐 Permissions & Security

Lambda IAM Role Permissions

# Lambda needs:
1. Basic Lambda execution (write to CloudWatch Logs)
2. NO other AWS permissions needed (just HTTP to Slack)

resource "aws_iam_role" "lambda_slack_role" {
  # Trust policy: Allow Lambda service to assume this role
  assume_role_policy = {
    "Effect": "Allow",
    "Principal": { "Service": "lambda.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }
}

SNS → Lambda Permission

# Allow SNS to invoke Lambda
resource "aws_lambda_permission" "allow_sns" {
  statement_id  = "AllowExecutionFromSNS"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.slack_notifier.function_name
  principal     = "sns.amazonaws.com"
  source_arn    = var.sns_topic_arn
}

This permission says: “SNS topic can trigger this Lambda function”

📦 Resources Created

Resource	Name	Purpose
IAM Role	`CloudWatch-Alarms-To-Slack-Role`	Lambda execution role
Lambda Function	`CloudWatch-Alarms-To-Slack`	Processes alarms & sends to Slack
CloudWatch Log Group	`/aws/lambda/CloudWatch-Alarms-To-Slack`	Lambda execution logs
SNS Subscription	(auto-generated)	Connects SNS → Lambda
Lambda Permission	`AllowExecutionFromSNS`	Allows SNS to invoke Lambda

🧪 Testing Flow

Test 1: Slack Webhook (Direct)

curl -X POST https://hooks.slack.com/services/T09CNE15C4E/B09E3K0JLTA/... \
  -H 'Content-Type: application/json' \
  -d '{"text": "Test from curl"}'

✅ Verifies: Slack webhook URL is valid

Test 2: Lambda Function (Direct)

aws lambda invoke \
  --function-name CloudWatch-Alarms-To-Slack \
  --payload '{"Records":[{"Sns":{"Message":"..."}}]}' \
  output.json

✅ Verifies: Lambda can parse messages and send to Slack

aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --message '{"AlarmName":"TEST",...}'

✅ Verifies: SNS → Lambda integration works

Test 4: CloudWatch Alarm → SNS

# Trigger a real alarm by breaching threshold
# Or use AWS Console to set alarm to ALARM state

✅ Verifies: Complete end-to-end flow

🎨 Message Formatting

Color Coding

🔴 ALARM   → Red (#ff0000)    - Critical issue
🟢 OK      → Green (#36a64f)  - All good
🟡 INSUFFICIENT_DATA → Yellow (#ffcc00) - Not enough data

Emojis Used

🚨 - Alert header
📌 - Alarm name
🔄 - State change
📝 - Reason
📊 - Metric details
🌍 - Region
⏰ - Timestamp

Message Structure

Header with emoji + state
Separator line
Alarm name (bold)
State transition (with arrow)
Detailed reason
Metric information
Dimensions (RDS instance, ALB name, etc.)
Region and timestamp
Footer separator

🔧 Configuration

Environment Variables

environment {
  variables = {
    SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/..."
  }
}

Customization Options

Change Slack Channel:

Generate new webhook URL from Slack
Update slack_webhook_url in locals block

Modify Message Format:

Edit lambda_function.py
Update the format_slack_message() function

Add More Details:

Parse additional fields from CloudWatch alarm JSON
Add to Slack message payload

Filter Alarms:

Add logic in Lambda to filter by alarm name prefix
Only send specific alarms to Slack

📊 Monitoring & Troubleshooting

CloudWatch Logs

# View Lambda logs
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --follow

# Search for errors
aws logs filter-log-events \
  --log-group-name /aws/lambda/CloudWatch-Alarms-To-Slack \
  --filter-pattern "ERROR"

Common Issues

❌ Slack message not appearing:

Check Lambda logs: /aws/lambda/CloudWatch-Alarms-To-Slack
Verify webhook URL is correct
Check Lambda execution errors

❌ Lambda not triggered:

Verify SNS subscription is “Confirmed” (not Pending)
Check Lambda permission for SNS
Verify alarm has SNS topic in alarm_actions

❌ Message format broken:

Check CloudWatch alarm JSON structure
Verify Lambda Python code syntax
Test with sample alarm JSON

💰 Cost Estimation

Monthly Costs (Approximate)

Lambda Invocations: 
  - 1000 alarms/month × $0.20 per 1M requests = ~$0.0002

Lambda Compute:
  - 1000 invocations × 1 second × 256MB = ~$0.0001

CloudWatch Logs:
  - 1 GB ingestion = ~$0.50
  - Storage (14 days retention) = ~$0.03

SNS:
  - 1000 notifications = ~$0.0005

Total: ~$0.50 - $1.00/month (for moderate alarm volume)

Terraform Commands & Concepts

TF Cloudflare

TF AWS CW Lambda Slack

SLACK INTEGRATION REFERENCE

CloudWatch Alarms → Slack Integration Reference

📊 Architecture Overview

🔄 Step-by-Step Flow

Step 1: CloudWatch Alarm Triggers

Step 3: Lambda Function Processes

Step 4: Slack Receives Notification

🔐 Permissions & Security

Lambda IAM Role Permissions

SNS → Lambda Permission

📦 Resources Created

🧪 Testing Flow

Test 1: Slack Webhook (Direct)

Test 2: Lambda Function (Direct)

Test 4: CloudWatch Alarm → SNS

🎨 Message Formatting

Color Coding

Emojis Used

Message Structure

🔧 Configuration

Environment Variables

Customization Options

📊 Monitoring & Troubleshooting

CloudWatch Logs

Common Issues

💰 Cost Estimation

Monthly Costs (Approximate)

📚 References

🚀 Deployment Checklist

Terraform Commands & Concepts

TF Cloudflare

TF AWS CW Lambda Slack

​CloudWatch Alarms → Slack Integration Reference

​📊 Architecture Overview

​🔄 Step-by-Step Flow

​Step 1: CloudWatch Alarm Triggers

​Step 2: SNS Topic Receives & Publishes

​Step 3: Lambda Function Processes

​Step 4: Slack Receives Notification

​🔐 Permissions & Security

​Lambda IAM Role Permissions

​SNS → Lambda Permission

​📦 Resources Created

​🧪 Testing Flow

​Test 1: Slack Webhook (Direct)

​Test 2: Lambda Function (Direct)

​Test 3: SNS Topic → Lambda

​Test 4: CloudWatch Alarm → SNS

​🎨 Message Formatting

​Color Coding

​Emojis Used

​Message Structure

​🔧 Configuration

​Environment Variables

​Customization Options

​📊 Monitoring & Troubleshooting

​CloudWatch Logs

​Common Issues

​💰 Cost Estimation

​Monthly Costs (Approximate)

​📚 References

​🚀 Deployment Checklist

CloudWatch Alarms → Slack Integration Reference

📊 Architecture Overview

🔄 Step-by-Step Flow

Step 1: CloudWatch Alarm Triggers

Step 2: SNS Topic Receives & Publishes

Step 3: Lambda Function Processes

Step 4: Slack Receives Notification

🔐 Permissions & Security

Lambda IAM Role Permissions

SNS → Lambda Permission

📦 Resources Created

🧪 Testing Flow

Test 1: Slack Webhook (Direct)

Test 2: Lambda Function (Direct)

Test 3: SNS Topic → Lambda

Test 4: CloudWatch Alarm → SNS

🎨 Message Formatting

Color Coding

Emojis Used

Message Structure

🔧 Configuration

Environment Variables

Customization Options

📊 Monitoring & Troubleshooting

CloudWatch Logs

Common Issues

💰 Cost Estimation

Monthly Costs (Approximate)

📚 References

🚀 Deployment Checklist