Testing CloudWatch Alarm → SNS → Lambda → Slack Pipeline

🧪 Testing Methods (From Safest to Most Realistic)

Method 1: Direct Lambda Test (Already Done ✅)

What it tests: Lambda function only
Risk: Zero
Time: Instant

aws lambda invoke \
  --function-name CloudWatch-Alarms-To-Slack \
  --cli-binary-format raw-in-base64-out \
  --payload file://test-event.json \
  --region ap-south-1 \
  /tmp/lambda-test-output.json && cat /tmp/lambda-test-output.json

Result: ✅ Already verified working

Method 2: SNS → Lambda Test (Recommended ⭐)

What it tests: SNS → Lambda integration
Risk: Zero
Time: Instant

aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

Expected Output:

{
  "Subscriptions": [
    {
      "SubscriptionArn": "arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic:cd60dbe9-28fc-4ea4-b83c-abac23be799c",
      "Protocol": "lambda",
      "Endpoint": "arn:aws:lambda:ap-south-1:3AWS-Account-ID-NO5:function:CloudWatch-Alarms-To-Slack",
      "TopicArn": "arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic"
    }
  ]
}

Step 2: Publish a test message to SNS

aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: CloudWatch Alarm Notification" \
  --message '{
    "AlarmName": "TEST-SNS-CloudWatch-Alarm",
    "AlarmDescription": "Testing SNS to Lambda integration",
    "AWSAccountId": "3AWS-Account-ID-NO5",
    "NewStateValue": "ALARM",
    "NewStateReason": "Testing SNS topic can trigger Lambda function",
    "StateChangeTime": "2026-01-12T13:30:00.000Z",
    "Region": "ap-south-1",
    "AlarmArn": "arn:aws:cloudwatch:ap-south-1:3AWS-Account-ID-NO5:alarm:TEST-SNS-CloudWatch-Alarm",
    "OldStateValue": "OK",
    "Trigger": {
      "MetricName": "CPUUtilization",
      "Namespace": "AWS/RDS",
      "StatisticType": "Statistic",
      "Statistic": "Average",
      "Dimensions": [
        {
          "name": "DBInstanceIdentifier",
          "value": "test-rds-instance"
        }
      ],
      "Period": 300,
      "EvaluationPeriods": 3,
      "ComparisonOperator": "GreaterThanThreshold",
      "Threshold": 80.0
    }
  }' \
  --region ap-south-1

Expected Output:

{
  "MessageId": "abc-123-xyz..."
}

Step 3: Verify Lambda was invoked

# Check Lambda logs
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --since 1m --format short

# Check for Slack notification
# Look in your Slack channel for the test message

Method 3: Manually Trigger an Alarm State (Safe 🟢)

What it tests: CloudWatch → SNS → Lambda → Slack (full pipeline)
Risk: Low (just changes alarm state, doesn’t affect resources)
Time: Instant

Option A: Use a test alarm (if you have one)

aws cloudwatch set-alarm-state \
  --alarm-name CW-RDS-customapp-CPUUtilization \
  --state-value ALARM \
  --state-reason "Manual test - verifying alarm can trigger Slack notification" \
  --region ap-south-1

Option B: Create a temporary test alarm

# Create a test alarm on a low-risk metric
aws cloudwatch put-metric-alarm \
  --alarm-name TEST-Slack-Integration-Alarm \
  --alarm-description "Temporary alarm to test Slack notifications" \
  --metric-name CPUUtilization \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 0 \
  --comparison-operator LessThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=customapp \
  --alarm-actions arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# Wait 5-10 seconds, then set it to ALARM state
aws cloudwatch set-alarm-state \
  --alarm-name TEST-Slack-Integration-Alarm \
  --state-value ALARM \
  --state-reason "Testing Slack notification integration" \
  --region ap-south-1

# Check your Slack channel for notification

# Clean up: Delete the test alarm
aws cloudwatch delete-alarms \
  --alarm-names TEST-Slack-Integration-Alarm \
  --region ap-south-1

Method 4: Wait for Real Alarm (Production Test 🔴)

What it tests: Everything in real production scenario
Risk: None (passive monitoring)
Time: Variable (when an actual alarm triggers) Just wait for one of your 40 existing alarms to trigger naturally. This is the ultimate test but requires patience.

Method 5: Temporarily Lower Threshold (Controlled Test 🟡)

What it tests: Real CloudWatch alarm with actual metrics
Risk: Medium (might cause unnecessary alerts)
Time: 5-15 minutes

Example: Test CPU alarm with lower threshold

# Backup current alarm configuration first
aws cloudwatch describe-alarms \
  --alarm-names CW-RDS-customapp-CPUUtilization \
  --region ap-south-1 > /tmp/original-alarm.json

# Temporarily lower the threshold to trigger alarm
aws cloudwatch put-metric-alarm \
  --alarm-name CW-RDS-customapp-CPUUtilization \
  --alarm-description "This metric monitors CPU utilization for customapp" \
  --metric-name CPUUtilization \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=customapp \
  --alarm-actions arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# Wait 5-10 minutes for alarm to trigger with actual metric data

# Restore original alarm using Terraform
terraform apply -target=aws_cloudwatch_metric_alarm.rds_cpu_utilization[\"customapp\"] -auto-approve

⚠️ Warning: This will cause a real alarm to trigger. Use with caution!

Method 6: CloudWatch Logs Insights (Verification)

What it tests: Check if Lambda is being invoked by SNS
Risk: Zero
Time: Instant

# Query Lambda logs to see SNS invocations
aws logs insights start-query \
  --log-group-name /aws/lambda/CloudWatch-Alarms-To-Slack \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /Received SNS message/ | sort @timestamp desc | limit 10'

🎯 Recommended Testing Sequence

For Initial Testing (Do this now):

# 1. Verify SNS subscription (should already be confirmed)
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# 2. Test SNS → Lambda by publishing directly to SNS
aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: CloudWatch Alarm" \
  --message '{"AlarmName":"TEST-SNS-Integration","NewStateValue":"ALARM","NewStateReason":"Testing SNS to Lambda","StateChangeTime":"2026-01-12T13:30:00.000Z","Region":"ap-south-1","Trigger":{"MetricName":"CPUUtilization","Namespace":"AWS/RDS","Threshold":80.0}}' \
  --region ap-south-1

# 3. Check Lambda logs
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --since 1m --format short

# 4. Check Slack channel for notification

For Full End-to-End Testing (Optional):

# Create temporary test alarm
aws cloudwatch put-metric-alarm \
  --alarm-name TEMP-TEST-Slack-Integration \
  --metric-name CPUUtilization \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 0 \
  --comparison-operator LessThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=customapp \
  --alarm-actions arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# Trigger the alarm
aws cloudwatch set-alarm-state \
  --alarm-name TEMP-TEST-Slack-Integration \
  --state-value ALARM \
  --state-reason "End-to-end integration test" \
  --region ap-south-1

# Wait 10 seconds, check Slack

# Delete test alarm
aws cloudwatch delete-alarms \
  --alarm-names TEMP-TEST-Slack-Integration \
  --region ap-south-1

📊 What to Look For in Each Test

In Slack:

🚨 CloudWatch Alarm: ALARM
━━━━━━━━━━━━━━━━━━━━━━━━━
📌 Alarm Name: TEST-SNS-Integration
🔄 State Change: OK → ALARM
📝 Reason: Testing SNS to Lambda
...

In Lambda Logs:

START RequestId: xxx
Received SNS message: {
  "AlarmName": "TEST-SNS-Integration",
  ...
}
Slack response status: 200
Slack response data: ok
END RequestId: xxx

# Check SNS delivery success
aws cloudwatch get-metric-statistics \
  --namespace AWS/SNS \
  --metric-name NumberOfMessagesPublished \
  --dimensions Name=TopicName,Value=PROD_Default_CloudWatch_Alarms_Topic \
  --start-time $(date -u -d '1 hour ago' --iso-8601=seconds) \
  --end-time $(date -u --iso-8601=seconds) \
  --period 300 \
  --statistics Sum \
  --region ap-south-1

🔍 Troubleshooting

Check:

# 1. Verify subscription status
aws sns get-subscription-attributes \
  --subscription-arn <your-subscription-arn>

# 2. Check Lambda permission
aws lambda get-policy --function-name CloudWatch-Alarms-To-Slack

# 3. Check Lambda errors
aws logs filter-log-events \
  --log-group-name /aws/lambda/CloudWatch-Alarms-To-Slack \
  --filter-pattern "ERROR" \
  --start-time $(date -u -d '1 hour ago' +%s000)

Problem: Alarm test doesn’t trigger

Check:

# 1. Verify alarm has correct SNS topic
aws cloudwatch describe-alarms --alarm-names <alarm-name>

# 2. Check alarm history
aws cloudwatch describe-alarm-history \
  --alarm-name <alarm-name> \
  --history-item-type StateUpdate \
  --max-records 5

✅ Quick Test Script

Save this as test-slack-integration.sh:

#!/bin/bash

echo "Testing CloudWatch → SNS → Lambda → Slack Integration"
echo "======================================================"
echo ""

echo "Step 1: Publishing test message to SNS..."
MESSAGE_ID=$(aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: Integration Check" \
  --message '{"AlarmName":"Integration-Test","NewStateValue":"ALARM","NewStateReason":"Automated integration test","StateChangeTime":"'$(date -u +%Y-%m-%dT%H:%M:%S.000Z)'","Region":"ap-south-1","Trigger":{"MetricName":"TestMetric","Namespace":"Test","Threshold":100}}' \
  --region ap-south-1 \
  --query MessageId \
  --output text)

echo "✅ Message published to SNS. MessageId: $MESSAGE_ID"
echo ""

echo "Step 2: Waiting 5 seconds for Lambda to process..."
sleep 5
echo ""

echo "Step 3: Checking Lambda logs..."
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --since 30s --format short | grep -E "(Received SNS|Slack response|ERROR)"
echo ""

echo "Step 4: Check your Slack channel for the test notification!"
echo ""
echo "Done! 🎉"

Run it:

chmod +x test-slack-integration.sh
./test-slack-integration.sh

📋 Test Checklist

Lambda function tested directly ✅ (Already done)
SNS subscription confirmed as “Confirmed” status
SNS → Lambda test passed (Method 2)
Test alarm created and triggered (Method 3)
Slack message received
Lambda logs show successful execution
CloudWatch metrics show SNS deliveries
End-to-end pipeline verified

🎯 Summary

Best practice for testing:

✅ Already tested: Direct Lambda invocation
🟢 Do now: SNS publish test (Method 2)
🟡 Optional: Create temporary test alarm (Method 3)
🔵 Wait: Real alarm will eventually trigger naturally

The SNS publish test (Method 2) is the sweet spot - it’s:

Safe (no risk to production)
Fast (instant results)
Comprehensive (tests SNS → Lambda → Slack)
Realistic (uses same path as real alarms)

Run this now to verify everything works:

aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: End-to-End Integration" \
  --message '{"AlarmName":"E2E-Integration-Test","NewStateValue":"ALARM","NewStateReason":"Verifying CloudWatch can trigger Slack notifications","StateChangeTime":"'$(date -u +%Y-%m-%dT%H:%M:%S.000Z)'","Region":"ap-south-1","Trigger":{"MetricName":"CPUUtilization","Namespace":"AWS/RDS","Threshold":80.0}}' \
  --region ap-south-1 && echo "Check your Slack channel now!"

Terraform Commands & Concepts

TF Cloudflare

TF AWS CW Lambda Slack

TESTING GUIDE

Testing CloudWatch Alarm → SNS → Lambda → Slack Pipeline

🧪 Testing Methods (From Safest to Most Realistic)

Method 1: Direct Lambda Test (Already Done ✅)

Method 2: SNS → Lambda Test (Recommended ⭐)

Step 2: Publish a test message to SNS

Step 3: Verify Lambda was invoked

Method 3: Manually Trigger an Alarm State (Safe 🟢)

Option A: Use a test alarm (if you have one)

Option B: Create a temporary test alarm

Method 4: Wait for Real Alarm (Production Test 🔴)

Method 5: Temporarily Lower Threshold (Controlled Test 🟡)

Example: Test CPU alarm with lower threshold

Method 6: CloudWatch Logs Insights (Verification)

🎯 Recommended Testing Sequence

For Initial Testing (Do this now):

For Full End-to-End Testing (Optional):

📊 What to Look For in Each Test

In Slack:

In Lambda Logs:

🔍 Troubleshooting

Problem: Alarm test doesn’t trigger

✅ Quick Test Script

📋 Test Checklist

🎯 Summary

Terraform Commands & Concepts

TF Cloudflare

TF AWS CW Lambda Slack

​Testing CloudWatch Alarm → SNS → Lambda → Slack Pipeline

​🧪 Testing Methods (From Safest to Most Realistic)

​Method 1: Direct Lambda Test (Already Done ✅)

​Method 2: SNS → Lambda Test (Recommended ⭐)

​Step 1: Verify SNS subscription is confirmed

​Step 2: Publish a test message to SNS

​Step 3: Verify Lambda was invoked

​Method 3: Manually Trigger an Alarm State (Safe 🟢)

​Option A: Use a test alarm (if you have one)

​Option B: Create a temporary test alarm

​Method 4: Wait for Real Alarm (Production Test 🔴)

​Method 5: Temporarily Lower Threshold (Controlled Test 🟡)

​Example: Test CPU alarm with lower threshold

​Method 6: CloudWatch Logs Insights (Verification)

​🎯 Recommended Testing Sequence

​For Initial Testing (Do this now):

​For Full End-to-End Testing (Optional):

​📊 What to Look For in Each Test

​In Slack:

​In Lambda Logs:

​In SNS Metrics:

​🔍 Troubleshooting

​Problem: SNS test doesn’t trigger Lambda

​Problem: Alarm test doesn’t trigger

​✅ Quick Test Script

​📋 Test Checklist

​🎯 Summary

Testing CloudWatch Alarm → SNS → Lambda → Slack Pipeline

🧪 Testing Methods (From Safest to Most Realistic)

Method 1: Direct Lambda Test (Already Done ✅)

Method 2: SNS → Lambda Test (Recommended ⭐)

Step 1: Verify SNS subscription is confirmed

Step 2: Publish a test message to SNS

Step 3: Verify Lambda was invoked

Method 3: Manually Trigger an Alarm State (Safe 🟢)

Option A: Use a test alarm (if you have one)

Option B: Create a temporary test alarm

Method 4: Wait for Real Alarm (Production Test 🔴)

Method 5: Temporarily Lower Threshold (Controlled Test 🟡)

Example: Test CPU alarm with lower threshold

Method 6: CloudWatch Logs Insights (Verification)

🎯 Recommended Testing Sequence

For Initial Testing (Do this now):

For Full End-to-End Testing (Optional):

📊 What to Look For in Each Test

In Slack:

In Lambda Logs:

In SNS Metrics:

🔍 Troubleshooting

Problem: SNS test doesn’t trigger Lambda

Problem: Alarm test doesn’t trigger

✅ Quick Test Script

📋 Test Checklist

🎯 Summary