> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ahmadraza.in/llms.txt
> Use this file to discover all available pages before exploring further.

# TESTING GUIDE

# Testing CloudWatch Alarm → SNS → Lambda → Slack Pipeline

## 🧪 Testing Methods (From Safest to Most Realistic)

***

## Method 1: Direct Lambda Test (Already Done ✅)

**What it tests**: Lambda function only\
**Risk**: Zero\
**Time**: Instant

```bash theme={null}
aws lambda invoke \
  --function-name CloudWatch-Alarms-To-Slack \
  --cli-binary-format raw-in-base64-out \
  --payload file://test-event.json \
  --region ap-south-1 \
  /tmp/lambda-test-output.json && cat /tmp/lambda-test-output.json
```

**Result**: ✅ Already verified working

***

## Method 2: SNS → Lambda Test (Recommended ⭐)

**What it tests**: SNS → Lambda integration\
**Risk**: Zero\
**Time**: Instant

### Step 1: Verify SNS subscription is confirmed

```bash theme={null}
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1
```

**Expected Output:**

```json theme={null}
{
  "Subscriptions": [
    {
      "SubscriptionArn": "arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic:cd60dbe9-28fc-4ea4-b83c-abac23be799c",
      "Protocol": "lambda",
      "Endpoint": "arn:aws:lambda:ap-south-1:3AWS-Account-ID-NO5:function:CloudWatch-Alarms-To-Slack",
      "TopicArn": "arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic"
    }
  ]
}
```

### Step 2: Publish a test message to SNS

```bash theme={null}
aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: CloudWatch Alarm Notification" \
  --message '{
    "AlarmName": "TEST-SNS-CloudWatch-Alarm",
    "AlarmDescription": "Testing SNS to Lambda integration",
    "AWSAccountId": "3AWS-Account-ID-NO5",
    "NewStateValue": "ALARM",
    "NewStateReason": "Testing SNS topic can trigger Lambda function",
    "StateChangeTime": "2026-01-12T13:30:00.000Z",
    "Region": "ap-south-1",
    "AlarmArn": "arn:aws:cloudwatch:ap-south-1:3AWS-Account-ID-NO5:alarm:TEST-SNS-CloudWatch-Alarm",
    "OldStateValue": "OK",
    "Trigger": {
      "MetricName": "CPUUtilization",
      "Namespace": "AWS/RDS",
      "StatisticType": "Statistic",
      "Statistic": "Average",
      "Dimensions": [
        {
          "name": "DBInstanceIdentifier",
          "value": "test-rds-instance"
        }
      ],
      "Period": 300,
      "EvaluationPeriods": 3,
      "ComparisonOperator": "GreaterThanThreshold",
      "Threshold": 80.0
    }
  }' \
  --region ap-south-1
```

**Expected Output:**

```json theme={null}
{
  "MessageId": "abc-123-xyz..."
}
```

### Step 3: Verify Lambda was invoked

```bash theme={null}
# Check Lambda logs
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --since 1m --format short

# Check for Slack notification
# Look in your Slack channel for the test message
```

***

## Method 3: Manually Trigger an Alarm State (Safe 🟢)

**What it tests**: CloudWatch → SNS → Lambda → Slack (full pipeline)\
**Risk**: Low (just changes alarm state, doesn't affect resources)\
**Time**: Instant

### Option A: Use a test alarm (if you have one)

```bash theme={null}
aws cloudwatch set-alarm-state \
  --alarm-name CW-RDS-customapp-CPUUtilization \
  --state-value ALARM \
  --state-reason "Manual test - verifying alarm can trigger Slack notification" \
  --region ap-south-1
```

### Option B: Create a temporary test alarm

```bash theme={null}
# Create a test alarm on a low-risk metric
aws cloudwatch put-metric-alarm \
  --alarm-name TEST-Slack-Integration-Alarm \
  --alarm-description "Temporary alarm to test Slack notifications" \
  --metric-name CPUUtilization \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 0 \
  --comparison-operator LessThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=customapp \
  --alarm-actions arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# Wait 5-10 seconds, then set it to ALARM state
aws cloudwatch set-alarm-state \
  --alarm-name TEST-Slack-Integration-Alarm \
  --state-value ALARM \
  --state-reason "Testing Slack notification integration" \
  --region ap-south-1

# Check your Slack channel for notification

# Clean up: Delete the test alarm
aws cloudwatch delete-alarms \
  --alarm-names TEST-Slack-Integration-Alarm \
  --region ap-south-1
```

***

## Method 4: Wait for Real Alarm (Production Test 🔴)

**What it tests**: Everything in real production scenario\
**Risk**: None (passive monitoring)\
**Time**: Variable (when an actual alarm triggers)

Just wait for one of your 40 existing alarms to trigger naturally. This is the ultimate test but requires patience.

***

## Method 5: Temporarily Lower Threshold (Controlled Test 🟡)

**What it tests**: Real CloudWatch alarm with actual metrics\
**Risk**: Medium (might cause unnecessary alerts)\
**Time**: 5-15 minutes

### Example: Test CPU alarm with lower threshold

```bash theme={null}
# Backup current alarm configuration first
aws cloudwatch describe-alarms \
  --alarm-names CW-RDS-customapp-CPUUtilization \
  --region ap-south-1 > /tmp/original-alarm.json

# Temporarily lower the threshold to trigger alarm
aws cloudwatch put-metric-alarm \
  --alarm-name CW-RDS-customapp-CPUUtilization \
  --alarm-description "This metric monitors CPU utilization for customapp" \
  --metric-name CPUUtilization \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 1 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=customapp \
  --alarm-actions arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# Wait 5-10 minutes for alarm to trigger with actual metric data

# Restore original alarm using Terraform
terraform apply -target=aws_cloudwatch_metric_alarm.rds_cpu_utilization[\"customapp\"] -auto-approve
```

⚠️ **Warning**: This will cause a real alarm to trigger. Use with caution!

***

## Method 6: CloudWatch Logs Insights (Verification)

**What it tests**: Check if Lambda is being invoked by SNS\
**Risk**: Zero\
**Time**: Instant

```bash theme={null}
# Query Lambda logs to see SNS invocations
aws logs insights start-query \
  --log-group-name /aws/lambda/CloudWatch-Alarms-To-Slack \
  --start-time $(date -u -d '1 hour ago' +%s) \
  --end-time $(date -u +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /Received SNS message/ | sort @timestamp desc | limit 10'
```

***

## 🎯 Recommended Testing Sequence

### For Initial Testing (Do this now):

```bash theme={null}
# 1. Verify SNS subscription (should already be confirmed)
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# 2. Test SNS → Lambda by publishing directly to SNS
aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: CloudWatch Alarm" \
  --message '{"AlarmName":"TEST-SNS-Integration","NewStateValue":"ALARM","NewStateReason":"Testing SNS to Lambda","StateChangeTime":"2026-01-12T13:30:00.000Z","Region":"ap-south-1","Trigger":{"MetricName":"CPUUtilization","Namespace":"AWS/RDS","Threshold":80.0}}' \
  --region ap-south-1

# 3. Check Lambda logs
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --since 1m --format short

# 4. Check Slack channel for notification
```

### For Full End-to-End Testing (Optional):

```bash theme={null}
# Create temporary test alarm
aws cloudwatch put-metric-alarm \
  --alarm-name TEMP-TEST-Slack-Integration \
  --metric-name CPUUtilization \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300 \
  --evaluation-periods 1 \
  --threshold 0 \
  --comparison-operator LessThanThreshold \
  --dimensions Name=DBInstanceIdentifier,Value=customapp \
  --alarm-actions arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --region ap-south-1

# Trigger the alarm
aws cloudwatch set-alarm-state \
  --alarm-name TEMP-TEST-Slack-Integration \
  --state-value ALARM \
  --state-reason "End-to-end integration test" \
  --region ap-south-1

# Wait 10 seconds, check Slack

# Delete test alarm
aws cloudwatch delete-alarms \
  --alarm-names TEMP-TEST-Slack-Integration \
  --region ap-south-1
```

***

## 📊 What to Look For in Each Test

### In Slack:

```
🚨 CloudWatch Alarm: ALARM
━━━━━━━━━━━━━━━━━━━━━━━━━
📌 Alarm Name: TEST-SNS-Integration
🔄 State Change: OK → ALARM
📝 Reason: Testing SNS to Lambda
...
```

### In Lambda Logs:

```
START RequestId: xxx
Received SNS message: {
  "AlarmName": "TEST-SNS-Integration",
  ...
}
Slack response status: 200
Slack response data: ok
END RequestId: xxx
```

### In SNS Metrics:

```bash theme={null}
# Check SNS delivery success
aws cloudwatch get-metric-statistics \
  --namespace AWS/SNS \
  --metric-name NumberOfMessagesPublished \
  --dimensions Name=TopicName,Value=PROD_Default_CloudWatch_Alarms_Topic \
  --start-time $(date -u -d '1 hour ago' --iso-8601=seconds) \
  --end-time $(date -u --iso-8601=seconds) \
  --period 300 \
  --statistics Sum \
  --region ap-south-1
```

***

## 🔍 Troubleshooting

### Problem: SNS test doesn't trigger Lambda

**Check:**

```bash theme={null}
# 1. Verify subscription status
aws sns get-subscription-attributes \
  --subscription-arn <your-subscription-arn>

# 2. Check Lambda permission
aws lambda get-policy --function-name CloudWatch-Alarms-To-Slack

# 3. Check Lambda errors
aws logs filter-log-events \
  --log-group-name /aws/lambda/CloudWatch-Alarms-To-Slack \
  --filter-pattern "ERROR" \
  --start-time $(date -u -d '1 hour ago' +%s000)
```

### Problem: Alarm test doesn't trigger

**Check:**

```bash theme={null}
# 1. Verify alarm has correct SNS topic
aws cloudwatch describe-alarms --alarm-names <alarm-name>

# 2. Check alarm history
aws cloudwatch describe-alarm-history \
  --alarm-name <alarm-name> \
  --history-item-type StateUpdate \
  --max-records 5
```

***

## ✅ Quick Test Script

Save this as `test-slack-integration.sh`:

```bash theme={null}
#!/bin/bash

echo "Testing CloudWatch → SNS → Lambda → Slack Integration"
echo "======================================================"
echo ""

echo "Step 1: Publishing test message to SNS..."
MESSAGE_ID=$(aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: Integration Check" \
  --message '{"AlarmName":"Integration-Test","NewStateValue":"ALARM","NewStateReason":"Automated integration test","StateChangeTime":"'$(date -u +%Y-%m-%dT%H:%M:%S.000Z)'","Region":"ap-south-1","Trigger":{"MetricName":"TestMetric","Namespace":"Test","Threshold":100}}' \
  --region ap-south-1 \
  --query MessageId \
  --output text)

echo "✅ Message published to SNS. MessageId: $MESSAGE_ID"
echo ""

echo "Step 2: Waiting 5 seconds for Lambda to process..."
sleep 5
echo ""

echo "Step 3: Checking Lambda logs..."
aws logs tail /aws/lambda/CloudWatch-Alarms-To-Slack --since 30s --format short | grep -E "(Received SNS|Slack response|ERROR)"
echo ""

echo "Step 4: Check your Slack channel for the test notification!"
echo ""
echo "Done! 🎉"
```

Run it:

```bash theme={null}
chmod +x test-slack-integration.sh
./test-slack-integration.sh
```

***

## 📋 Test Checklist

* [ ] Lambda function tested directly ✅ (Already done)
* [ ] SNS subscription confirmed as "Confirmed" status
* [ ] SNS → Lambda test passed (Method 2)
* [ ] Test alarm created and triggered (Method 3)
* [ ] Slack message received
* [ ] Lambda logs show successful execution
* [ ] CloudWatch metrics show SNS deliveries
* [ ] End-to-end pipeline verified

***

## 🎯 Summary

**Best practice for testing:**

1. ✅ **Already tested**: Direct Lambda invocation
2. 🟢 **Do now**: SNS publish test (Method 2)
3. 🟡 **Optional**: Create temporary test alarm (Method 3)
4. 🔵 **Wait**: Real alarm will eventually trigger naturally

The SNS publish test (Method 2) is the **sweet spot** - it's:

* Safe (no risk to production)
* Fast (instant results)
* Comprehensive (tests SNS → Lambda → Slack)
* Realistic (uses same path as real alarms)

**Run this now to verify everything works:**

```bash theme={null}
aws sns publish \
  --topic-arn arn:aws:sns:ap-south-1:3AWS-Account-ID-NO5:PROD_Default_CloudWatch_Alarms_Topic \
  --subject "TEST: End-to-End Integration" \
  --message '{"AlarmName":"E2E-Integration-Test","NewStateValue":"ALARM","NewStateReason":"Verifying CloudWatch can trigger Slack notifications","StateChangeTime":"'$(date -u +%Y-%m-%dT%H:%M:%S.000Z)'","Region":"ap-south-1","Trigger":{"MetricName":"CPUUtilization","Namespace":"AWS/RDS","Threshold":80.0}}' \
  --region ap-south-1 && echo "Check your Slack channel now!"
```