Implementing Backup of Logs Before EC2 Instance Termination in AWS Auto Scaling Group Using systemd

In this guide, we will implement a solution that automatically backs up logs from an application hosted on EC2 instances in an AWS Auto Scaling Group (ASG) before the instance is terminated. We will leverage systemd services to ensure the backup process completes before termination, and we will also mention how AWS Auto Scaling Lifecycle Hooks can be used as an alternative.

Prerequisites

• AWS Auto Scaling Group (ASG)
• EC2 instances with IAM roles allowing access to S3
• Application generating logs
• Backup script /var/www/backup/backup-to-s3.sh

Steps for Implementation

Step 1: Create and Configure systemd Service for Backup

We will create a systemd service on EC2 instances that will run the backup script when the instance is shutting down.

1. Create the systemd service:

   sudo nano /etc/systemd/system/backup.service

2. Service file configuration: Here is the configuration for the systemd service that ensures the backup happens before the EC2 instance is terminated:

   [Unit]
   Description=Backup Logs to S3 Before Termination
   DefaultDependencies=no
   Before=shutdown.target
   After=network.target
    
   [Service]
   Type=oneshot
   ExecStart=/bin/bash /var/www/backup/backup-to-s3.sh
   TimeoutSec=300
    
   [Install]
   WantedBy=shutdown.target

3. Enable the service: After creating the service file, enable it so that it runs during the shutdown process:

   sudo systemctl enable backup-to-s3.service

4. Ensure the backup script runs within the allowed time:

   • TimeoutSec=300 ensures that the instance waits for up to 5 minutes for the backup to complete.
   • If your backup script runs faster (e.g., under 60 seconds), the instance will terminate after the script completes successfully.

Step 2: IAM Permissions

Ensure that the EC2 instance's IAM role has the following permissions:

• s3:PutObject (for uploading logs to S3).

Step 3: Test the Setup

1. Manually terminate an instance in the ASG.
2. Verify that the backup script runs and uploads logs to S3.
3. The instance will terminate automatically after 60 seconds, once the backup is complete.

Using AWS Auto Scaling Lifecycle Hooks (Alternative)

Alternatively, you can use AWS Auto Scaling Lifecycle Hooks to pause the termination process and allow sufficient time for the backup process. Here's how:

1. Add a Lifecycle Hook to your ASG that pauses instance termination while the backup is running:
   • Go to the AWS Management Console → EC2 → Auto Scaling Groups.
   • Select your Auto Scaling group and go to Lifecycle Hooks.
   • Create a new hook:
     • Name: BackupLogsHook
     • Lifecycle transition: Instance Terminating
     • Timeout: Set to 300 seconds (or enough time to complete the backup).
2. Configure the backup script to notify Auto Scaling when the backup is complete by calling the complete-lifecycle-action API.

In this approach, Auto Scaling waits for the instance to signal that the backup is complete before terminating it.

Conclusion

By using systemd services, you can automate the backup of logs before instance termination, ensuring that no critical logs are lost during scaling operations. While systemd is a great approach, you can also leverage AWS Auto Scaling Lifecycle Hooks as an alternative to manage the termination process and ensure backups are completed before instance termination.