Snapshots in Amazon when taken manually, are not available for scheduling. In order to accomplish this you could run Amazon CLI but for that you would need to execute from a server based on a cron job or something similar. This solution will allow you to run a serverless architecture which leverages existing Amazon services.

Lamba Files

Follow the link below to download the Lambda files you will need to run the automated scripts.

https://github.com/aws-samples/aws-rds-snapshot-management

Click the “Clone or download” button on the right , right click “Download Zip” and click “Save link as…”, save this file where you can find it and unzip the file.

Amazon RDS - Serverless Automated Snapshot

S3 Bucket

User Permissions

Before you can create the bucket we’ll need to assign the AmazonS3FullAccess policy to your user.

Amazon RDS - Serverless Automated Snapshot - 2

Select Service at the top and type IAM in the box, click IAM in the drop down.

Amazon RDS - Serverless Automated Snapshot - 3

Select Users on the left side of the screen.

Amazon RDS - Serverless Automated Snapshot - 4

Select your user on the right.

Amazon RDS - Serverless Automated Snapshot - 5

Click add Permissions.

Click Attach existing policies directly. (alternatively you can add it to a role)

Amazon RDS - Serverless Automated Snapshot - 6

In the search box, type AmazonS3 and you’ll see the AmazonS3FullAccess Policy, click the check beside it to select.

Amazon RDS - Serverless Automated Snapshot - 7
Amazon RDS - Serverless Automated Snapshot - 8

Click Next: Review at the bottom.

Amazon RDS - Serverless Automated Snapshot - 9

Click Add Permissions

Create Bucket

Amazon RDS - Serverless Automated Snapshot - 10

Next we’ll create the bucket to store our Lambda Functions, click services at the top and select S3 under storage.

Amazon RDS - Serverless Automated Snapshot - 11

Click Create Bucket.

Amazon RDS - Serverless Automated Snapshot - 12

Name your bucket. Click Next.

Click Next, Next and Create Bucket.

Upload Files

Amazon RDS - Serverless Automated Snapshot - 13

Click on your bucket name.

Amazon RDS - Serverless Automated Snapshot - 14

Click Upload

Amazon RDS - Serverless Automated Snapshot - 15

Click Add files

Go to the location where you unzipped the files , go to the lambda folder, select all 4 files and click Open.

Amazon RDS - Serverless Automated Snapshot - 16
Amazon RDS - Serverless Automated Snapshot - 17

Make sure all files are added to the view and click Upload.

Amazon RDS - Serverless Automated Snapshot - 18

The files should now show up in your bucket.

Amazon RDS - Serverless Automated Snapshot - 19

Click on Permissions, then click on Access Control List

Amazon RDS - Serverless Automated Snapshot - 20

Click on Everyone

Click List Objects and Read bucket permissions

Amazon RDS - Serverless Automated Snapshot - 21

Remove these public permissions after the stack is created. See section on S3 Permissions for assigning distinct permissions.

Cloud Formation Stack

In order to execute the Lambda files in your S3 bucket, you need to create a stack. The stack will enable the parameters of the scripts for customization.

Download Template

Go to the link below and save the rds_snapshot_management.yml file.

https://github.com/aws-samples/aws-rds-snapshot-management/tree/master/cftemplate

Create Stack

Amazon RDS - Serverless Automated Snapshot - 22

Click Services at the top of the page, type in formation and then click CloudFormation.

Amazon RDS - Serverless Automated Snapshot - 23

Click Create Stack

Make sure “Template is ready” and “Upload a template file” are selected, then click “Choose file”.

Amazon RDS - Serverless Automated Snapshot - 24

Go to where you unzipped the files, open the cftemplate folder, Select rds_snapshot_management.yml in the prompt and click Open.

Amazon RDS - Serverless Automated Snapshot - 25

You should now see the S3 generated url for your template, click Next.

Amazon RDS - Serverless Automated Snapshot - 26

Configuration

On this screen you will see a list of parameters, these parameters will determine how your snapshots will be retained, when they will be taken and several others.

Amazon RDS - Serverless Automated Snapshot - 27

Assign a unique name to the stack.

For BackupSchedule I have chosen to run a snapshot every 30 days, which is 720 hours.  I want my backup to run at 2am every month so I’ve input “0 2 1 * ? *”.

Amazon RDS - Serverless Automated Snapshot - 28

Other format examples:

Amazon RDS - Serverless Automated Snapshot - screenshot2
Amazon RDS - Serverless Automated Snapshot - screenshot1

Input the name of the bucket you created earlier in the CodeBucket box. You can set DeleteOldSnapshots to true if you don’t want to delete them manually. This is dependent on the number of retention days that you specify further down.

Amazon RDS - Serverless Automated Snapshot - 29-

Set the loglevel which is the notifications you’ll get if something goes wrong.  This level is a threshold , so if you set it to DEBUG, you’ll get everything from DEBUG-CRITICAL. If you set it to ERROR, you’ll get ERROR-CRITICAL.

Enter the email address that you want notified when a snapshot is taken.
RetentionDays is the number of days that a snapshot will be retained. I am retaining my backups for an entire year, so I’ve chosen 366 days to account for a leap year. If you don’t want snapshots to be deleted go back and set DeleteOldSnapshots to false.

Amazon RDS - Serverless Automated Snapshot - 29

Once you have your configuration setup and you’ve checked it, click Next.

Permissions

This is optional, you can add a role that will be responsible for managing resources on the stack.

Amazon RDS - Serverless Automated Snapshot - 30

Whichever option you pick, you will need these permissions in order to successfully create the stack.

Amazon RDS - Serverless Automated Snapshot - 31

Once you’re done, click next.

At this point you’re done with setup, review your configuration.  At the bottom of this page you’ll need to acknowledge that the stack will be creating IAM Access Roles that it uses to operate.

Amazon RDS - Serverless Automated Snapshot - 32

Review your setup and click Create stack.

Amazon RDS - Serverless Automated Snapshot - 34

Post Configuration

Now we can configure which RDS instances will have snapshots taken manually.

Amazon RDS - Serverless Automated Snapshot - 35

Create a rds_backups_list.txt file and input each rds name you want to take snapshots of, on it’s own line in the file

Now go to your S3 environment, there will be an S3 bucket which was created by the Cloud Stack for you to upload the text file. I named my Cloud Stack automated-snapshot-monthly, below you can see the bucket it created for me.

Upload your file into the bucket.

Amazon RDS - Serverless Automated Snapshot - 36

Testing

Now we can test our deployment and make sure snapshots are taken manually.

Amazon RDS - Serverless Automated Snapshot - 37

Click services and type step into the search box, click on Step Functions.

Amazon RDS - Serverless Automated Snapshot - 38

If you don’t see state machines right away, click Get Started on the right side.

Amazon RDS - Serverless Automated Snapshot - 39

Now click State Machines in the breadcrumb at the top.

Amazon RDS - Serverless Automated Snapshot - 40

Click on your state machine with the text of RDSBackupTool

Amazon RDS - Serverless Automated Snapshot - 41

Click Start Execution

Amazon RDS - Serverless Automated Snapshot - 42

Enter a name for the execution task

Amazon RDS - Serverless Automated Snapshot - 43

Click Start Execution

In the visual workflow you can see the steps and status of each step.  For instance the Create Snapshots step has failed in this example below.

Amazon RDS - Serverless Automated Snapshot - 44

If you click on the Create Snapshots box, on the right side you can see the details, the Exception field will give you more information on why a step failed.

This error has to do with S3 bucket permissions. If you setup a role for CloudStack in previous steps, make sure it has access to your S3 bucket and contents.  If you didn’t setup a role make sure your user has access to the S3 bucket. You can test this by setting your buckets to Public and testing again, if it succeeds then you need to modify permissions. I don’t recommend keeping your buckets public.

Amazon RDS - Serverless Automated Snapshot - 45

The below error means most likely that your rds_backups_list.txt file is not named correctly, check the name of the file and make sure it is exact.

Amazon RDS - Serverless Automated Snapshot - 46

S3 Permissions

Handling bucket permissions can be a hassle if you’ve never done it before. Especially if you don’t want them to be public. Here is how I setup my S3 policy for the two buckets needed in this tutorial.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-monthly-s3sourcelistbucket-153ftue9h0bzj/automated-snapshot-monthly-s3sourcelistbucket-153ftue9h0bzj"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-lambda/automated-snapshot-lambda"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-lambda/check_snapshots.zip"
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-lambda/delete_old_snapshots_rds.zip"
        },
        {
            "Sid": "VisualEditor4",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-lambda/email_notification.zip.zip"
        },
        {
            "Sid": "VisualEditor5",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-lambda/take_snapshots_rds.zip.zip"
        },
        {
            "Sid": "VisualEditor6",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::automated-snapshot-monthly-s3sourcelistbucket-153ftue9h0bzj/automated-snapshot-monthly-s3sourcelistbucket-153ftue9h0bzj/rds_backups_list.txt"
        }
    ]
}

Basically, you need to add a permission for GetObject (read permission) on each bucket and on each object in the bucket.  Attach this policy to your user or to the role you assigned for your CloudFormation.

That’s pretty much it, once you get those issues resolved you will see a workflow that looks like the one below. Check your RDS Snapshots and you’ll see the snapshot that was taken.

Amazon RDS - Serverless Automated Snapshot - 47
Share This