GOAL: To describe a fully automated process that will create new AMIs when Amazon releases updated AMIs, utilizing AWS Lambda, VSTS Build, Packer, and Octopus Deploy.
Amazon releases new AMIs (Amazon Machine Images) on a regular basis to incorporate security updates and feature enhancements. AWS AMIs also expire, which means you can no longer deploy instances utilizing the expired AMI version. This becomes an issue if you rely on AutoScaling Groups to launch instances, as the scale-up process will fail when adding a new instance with an expired AMI.
Creating AMIs is fairly straightforward, you can see the manual processes here, for Linux and Windows. A more automated process is desired, and one utility that is excellent at automating AMI creation is Packer. By integrating Packer with your build and deploy process you can easily automate the entire process.
Our current Build solution utilizes Visual Studio Team Services to create release packages, and deliver to our Deployment solution, Octopus Deploy which finalizes the deploy.
Assumptions: As there are many different aspects to deploying and automation, it is difficult to describe every detail. This post is to provide guidance to how to automate the process, but does not intend to provide a complete solution. For instance, it is assumed that Packer templates are already utilized and does not provide information on how to configure Packer templates.
In order to put everything together, we need to work backwards, starting with the actual deployment and finishing with the initial notification from AWS as each step integrates with the next.
1. Create an Octopus Project
We use Octopus Deploy to manage deployments and update our AWS Cloudformation stacks. The project runs a script on Octopus to update an AWS Cloudformation stack with updated AMI ids. The project is deployed by the VSTS build definition task. This blog post will not go into details of the process of updating Cloudformation, but will hopefully be described in a future post.
The Octopus Project name will be used in the VSTS build definition that will deploy the Octopus project.
2. Create VSTS build defintion
VSTS is used to create and package the build. The build definition tasks are:
- Run PowerShell script to invoke Packer and set VSTS variable with the new AMI ID. Packer must be installed and configured on the VSTS Build servers.
A simplified version of the script looks like this:packer build $debug packer-template.json -var “regions=us-east-1″| Tee-Object PackerOutput.txt
$output = Get-Content PackerOutput.txt
$AMIID = $output -split “us-east-1: ami-”
“##vso[task.setvariable variable=AMIID;]$AMIID” - Run PowerShell script to Update Octopus project variable with the new AMI ID, a sample script can be found here:
Octopus-update-variable.ps1
- Create and deploy the Octopus project release. Octopus Deploy provides a VSTS task within the VSTS Marketplace. More info can be found here: octopus-deploy-build-release-tasks
3. Create AWS Lambda to queue the VSTS Build
We will utilize AWS Lambda to queue our VSTS build by making an API request to VSTS. Create a new AWS Lambda following the steps from here:
Lambda-VSTS-Queue-Build
4. Subscribe the Lambda to the SNS topic
AWS provides an SNS topic that you can subscribe to when new AMIs are released for Linux and Windows.
Linux SNS Topic: arn:aws:sns:us-east-1:137112412989:amazon-linux-ami-updates
Windows SNS Topic: arn:aws:sns:us-east-1:801119661308:ec2-windows-ami-update
With the SNS topics, you can then subscribe the Lambda to trigger when a new AMI is released. Here is some info on how to subscribe the Lambda to the SNS topic: https://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html
Conclusion
With everything in place, you now have an automated process to create new AWS AMIs when AWS creates new AMIs.
SNS notification –> Trigger Lambda –> Queues VSTS Build –> Invokes Packer to create new AWS AMI –> Update and Deploy Octopus release
Disclaimer: All data and information provided on this site is for informational purposes only. www.nitedesign.com makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this site and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.