An article by Steve O’Gallagher
As I’ve discussed in my previous article, AWS can provide some super useful services to make your development life easier and your infrastructure more intelligent. This does however require a good chunk of effort to set everything up correctly with massive attention to detail, otherwise things are likely to break. Many pieces of the Amazon puzzle are not as intuitive as you might hope either, and there are traps along the way. In this article I’ll be highlighting some aspects to look out for which can save you going down some of the false paths I have previously found myself on.
Check Your Region
If you log into your AWS Console and discover that all of your stuff has disappeared, don’t immediately panic! While there is always the remote chance that everything has been completely deleted from your account (because of gremlins and trolls) it’s more likely that you’ve logged into the Console in the wrong region. If for example if you’ve created all of your resources specifically in the EU West 1 region then this is where they live in the AWS Console as well. If you then change the region of the Console to somewhere in America for example, then none of your resources will be presented to you. They are in a parallel universe called EU West 1, which apparently is in Ireland somewhere. Hop back into the right region and there they all are waiting for you. Panic over.
Linking Stack Resources by Name
Keeping track of your environment resources is made much easier by opening up relevant parameters of your CloudFormation json file to be editable in the AWS Console. This means that when you’re creating a new stack of servers/databases/roles etc., you can use the same cloud formation template file but completely rename everything if you use a parameter’s value to name your resources. When declaring the names of your resources in the template you can concatenate a string parameter into the names allowing them to use a unified naming convention. In practical terms, this means that if for example you create a parameter called “phase” you can enter “dev”, or “uat”, or “prod” as the value when you create your stack, and all of your stack resources can have the phase automatically included in their names. This way, all resources with “prod” in the name can easily be identified as belonging to the production stack for clearer resource management.
Stacks Failing to Update
Sometimes updating a stack in AWS doesn’t go according to plan: Cloud Formation gets completely stuck and is unable to roll back to the previous safe version. If this happens on your production environment, it’s pretty horrifying. Unfortunately, I know this pain. In my experience, this tends to happen if one or more of your Load Balanced instances is listed as “Out of Service” at the moment that the update is executed. What happens then is that Cloud Formation will send a message to terminate this Out of Service instance, but since the instance is unable to send confirmation that it has terminated, Cloud Formation waits for a signal that never comes until it times out. This breaks the stack in Cloud Formation and renders updates impossible. The only action available now? “Delete stack”. Yep. Big baddaboom. Check your instances are all healthy and “In Service” before you make a sweeping update!
Route 53 Saves the Day
If the worst does happen and your environment goes bang then a clever use of the Route 53 service behind the scenes can be super useful. Through Route 53 you can connect your public-facing URL to your Load Balancer or EC2 instance’s AWS URL. If your stack can’t be updated or fixed easily you can fire up a new stack, point your public URL to that new identical stack and then get to work on fixing or replacing the broken stack. Once fixed, point your URL back to the other stack and the users of your site will be none the wiser. Of course, it gets a little more complicated if you’re saving information to a stack-specific database, in which case you’ll need to do some migrations from one database to the other for continuity.
Set up your users, roles, and security groups in your Cloud Formation template rather than manually in the Console. This allows them to be baked into stacks so you know exactly what permissions everything should have when it gets launched or started up. If you correctly scope your permissions into your Cloud Formation template file, then all future stacks you create will have the same setup locked down from the beginning. This should save you from doing the bad thing of just opening up all roles to full admin-level access to get things to work in a hurry. Bad developer. Don’t do it.
Digging Around in Servers
Get comfortable with connecting to web servers via SSH. You can look at the system logs of your instances in the Console, but it’s pretty limited. If your environments aren’t working the way you expect them to you, more often than not you will benefit from using SSH to connect to your instincts and have a poke around to see what files are actually there and how they are configured. Don’t forget the golden rule though: don’t start petting your instances to make them work properly. Find the problem, and fix it globally by correcting how all your instances are being initialised and run. That way, you fix not only that instance but also all future instances too.
Unfortunately, a lot of the above is easier said than done. It takes a good amount of reading documentation and dialling into the nitty gritty to build things properly in AWS. If you don’t set it all correctly from the beginning, it will come back to bite you. Add plenty of time into your project setup phase to effectively build and test your infrastructure so that it runs smoothly once development properly begins. Randomly terminate your instances and check that they pop back up healthily again every time. Hit your infrastructure with load tests. The upfront cost to do this right could save you tons of time and cost later on.
If you have any tips of your own that you’ve gathered from time spent in AWS, please post them in the comments section of this article and share the knowledge!