One-liner to clean up all unattached EBS volumes

So you want to clean up all those EBS volumes that are hanging around bleeding money? Unfortunately the EC2 console is woefully inadequate for the task when you’ve got more that a handful of volumes.

Here’s a bash one liner that gets the job done:

aws ec2 describe-volumes –filters “Name=status,Values=available” | python -c “from __future__ import print_function; import json,sys;data=json.load(sys.stdin); [ print(v[‘VolumeId’]) for v in data[‘Volumes’]]” | xargs -n 1 -I % aws ec2 delete-volume –volume-id=%

You’re welcome.

Tagged , , , ,

Docker Makes Creating Secure Sandboxes Easier Than Ever

Today dotCloud announced that they have open sourced their LXC container runtime Docker. This is exciting news because Docker features many important things that have been missing from the stock LXC packages. The two most significant features in my mind are the out of the box support for creation of AUFS based images for true copy-on-write read-only file systems (similar to how distro live CDs work) and fast launching of ephemeral containers! With these two components in place quick launching of sandboxes to run arbitrary code in relative safety is now super easy.

To give you an example of just how easy, lets take a look at how to run some JavaScript on node safely inside a sandbox with Docker.


  1. Access to an Ubuntu Machine or an Ubuntu VM
  2. Install docker using the instructions here:

Let’s Do It!

The base docker image is a bare bones ubuntu server install. We’re going to use that as a base and create our own image with node installed.

In case you’d rather just watch check out the screencast:

Step 1) Open a terminal and start an instance using the base image. This will launch you into a shell where we can begin to customise the image:

$ sudo docker run -i -t base /bin/bash

Step 2) Install node:

$ apt-get update
$ apt-get install python-software-properties python g++ make
$ add-apt-repository ppa:chris-lea/node.js
$ apt-get update
$ apt-get install nodejs

Note: replace the `python-software-properties` package with `software-properties-common` on Ubuntu 12.10 and above.

Step 3) Next bake our own image. Open another terminal session leaving the other one active in the background, then:

$ sudo docker ps

This should show the ID of the running container in the other terminal. Copy it then run:

$ sudo docker commit <paste your container ID here> node

Step 4) Check that the image was created:

$ sudo docker images

If all went to plan you should now be able to see your new image called “node” appearing in the list.

Step 5) Run some code inside your new sandbox:

$ echo "console.log('Hello World');" | sudo docker run -i node /bin/bash -c "cat > hello.js; node hello.js"


And that’s all there is to it! You’ve just successfully run arbitrary code in a safe, secure sandbox. Better still, since it’s based on LXC there are options for setting resource quotas to limit CPU and Memory usage meaning that denial of service by resource starvation is now a thing of the past.

I’m really looking forward to seeing how this develops over the coming months and how others put it to use.

Tagged , , , , , , , , , , ,

Experimenting with Linux Containers

Something I’ve been keeping a close eye on of late is Linux Containers otherwise known as LXC. For the uninitiated Linux Containers allow you to create virtual machines without the overhead of conventional virtualization. Actually, they’re not really virtual machines but virtual environments. Containers share the same kernel as the host operating system however thanks to a new-ish kernel feature called cgroups offer almost complete isolation of processes and system resources. These are more or less equivalent to Solaris Zones or FreeBSD jails which have been around for years but never jumped the *nix divide, until now. Support for containers has been in the kernel since version 2.6.29 however early users complained of the paucity of documentation and lack of tooling. Fortunately great strides have been made in recent Ubuntu releases (particularly 12.04 and later) to make them simple and fast to setup. To see what I mean take a look at this great guide from one of the LXC project developers: Stéphane Graber.

Personally I’m excited about Linux containers because being a heavy EC2 user Solaris and FreeBSD aren’t an option (Once upon a time, before Oracle, Open Solaris was available but then it suddenly vanished. FreeBSD on the other hand can be made to work though it involves defenestrating a Windows AMI which means paying Windows prices and is generally a nasty, if not ingenious, hack). Moreover I’ve evolved into somewhat of an Ubuntu specialist so the idea of changing out OS to gain container support isn’t really all that great. Of course there’s also OpenVZ but this involves patching the kernel which I’m also not mad keen on. Keeping things stock as possible really helps when it comes to upgrade cycles and eliminates a whole area of investigation when tracking down difficult bugs.

So why would anyone care about containers? Well besides kernel development and running VPSs, it turns out they’re pretty useful for what we do over at At we need to run untrusted code from our customers safely and securely. Doing this requires creating a secure sandbox in which to run code so that if anything goes wrong (either through malicious intent or raw boned stupidity) it won’t affect any of our other clients. This is a complex thing to do as it involves limiting access to sensitive resources such as disk, memory and processor time. Some platforms such as Google AppEngine have taken the approach of neutering the language runtime to make such operations theoretically impossible, however this is a costly and error prone approach. Earlier this year several security flaws were found in the Java Virtual Machine’s security policy manager and left addressed for months leaving many vulnerable. We felt a more robust and broadly applicable approach was to design around the inherent security features provided by the operating system.

Traditionally the way to sandbox processes in Linux environments is to implement a chroot jail. Jails like leave a lot to be desired. For a start the only thing being protected is the file system. Forget process isolation or resource quotas, with the right permissions everything is fair game. Creating the chroot file system is also a veritable minefield. One must be extremely careful to eliminate any programs which would allow an sort of privilege escalation. This is much harder than it sounds as hidden in many utilities are innocuous little programs with their setuid bits set. Breaking out of a chroot jail is easy if you have root permissions. Lets assume for a second you have created a safe file system though, and no possible way exists for a user to gain elevated privileges. There’s still the issue of resource starvation which is, after a fashion, a kind of denial of service attack. A simple fork bomb, allocating large chunks memory till it exhausts, loading the NIC with bogus traffic or just writing a never ending loop to consume all the CPU time are just some of the challenges you’ll encounter. Well configured PAM limits certainly can go a long way to alleviating these issues but it’s still far too coarse grained for my liking.

Containers are awesome because they address all these issues. They provide an isolated file system which can’t be broken out of via privilege escalation. They also give us process isolation (i.e. ps aux only shows processes running inside the container) and the ability to set resource quotas for memory, CPU time, disk and network thanks to cgroups. This goes a long way to making arbitrary code execution within a multi-tenant environment much, much safer and means I can sleep easier at night knowing that the only person a user can screw, whether intentionally or unintentionally, is themselves.

Tagged , , , , , , ,

Scaling Your Engineering Team

The last few months of building Hubify have taught me a lot about scaling infrastructure. When we talk about scale we often think in terms of raw performance (i.e. how many, how fast etc.). While scaling a website to handle a large amount of traffic is a big enough challenge many neglect to consider the effect of scaling on your team. Modern web-scale architecture involves many moving parts. Web servers, load balances, memcaches, databases, message queues, the list goes on. Without a solid strategy for managing all these components you may find your team and your ability to iterate rapidly (a startup’s primary advantage) breaks down long before your infrastructure does.

Before I launch into specific tactics to manage complexity it might be helpful to gain some perspective of the size of the challenge in front of many startups. Using Hubify as an example, we have an engineering team of just two people. Only it’s actually less than that. While my co-founder Trindall is an engineer, his great strength lies in sales and customer relations so he spends most of his time there. That just leaves little ol’ me. Still that’s not too bad really, I mean one engineer can do a lot right? True, and I’m pretty good if I don’t say so myself, but even I have limits and Hubify severely tests those limits. Today we have close to 60 servers which is a mix of of 6 distinct server archetypes (that might sound shocking but there’s some good reasons behind those numbers which I won’t get into right now). For comparisons sake I used to work in a team of approx. 30 developers who looked after roughly two-thirds that number of servers.

Anyone who’s ever managed that number of servers will tell you that things can get hairy pretty quickly. Sorting out issues for one can be downright baffling. What server did the error occur on? Which log is the error likely to appear in? How can this be reproduced locally? It’s like looking for a needle in the proverbial haystack. When you do eventually find the cause, deploying a fix can take absolutely ages and involve many complex steps. Add a requirement for high-availability to the mix and you’re looking at a looong night. Salting an already gaping wound, being a startup your code is in perpetual beta and probably unstable and flaky at the best of times which will likely cause you to spend all day dealing with support rather than shipping product. That is unless you’ve got a plan.

The plan I’m going to share with you contains no silver bullets but is an accumulation of things I’ve learned the hard way. You may have even heard these things before however I will attest that these *actually work* and are free and clear of any marketing FUD.

  1. Fabric (or Capistrano et. al.) is your very own MacGyver. Use it to automate everything. Many teams get away with not automating tasks simply because they’re overstaffed. They kid themselves saying “it’s only a small thing” or “it won’t take that long”. When it’s just one guy you don’t have the luxury of doing things manually. In fact your large team doesn’t have that luxury either but you’re on the highway to gridlock and don’t even know it. It may seem like you’re taking a hit knocking up a quick Python script to do a relatively simple task but you’ll sigh with relief when things go bump in the night and you’ve got a veritable trove of tools at your fingertips to get you back on track. Best of all you can write these scripts while you’re lucid, in-the-moment and know exactly how system x or task y works. This has two main benefits: implicit documentation and can be executed safely when you’re completely wiped out trying to resolve an issue at 3am.
  2. Jenkins for scheduling. Forget Cron unless your needs are so anemic that you really can deal with either silent job failure or an inbox full of cryptic error emails. Many write off Jenkins thinking it’s just a Java thing but it’s so much more useful than that. It can connect to source code repos, run any script and alert you when jobs go bad. We use primarily for running scheduled Fabric scripts and continuous integration across our codebase.
  3. Use Loggly. Seriously just do it. It’s about half as good as Splunk feature-wise but priced pretty reasonably assuming you don’t want to keep your data for long. If you’re still concerned about the price just consider how much time you’ll waste trying to track down issues across thousands of servers. Don’t even get me started on writing your own log aggregator, you don’t have time for that kind of shit.
  4. Choose a hosting provider with an awesome API. It might cost a little more but it’s absolutely worth it. Somewhat related to point 1, your ability to automate everything is heavily reliant on the functionality that your hosting provided supplies. AWS and Rackspace are good choices here.

Follow those four tips and you’ll be well on the way to having a lean team that’s able to scale with your infrastructure. In a future post I’ll provide some more detail on how we handle provisioning and configuration management our servers using only Fabric.

Tagged , , , ,

Fixing my broken MacBook’s DVD drive

Recently the DVD drive on my late 2008 unibody MacBook stopped accepting disks. It was as though there was something blocking the entrance to the drive. Initially I tried a bunch of suggestions (such as drutil eject, PRAM reset etc.) over on Mac Rumors and the Apple support forums to no avail. After a while of looking for solutions I resigned myself to the idea that since it’s an older machine it’s probably about time that stuff started breaking. Since I don’t use it all that much I didn’t think much more of it.

Today however I wanted to setup the MacBook to watch DVDs on for my wife who isn’t feeling too well. Having given up on software fixes I decided it was time to get out the screwdrivers. Following this guide on YouTube showed me how to remove the drive so I could get a good look at it. Once the drive was out the culprit was immediately obvious. A small strip of metal that usually prevents a disc from being inserted into the drive when one is already loaded (I’m calling this the gate) had somehow jumped over the little tag/retainer that keeps it in place. Ironically the tag that is supposed to stop this from happening was the very thing now stopping the gate from moving back out of the way and into it’s home.

The solution was rather simple*. Get a small pair of needle nose pliers and gently lifted the gate back up and over the tag. There’s not much room in there so you may have to gently lift the top of the drive housing while doing this to give the gate enough clearance to move back over the tag. Once it’s over the tag let go of the gate and it should snap back into place. This worked brilliantly for me and I now have my DVD drive back!

*A word of caution: doing this kind of thing will void any warranty you have. If your warranty is out like mine or you just don’t care then happy fixing!

Tagged , , , , ,

Fear is the mindkiller

“I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration.”

—Frank Herbert, Dune (Excerpt from the Bene Gesserit Litany Against Fear)

It won’t come as a surprise to those that know me well that I struggle almost daily with fear and anxiety. It begins first thing in the morning, right from the very first moment consciousness returns and I become aware of my surroundings. With my eyes still closed thoughts of what lies ahead flood my mind and with it comes the fear. I hear the blood start rushing through my ears and feel the skin around my eyes tighten. Adrenalin surges through my body making any thought of returning to sleep impossible. Most of the time this conveniently occurs around the time I would normally get up, but every so often it happens in the wee hours of the morning or before I go to sleep. I dread the next morning because being tired only makes it worse.

For what seems like the longest time I struggled to even acknowledge I was afraid. Until recently if you had of asked me “what are you afraid of?” I would have told you with a straight face “not a whole lot”. Having joined the Army Reserve at 17 and forged my own way though the world achieving modest success, I thought I was pretty tough. Certainly not someone who was anxious or afraid of anything. So what exactly was it that made me acknowledge my fear? Putting it simply; being out of my comfort zone.

Pre-startup life was pretty good in many ways. I had a respectable job, a steady paycheque, living in one of the most beautiful cities in the world (Sydney) and great friends. My wife and I owned an apartment, a new car and could afford to eat out whenever we wanted. Actually I take it back. Life was excellent! Fast forward to today and all the security has been removed or changed in a dramatic way. Such is the life of an entrepreneur. Now don’t get me wrong, I’m not complaining. It was my choice to do this and gladly I made it. What I am saying is that if there’s a recipe for revealing fear in your life then the above comes pretty close to perfect in helping you find it.

Recognising fear

One of the first challenges I faced in overcoming fear is recognising it at all. It sounds strange but even though I was fully caught in the throes of fear I was largely unaware of it. What I was aware of were symptoms.

Most notable of all my symptoms was the effect on my relationships, particularly with my co-founder Dave Trindall. It all came to a head one morning after I’d received some bad news from one of our investors via email. They were questioning their investment in us and were threatening to pull out their money. The news caught me off guard and I flipped out. The resulting argument with Trindall was pretty ugly but then something great happened. Trindall called me on my bullshit. It was the first time in positively ages that someone had brought the searing hot sword of truth to bear on me. To be honest I didn’t like it one bit. It hurt too much, left me exposed and undermined my view of myself as being tough.

My relationship with Trindall wasn’t the only thing that was suffering though. Bit by bit fear was eroding my ability to concentrate. My ability to perform even simple tasks quickly and effectively was seriously compromised. Focus was almost impossible when each moment a new thought of some looming deadline with terrible consequences would pop into my head and demand my attention. If you’ve been there you’ll know how frustrating, not to mention exhausting that is. Each day I would try to rev myself up, try to get excited about what lay ahead and end up burnt out about 2 hours later.

Another thing that was dying was the vision. When you’re held in thrall by fear it becomes nearly impossible to imagine a bright future. This is absolutely essential to generating new ideas, getting excited and maintaining confidence. As I allowed these thoughts to dominate my mind the light of hope was slowly being extinguished. I call this the death spiral. Once you start believing your doomed your mind ceases to offer up helpful suggestions and ideas about how to improve things. All I could visualise was my life slowly circling the drain. Pretty bleak really.

Breaking out

It actually took a while for me to do anything about my fear, even after I recognised its presence. Honestly I didn’t really know what to do about it. One thing I did do was start talking about it with people I trust. That was enough to get the ball rolling. It began slowly but what I realised along the way was that I didn’t possess the answers or the skills to beat this on my own. Thanks to my wife, a very patient co-founder, various other family members and friends I am slowly piecing together a plan.

Making a plan

While I’m certainly not out of the woods I’m feeling better than I have since I can’t remember when. I want to share with you three activities that are making a huge difference in my life today:

  • Prayer & meditation
  • Exercise
  • Journalling

If you haven’t tried meditating then you really should. Done right, it’s one of the most amazing things you’ll ever discover. I’ve balked at meditation before because the last thing I wanted was to be alone with my thoughts; there be dragons! As usual however I was doing it wrong. Trindall introduced me to a secular form of Zen meditation (which fits with my Christian beliefs) that showed me how to avoid being trapped by my thoughts. Simply allowing them to be and letting them pass by, good and bad, without being distracted by them. Where before I would fight these thoughts and expend vast amounts of energy just to stay focused I’m now learning to simply let these thoughts come and go without giving them the time of day.

Once I’ve finished meditating (which only takes 20 mins max) I head out for a walk. It’s such an amazing feeling to just take in the morning once I’ve cleared out all the rubbish in my mind. Just 30 mins has me feeling reconnected with the world and puts things into perspective. Given the crazy hours I often work I’ve noticed that it’s easy to start feeling claustrophobic and distant from people when I don’t get out into the world.

Journalling caps it all off by allowing me to reflect on and take note of what’s changing. I note what new things are happening and also the passing of things out of sight. It’s a great opportunity to also explore your feelings and opinions about things without hurting anyone. I find that so many of my thoughts are only half formed. Writing them down helps me find the holes and rationalise what’s on my mind. Mountains often become molehills once I reckon with them on the page.

As a parting thought I’ll leave you with this:

Your mind is like a garden and you are the gardener. It’s up to you what you allow to take root and flourish there. Do you cultivate your mind to grow beautiful things or do you let the weeds take over?

Tagged , ,

Why I still love SMS

Personal communication has been on my mind a lot lately. I believe that this is partly because, in the sea of uncertainty that a startup is, focus can be difficult to achieve and maintain. There are a plethora of things to distract you that you can internally justify as a legitimate use of your attention. Email is probably the worst of these distractions for me. I could easily spend a whole day (and sadly have) working through emails feeling productive but not actually achieving anything of significance. Frankly it’s an awful waste of a startup’s most valuable resource; time.

Consequently in an attempt to gain focus I’ve done things like email blackouts. This is only part of the distraction cycle though, to make myself truly productive I also need to block out twitter, facebook and LinkedIn. With all these shut out not surprisingly my focus improves dramatically but it creates another problem; how can I be reached in the event of an emergency or something else of sufficient gravity?

The title of this post of course gives it away: SMS. Of all the ways to reach me SMS is still superior by a wide margin and I’ll tell you why:

  1. It’s “push” by default;
  2. It operates worldwide;
  3. It works great in areas with poor reception, and;
  4. The noise to signal ratio is extremely low.
The last one is perhaps most significant for me. Many have bitched about the high cost of data in SMS, though I’ve come to view it as a feature over the years. As a kid who grew up counting the cost of each text, my friends and I were sparing in the volume of texts we sent.  It’s burned into my subconscious to be judicious in my use of texting. Moreover it’s rare and considered bad form (at least in the circles that I move) to send texts to randoms or even mere acquaintances. This is most certainly not the case with social networks or emails.
With the introduction of iOS 5’s iMessage feature (plus similar features on other platforms such as Android) and the near ubiquity of unlimited texting plans I wonder if later generations will share the same respect for the medium and continue to reap the benefits that I am today.
How do the texting habits of you, your friends and colleagues compare to mine?
Tagged , , , ,