By far one of the most rewarding projects we have ever delivered also happened to be one of the smallest. We met Bunnie at Linux Conf AU in January 2013, where his keynote mentioned some of the difficulties he was having with providing new developers with resources for the Novena open computing platform. If you’re into hacking and hardware Bunnie writes an awesome blog and recently Make published this article on the Novena laptop. We took Bunnie’s concerns on board and worked out what we think is the neatest little private cloud deployment for he and his team.
The Challenge
In Bunnie’s words:
The core need was the ability to lower the barrier to acquiring new developers. Creating a cross-compilation environment for Linux is a slow and painful process. Even when everything runs flawlessly, building a bootable image from source takes about 8-12 hours on a well-appointed system and a 100Mbit fibre link. Most unpaid developers find this to be an insurmountable barrier.
The basic idea is to create a provisioned cloud image that a developer can simply take over, so that all the basic packages, kernel, and disk image are pre-built. This allows them to add their unique contribution without having to “build the world”. Amazon EC2 was originally used to provide this solution, but the workload of platform builds isn’t a good match to their system. A fast, multi-threaded build will consume gigabytes of memory, requires fast disk I/O, and churn through 70GB of disk space before pruning back to a core of about 10GB — but this resource consumption level only happens in short bursts. We tried going with the discipline of stopping instances after builds, but when you forget and leave the tap running, the hosting bills were painful.
Furthermore, getting other people involved meant asking them to surrender their credit card to Amazon and fork out $50/month for their instance — again, this wasn’t lowering any barriers.
Getting a private cloud in allows us to create well-provisioned build instances on hardware we own, and hand out instances to developers for free, at least to try things out initially; if they get serious, they can then move to an EC2 instance of their own. In particular, the ability to export from an Aptira instance to EC2 is important to us, because our goal isn’t to host all builds for our system, it’s simply to lower barriers so interested/curious people can try it out, without creating a prohibitive barrier to our cost of developer acquisition.
The Aptira Solution
Aptira used our OpenStack puppet modules to run up a single node private cloud capability alongside existing open source packages which were already running on the server. All OpenStack API and dashboard services were secured with SSL. Backend stack consisted of MySQL, RabbitMQ (SSL), KVM with mdraid backed filesystem backend for glance and nova. After discussing with Bunnie, we agreed FlatDHCP nova-network would meet his requirements.
Shortly after we scaled the cloud to a second node which provided extra compute capability and an (OpenStack Cinder) volume as a service capability.
With Aptira’s proven experience in deployment efficiency of all scales, the total project time was approximately one man day worth of effort (mostly ensuring OpenStack would play nice with other services) and is rock solid.
The Results
In Bunnie’s words: We’ve been using our private cloud exclusively to do builds for our system internally, and we have had some success getting developers to use our system as well. The biggest challenge actually has now come to the fact that users have to set up a VPN into our network to access the instances — due to limitations of our network hardware, it’s been a bit cumbersome and we’ve experienced developer drop-out in part because they can’t just ssh to a public IP address. We’re considering other solutions, such as IPV6, which is starting to gain enough traction among our target demographic of developers, but we haven’t figured out all the details of how to execute this idea.
We were also able to successfully clone our Aptira instances to EC2, and through that some users who are already EC2 users or have their EC2 bills paid by a third party have cloned our AMI as a starting point for their own work.
Overall, the cluster has been stable, although since we do operate in an “informal” environment — our servers just sit on the floor of our office with no UPS — the occasional power-outage (typically brought about by the cleaner pulling the wrong cord or someone plugging a 120V-only device into our 220V sockets) has proven to be challenging at times, but Aptira has been able to help us through a couple tight spots where critical instances went missing.
