Almost all of my favourite posts over the last few years have been tech companies posting about what actually powered theirs sites and servers.   Reading their stuff has really helped me get my head around how to do some really really complex things, as well as how to dodgy a few bullets in the really simple stuff too (have a look at how Flickr serves up unique id numbers for example.)

So, after being here at wishlil.st for a little over a month, I thought it was time to get some of the little things we’ve done out into the world, and maybe help a few other people.  Developing Facebook applications has it’s own interesting infrastructure problems, and whilst we’ve not solved everything (there’s no equivalent of using a local MAMP install as I would for website development for example) we have solved a few things.

Using a code framework

We’ve ported all of our facebook app across to run on top of the amazing CodeIgniter.  As a PHP house it was all to easy to hack about in our own handrolled structure, but there comes a time to bite the bullet and use an MVC structure.  We chose CI because I had worked with it in the past, and loved the way it didn’t get under your feet.   It’s lightweight, and nicely flexible, but comes with lots of freebies for developers.

Getting CI to play nicely with FB will get a separate post later (you have to set lots of config right to stop FB killing your app with it’s crazy request strings) but the library from Elliot Haughin was a good starting point.  We’ve hacked it about a little and replaced it’s core FB wrapper with the latest one, but it got us off the ground.

Version management

We started with a Subversion repository on Beanstalkapp.com running a trunk for live, and a branch for development work.  Once we got Coda integrated with SVN and SFTP (you have to edit on a “live” server to develop FB apps) it was all running pretty smoothly.  I build a couple of deployment scripts, and all ways well until we had to merge the dev branch back into live, and then the version management hell started.  Merging is hard, should be hard, and will always be hard, but it shouldn’t be this hard.  We bit another bullet, and moved to Mercurial.

We’d heard a lot about it over the last few months and it was a quote from Joel Spolsky that encouraged us to go for it…

Or, to put it another way, I’d go back to C++ before I gave up on Mercurial. -Joel Spolsky

In the end it wasn’t that hard.  We’ve taken up a hosted Mercurial account with Kiln to get started (I’ll in-house this at some point) with a single repository for live.  We’ve branched this for dev.

Importing SVN to Mercurial turned out to be reasonably easy.  You create your local clone of the live repository (which is empty), run an “svn export” to flatten you SVN trunk of it’s history, and then “hg add” “hg commit” and “hg push”.  Job done.  The HGInit guide is a good starting place too.

So now we bug fix live on the equivalent of trunk, and “push” branches back up to the trunk when we’re ready to go live.  Command line all the way, but that’s just how we roll.

Deployment scripts

One of the big lessons I learned, and Flickr’s notes re-enforced as the need for easy deployment.  Gone are the days of me spending 3 days deploying code into a live environment, living on coffee and Haribo for a week and risking it all.

You really must be able to deploy the majority of your environment from a single script, and thus a single mouse click.

At the moment we do it in the easiest dirtiest way possible, we “hg clone” the repository into a fresh directory, and then symlink to it from the Apache document root.  You don’t even need to restart Apache (but it’s a good idea).  It’s a single command (./push_live.sh) and takes about 30 seconds.

A dose of monitoring

Having an idea of what is trending on your server is really really important.  I love Munin for it’s chronological view of the world, letter you see what has happened in the short term (why have we got 4 gigs more memory committed that 2 minutes ago) and whats going on over a more meaningful time frame (oh – we do this everyday about the same time – might be connected to a scheduled process).  Munin lets you be proactive, spotting things that are creaping up over time.  It’s also handy during a crisis.

Oh man - what's going on?

Oh no, it's ok.

Munin is a good partner to Nagios.  I’ve not got Nagios up and installed yet.  Munin lets you stop things going wrong, Nagios will tell when they have (and it’s a amazing how often you don’t know).

Development scripts

There are lots of things you end up doing during dev which can really benefit from a drop of scripting.  One we run several times a day on our personal instances, and every night on a schedule, is cloning our live database across to the development instances.  It’s not hard, in fact it’s a one liner really.  We run mysqldump to drop the database out to sql files, and then mysql < dumpfile.sql to import it into the dev databases. Call it and you can have a fresh new cut of live with all the crazy data people have sneaked past the filtering for you to play with.

Backups

Backing up is a pain in the neck, but so import it’s unreal.  Eventually I’d like to get to a point where we can deploy everything including server config form a central point, but until then we backup two things, the code and the data.

Database backups are a simple matter of mysqldump being run, and a little bit of tar, gzip and scp to bundle them up with the date and move them off site.  We keep an on site copy too for fast access though.

The live code is backed up using rsync to a mount point that is mounted before the rsync and unmounted afterwards (no accidental rm -rf).  Getting it to work well was a bit of a nightmare, and we only had FTP access to the backup space, but curlftpfs came to the rescue letting us mount a remote ftp connection as a local mount.

A bunch of silly things that save you time

If you find yourself doing the save thing ten times a day, stop it, and script it.  You remove the chances of you screwing it up because you’re tired.  Here’s a handy guide to get you started, and do remember to make yours scripts check their input before they run, kickout if something goes wrong, and log (so you can tell what went wrong)

Set yourself up  SSH keys between your trusted boxes and laptops.  Mistyping a password at 1am is frustrating, and sorting your passwords in a text file is worse.  It’s not hard , though it does vary between install types but Bush Cures Cancer had a great guide.

We’ll write up some more detailed insights into some fo the stuff we’re doing with time, but if you’re stuck, drop us a line.