For the first time ever, instead of just speaking, I’ll be running a 2 day tutorial/hands-on workshop on how to scale a webserver/site. Both intensive performance tuning, and the architectural considerations that come with high scalability.

Based on what it takes to performance-tune;

• a single webserver to handle over 80,000 concurrent downloads
• a website to many thousands of SSL connections per second

and the thought, state-avoidance, cacheability and expandability that goes into turning;

• clusters of slow dynamic back-ends into a speedy and responsive user experience

Everything from the basics of configuration and benchmarking to high level monitoring will be covered.

It should be fun! There might be a small bit of theory; how virtual memory managers, file-systems and schedulers work, but the emphasis will be on the practical and experimental. Attendees will be given remote machines for server and client testing, and by going and doing it, we’ll step through the configuration, test and tune iteration cycle.

Let me know if you have any suggestions!

The presentation wasn’t exactly well-prepared, so there are only 13 slides, but for the record … here they are. And yep, even the Hello World example contains a bug!

The prize and length of time seems disproportionate with the challenge though, I don’t think it will be as hard as the ZD-net piece speculated to find zero-day flaws when sufficiently motivated. I do think it’ll easily degenerate into arguments about the nature of the flaws and their severity more than anything else, but still it’s a good thing to see.

It’s about 6 years since the last time I was in Brussels (when I was here to play trad music in a series of very fun gigs), but it’s hardly changed, it’s a surprisingly easy city to navigate. I did get a very dodgy hotel key card though …

More photos in my FOSDEM flickr set. There’s a bunch of us here from Joost and it’s very useful to come and get a perspective on what else is happening in the Mozilla world.

It’s been a very strange, mixed and extremely busy month for me. I’ve squeezed a lot of travel into a very small month, my Dad was in hospital for a bit, Nóirín got engaged, the already hectic pace of work at Joost is now picking up even further (it’s pretty cool that we signed Viacom, but that also boils down to a lot of work to get done!), my mother came to visit me in Leiden, and other things have happened which give me good cause to be happy.

It’s been both my worst and best month since moving to the Netherlands, and until we launch (and go out of beta) and for a good while after that, I think it’s going to be even more of the same. It’s also about the half-way point of my (planned, at least) stay in the Netherlands, and there’s still plenty of places I haven’t managed to see yet, which is making me think!

Still, I’m having a great time, I’m meeting an insane amount of people, getting to see a great variety of places and am spending an awful lot more time out and about than I would be in Ireland, so I’m making sure I’m making the most of it too. Now, I just need a teleportation device.

The keynotes are going to be great too, we have Canonical and Ubuntu’s Mark Shuttleworth to give the opening plenary, and – somewhat unusually – we’re going to have Microsoft Ireland’s Robert Burke delivering a sponsored keynote on Microsoft Atlas, their AJAX framework (which is permissively licenced).

There’s going to be a hackathon, and a docathon too, which Noirin is particularly looking forward to helping with. Sun are very kindly sponsoring Noirin to be at the Con, and are putting her up in the hotel, so I’ll be staying with her, despite only living about 10 minutes walk away. Now that’s lazy!

It’s not too late to register for the con, and if you’re based in Ireland, you can still just turn up as late as the Wednesday or Thursday and pay there and then. If you havn’t been to an Apachecon before, keep an eye on the wiki for the KeySigning event, as that’s one of the best ways to get to know people (it’s what I did!).

On a completely unrelated note, a new Lidl opened right around the corner from my parents’ house, where I grew up. As usual, it’s got some cool deals, and I bought some shoes. Normally I dislike shoe shopping, to the point where my last two pairs have been bought online. But the lidl shoes are great, really comfortable and they seem durable and well made. Best of all, they cost me a mere €12.99!

The application process was gruelling, with rigourous peer evaluation and feedback at each stage. Strong input from the teams I’ll be working with was, of course, prioritised. Challenge after challenge, test after test, and some of the toughest technical problems I have faced in my career. And that was just the screening process. The real emphasis was on how I behaved as a person, how I interact with the team, could I join in in the community, would I fit in? The usual story these days. I guess no company wants to be lumbered with bad apples.

Naturally I’ve always held a very strong interest in the company. I’ve been using their products as long as they’ve been around, and I’ve helped them out on a freelance basis here and there for the last 4 years. I even had a very small part to play in a project they later aquired, such is the nature of the market these days.

It’s yet another non-profit, so the benefits are pretty terrible and the salary is even worse, but the possibilities and prospects are pretty great. It’s an amazingly exciting opportunity for me. I don’t think it’s hyperbole to say that some of most gifted people in the field are all working here.

Of course, it does come as a great pity that it happens just a few weeks before getting my new HEAnet business cards, but I’ll be equally proud of my new card and my new title:

Update: Despite the “humour” tag, it appears I may have taken the satire ever so slightly over the edge and managed to confuse quite a few people with this post. To be clear; I am still, and will continue to be, gainfully employed by HEAnet, but am now a member proper of the Apache Software Foundation.

Incidentally, if you’re thinking of going to Apachecon Europe, you should register soon, as Mark Shuttleworth is going to be giving the opening keynote and a session on the business track. I’m very much looking forward to hearing him speak again.

He also pointed me at Dave Miller‘s latest kernel with T1 fixes, which I’ve built from git, and showed me the way to the libc6-sparcv9v and libc6-sparc64v packages, which contain runtime optimisations for the platform. And the result is stunning. Ubuntu is now outperforming even Solaris express, and we’re sustaining 22,183.43 requests per second – using out of the box Apache 2.2.0. Not a single kernel tuning, Apache tuning, or anything beyond “CFLAGS=-Os” applied.

Frankly, I’m amazed. Amazed enough that I’ve rebooted into Nevada twice now just to confirm it’s not a change in the test environment. This machine just gets better and better, and Linux/Ubuntu really helps it get there. Of course things like hardware SSL acceleration don’t quite work in Linux yet, but I’m sure it’ll get there.

Now I know we’re going to be buying a few of T1 boxes this year, and although I’ll be using Solaris for debugging and development work (where it is a superb environment), it’s looking less and less attractive for production deployments.

The debian-install system isn’t as optimised for 9600 baud serial as the Nevada or Solaris 10 installers were, so it is occasionally annoying to sit through the screen refresh, but overall the install took much less time (about 30 minutes, compared to about 1 hour and 30 minutes) and worked without any big problems.

The ubuntu sparc distribution is a bit hairy right now, and I wouldn’t recommend it for anyone who isn’t very experienced with linux. For example, out of the box, /dev/null, /dev/zero, /dev/random and more have the wrong permissions – which breaks a lot of things – and the dummy start-stop-daemon binary has been left installed, which means that almost no init scripts actually work.

However, for all that, apt does work – and it kicks the living daylights out of anything Solaris has to offer in the way of patch or package management. It took me about 3 minutes to get a fully working compile environment up and running, and it took about 20 seconds to get the box fully up to date. Sun badly need this kind of ease-of-deployment, or there’s simply no way people like me could consider deploying Solaris on hundreds of machines. We keep over 100 Debian boxes up-to-date in about an average of 15 minutes work per week, using a combination of apt, sudo and our own apticron. Solaris simply has nothing to compete.

Ok, problem number 1 is that Solaris isn’t a distro, so it can’t really provide updates for most packages, but there isn’t even a credible way of upgrading the native Solaris-bundled stuff. Solaris may be great at providing the tick-box-compatible certification neccessary for running many commercial applications, but it really is dire when compared to a good Linux distro in terms of ease of administration. It’s literally a waste of time. Everyone in Sun who works on Solaris should be forced to abandon BFU and start using what their customers have to deal with. Allow to bake for about 10 weeks, and out will come apt.

Anyway, back to Ubuntu;

colmmacc@murphy:~$ cat /proc/cpuinfo
cpu             : UltraSparc T1 (Niagara)
fpu             : UltraSparc T1 integrated FPU
prom            : OBP 4.19.0 2005/10/27 17:24
type            : sun4v
ncpus probed    : 32
ncpus active    : 32
D$ parity tl1   : 0
I$ parity tl1   : 0
Cpu0Bogo        : 2001.08
Cpu0ClkTck      : 000000003b9aca00
Cpu1Bogo        : 2000.02
Cpu1ClkTck      : 000000003b9aca00
Cpu2Bogo        : 2000.02
Cpu2ClkTck      : 000000003b9aca00
Cpu3Bogo        : 2000.01
Cpu3ClkTck      : 000000003b9aca00
Cpu4Bogo        : 2000.03
Cpu4ClkTck      : 000000003b9aca00
Cpu5Bogo        : 2000.03
Cpu5ClkTck      : 000000003b9aca00
Cpu6Bogo        : 2000.01
Cpu6ClkTck      : 000000003b9aca00
Cpu7Bogo        : 2000.02
Cpu7ClkTck      : 000000003b9aca00
Cpu8Bogo        : 2000.03
Cpu8ClkTck      : 000000003b9aca00
Cpu9Bogo        : 2000.03
Cpu9ClkTck      : 000000003b9aca00
Cpu10Bogo       : 2000.02
Cpu10ClkTck     : 000000003b9aca00
Cpu11Bogo       : 2000.02
Cpu11ClkTck     : 000000003b9aca00
Cpu12Bogo       : 2000.03
Cpu12ClkTck     : 000000003b9aca00
Cpu13Bogo       : 2000.02
Cpu13ClkTck     : 000000003b9aca00
Cpu14Bogo       : 2000.02
Cpu14ClkTck     : 000000003b9aca00
Cpu15Bogo       : 2000.02
Cpu15ClkTck     : 000000003b9aca00
Cpu16Bogo       : 2000.03
Cpu16ClkTck     : 000000003b9aca00
Cpu17Bogo       : 2000.02
Cpu17ClkTck     : 000000003b9aca00
Cpu18Bogo       : 2000.02
Cpu18ClkTck     : 000000003b9aca00
Cpu19Bogo       : 2000.02
Cpu19ClkTck     : 000000003b9aca00
Cpu20Bogo       : 2000.03
Cpu20ClkTck     : 000000003b9aca00
Cpu21Bogo       : 2000.02
Cpu21ClkTck     : 000000003b9aca00
Cpu22Bogo       : 2000.02
Cpu22ClkTck     : 000000003b9aca00
Cpu23Bogo       : 2000.02
Cpu23ClkTck     : 000000003b9aca00
Cpu24Bogo       : 2000.03
Cpu24ClkTck     : 000000003b9aca00
Cpu25Bogo       : 2000.02
Cpu25ClkTck     : 000000003b9aca00
Cpu26Bogo       : 2000.03
Cpu26ClkTck     : 000000003b9aca00
Cpu27Bogo       : 2000.02
Cpu27ClkTck     : 000000003b9aca00
Cpu28Bogo       : 2000.03
Cpu28ClkTck     : 000000003b9aca00
Cpu29Bogo       : 2000.02
Cpu29ClkTck     : 000000003b9aca00
Cpu30Bogo       : 2000.02
Cpu30ClkTck     : 000000003b9aca00
Cpu31Bogo       : 2000.02
Cpu31ClkTck     : 000000003b9aca00
MMU Type        : Hypervisor (sun4v)
State:
CPU0:           online
CPU1:           online
CPU2:           online
CPU3:           online
CPU4:           online
CPU5:           online
CPU6:           online
CPU7:           online
CPU8:           online
CPU9:           online
CPU10:          online
CPU11:          online
CPU12:          online
CPU13:          online
CPU14:          online
CPU15:          online
CPU16:          online
CPU17:          online
CPU18:          online
CPU19:          online
CPU20:          online
CPU21:          online
CPU22:          online
CPU23:          online
CPU24:          online
CPU25:          online
CPU26:          online
CPU27:          online
CPU28:          online
CPU29:          online
CPU30:          online
CPU31:          online

Nice! But how does it perform? Surprisingly well actually. Linux performs much better on the single-threaded I/O test. Again, I’ll hold off on the graphs, but the dd tests under Linux were roughly 10 times faster than Solaris. In rather an odd way, this didn’t translate into better single-download performance though – instead, under Linux we could only push about 60MB/sec.

And for the number that really matters; requests per second. Linux managed a neat 18,210.57 requests per second, which is within 10% of Nevada and more or less identical to Solaris 10. I should point out that the ridiculous Linux OOM killer did rear its ugly head during our more insane scalability testing though (try starting 500,000 threads due to a typo and you’ll find out all about it!). Solaris handles OOM much more gracefully IMO.

Another note is that “make -j 64″ under Linux built Apache (and apr, apr-util) in about 2 minutes, compared to 5 minutes for “dmake -j 64″ under Solaris, but that could be due to almost anything, it probably isn’t indicative of better FLOP performance. All in all, I’d say that Linux – which only started working on T1 a few short weeks ago – compares reasonably favourably with Solaris on the T2000. I’d also say that the T2000 is even better value for money with this information in hand, because it presents a greater range of options.

I wouldn’t rush out and run Linux on the box in production quite yet though. Ubuntu-sparc still needs some work, and there are doubtless many T1 kernel bugs yet to be found and ironed out. dtrace still represents a huge win on Solaris, and when the T1000 arrives, I can see us running dtrace dozens of times a day – it’s already helped me determine a huge amount of information useful for tuning Apache, and for reworking some of the code. But that said, for a production environment, once the Linux kernel and the Ubuntu distro get a bit more stable, they would be my personal choices for the T2000. The working day is just too short to sacrifice the wins of apt.

This is probably my last really large post on the T2000, as in a week, I’m going to drive it over to RedBrick, but it’s been great fun benchmarking the box, and I’m very, very grateful to Sun – for both their kind donations, and the opportunity to test the platform. It is very, very impressive kit. My Scaling Apache talk is going to be on at ApacheCon EU 2006 (check out the new logo btw), and I’ll be including some more detail there too, including some of the more useful information we’ve gleaned from using dtrace (I’m currently working on a per-nanosecond break down of an Apache request – how cool is that!), so do come along to that.

Before upgrading, we applied some more tunings that found on the T2000 SpecWeb page, but they didn’t make much difference it has to be said. As I previously blogged, regular Solaris 10 managed to push 15298.68 requests per second. Well straight out of the box, Nevada was pushing over 18,000 – which is a 20% improvement, not bad! The kernel bug which caused the event MPM to crash is now fixed, and using the event MPM gets us to … drum-roll … 20,417.33 requests per second. Better yet, the really nifty kssl functionality on Solaris means that you can use the event MPM without having to worry about its lack of support for mod_ssl right now.

Although the single-threaded I/O numbers have not really improved, Nevada is better for single-threaded downloads, and here we see a neat doubling of performance and the T2000 has no problem maintaining 80MB/sec (that’s 640 Mbit/sec) downloads, which is more than good enough.

One funny thing I have noticed is a bug in the T2000 firmware;

Sun Fire T200, No Keyboard
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.

A google search for it seems to indicate it’s universal. But I can’t find a bug ID :-) Maybe the product line got renamed, or maybe it took legal a while to negotiate with the Terminator people, or maybe it’s just a typo, but either way I like it, it’s fun to have a few anachronisms, it adds character.