/~colmmacc/ » coding

Weighty matters

colmmacc — Sun, 15 May 2011 03:49:57 +0000

One of my favourite programming exercises is to write a text generator using Markov Chains. It usually takes a relatively small amount of code and is especially useful for learning new programming languages. A useful goal is to able to have a tool that ingests a large body of text, breaks it into N-tuples (where N is chosen by the operator), and then emits a random text of length L where that text uses those tuples at the same frequency that the original body did.

On its own, that task can produce some fun results, but it’s also a very repurpose-able technique. You can use this kind of statistical generation to simulate realistic network jitter (record some N-tuples of observed RTT with ping), or awesome simulated-user fuzz tests (record some N-tuples of observed user inputs). It’s surprising that it isn’t more common.

But when approaching these problems, from experience of working with newcomers, what seems to be a common first tripping point is how to do weighted selection at all. Put most simply, if we have a table of elements;

element	weight
A	2
B	1
C	1

how do we write a function that will choose A about half the time, and B and C about a quarter each? It also turns out that this is a really interesting design problem. We can choose to implement a random solution, a non-random solution, a solution that runs in constant time, a solution that runs in linear time and a solution that runs in logarithmic time. This post is about those potential solutions.

Non-random solutions

When coming to the problem, the first thing to decide is whether we really want the selection to be random or not. One could imagine a function that tries to keep track of previous selections, for example;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

elements = [ { 'name' : 'A' , 'weight' : 2 },
{ 'name' : 'B' , 'weight' : 1 },
{ 'name' : 'C' , 'weight' : 1 } ]

def select_element(elements, count):
total_weight = 0
for element in elements:
total_weight += element['weight']
if count < total_weight:
return ( element ,
(count + 1) % len(elements) )

# select some elements
c = 0
(i , c) = select_element(elements, c)
(j , c) = select_element(elements, c)
(k , c) = select_element(elements, c)

This function uses the “count” parameter to remember where it left off last time, and runs in time proportionate to the total number of elements. Used correctly, the first two times it’s called this function will return A, followed by B, followed by C, followed by A twice again and so on. Hopefully that’s obvious.

There’s a very simple optimisation possible to make it run in constant time, just sacrifice memory proportionate to the total sum of weights;

1
2
3
4
5
6
7
8

elements = [ { 'name' : 'A' },
{ 'name' : 'A' },
{ 'name' : 'B' },
{ 'name' : 'C' } ]

def select_element(elements, count):
return ( element[count] ,
(count + 1) % len(elements) )

This basic approach is surprisingly common, it's how a lot of networking devices implement weighted path selection.

Random solutions

But if we're feeding into some kind of simulation, or a fuzz-test, then randomised selection is probably a better thing. Luckily, there are at least 3 ways to do it. The first approach is to use the same space-inefficient approach our "optimised" non-random selection did. Flatten the list of elements into an array and just randomly jump into it;

1
2
3
4
5
6
7
8
9

import random

elements = [ { 'name' : 'A' },
{ 'name' : 'A' },
{ 'name' : 'B' },
{ 'name' : 'C' } ]

def select_element(elements):
return random.choice(elements)

As before, this runs in constant time, but it's easy to see how this could get really ugly if there's some weights that are more like;

element	weight
A	998
B	1
C	1

Unfortunately, common datasets can follow distributions that are just like this (Zipfian, for example). It would take an unreasonably large amount of space to store all of the words in the English language, proportionate to their frequencies, using this method.

Reservoir sampling

But luckily we have a way to avoid all of this space, and it's got some other useful utilities too. It's called reservoir sampling, and it's pretty magical. Reservoir sampling is a form of statistical sampling that lets you choose a limited number of samples from an arbitrarily large stream of data, without knowing how big it is in advance. It doesn't seem related, but it is.

Imagine you have a stream of data, and it looks something like;

"Banana" , "Car" , "Bus", "Apple", "Orange" , "Banana" , ....

and you want to choose a sample of 3 events. All events in the stream should have equal probability of making it into the sample. A better real-world example is collecting a sample of 1,000 web server requests over the last minute, but you get the idea.

What a reservoir sample does is simple;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def reservoir_sample(events, size):
events_observed = 0
sample = [ None ] * size

for event in events:
if events_observed >= len(sample):
r = random.randint(0, events_observed):
else:
r = events_observed

if r < len(sample):
sample[r] = event
events_observed += 1

return sample

and how this works is pretty cool. The first 3 events have a 100% likelihood of being sampled, r will be 0, 1, 2 in their case. The interesting part is after that. The fourth element has a 3/4 probability of being selected. So that's pretty simple.

But consider the likelihood of an element already in the sample of "staying". For any given element, there are two ways of "staying". One is that the fourth element is not chosen (1/4 probability) , another is that the fourth element is chosen (3/4 probability) but that a different element is selected (2/3) to be replaced. If you do the math;

1/4 + (3/4 * 2/3)   =>
1/4 + 6/12          =>
3/12 + 6/12         =>
9/12                =>
3/4

We see that any given element has a 3/4 likelihood of staying. Which is exactly what we want. There have been four elements observed, and all of them have had a 3/4 probability of being in our sample. Let's extend this an iteration, and see what happens at element 5.

When we get to this element, it has a 3/5 chance of being selected (r is a random number from the set 0,1,2,3,4 - if it is one of 0, 1 or 2 then the element will be chosen). We've already established that the previous elements had a 3/4 probability of being in the sample. Those elements that are in the sample again have two ways of staying, either the fifth element isn't chosen (2/5 likelihood) or it is, but a different element is replaced (3/5 * 2/3) . Again, let's do the math;

3/4 * (2/5 + (3/5 * 2/3))  =>
3/4 * (2/5 + 6/15)         =>
3/4 * (2/5 + 2/5)          =>
3/4 * 4/5                  =>
3/5

So, once again, all elements have a 3/5 likelihood of being in the sample. Every time another element is observed, the math gets a bit longer, but it stays the same - they always have equal probability of being in the final sample.

So what does this have to do with random selection? Well imagine our original weighted elements as a stream of events, and that our goal is to choose a sample of 1.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import random

elements = [ { 'name' : 'A' },
{ 'name' : 'A' },
{ 'name' : 'B' },
{ 'name' : 'C' } ]

def select_element(elements):
elements_observed = 0
chosen_element = None
for element in elements:
r = randint(0, elements_observed):
if r < 1:
chosen_element = element
return chosen_element

So far, this is actually worse than what we've had before. We're still using space in memory proportionate to the total sum of weights, and we're running in linear time. But now that we've structured things like this, we can use the weights to cheat;

1
2
3
4
5
6
7
8
9
10
11
12
13
14

elements = [ { 'name' : 'A' , 'weight' : 2 },
{ 'name' : 'B' , 'weight' : 1 },
{ 'name' : 'C' , 'weight' : 1 } ]

def select_element(elements):
total_weight = 0
chosen_element = None
for element in elements:
total_weight += element['weight']
r = randint(0, total_weight -1)
if r < element['weight']:
chosen_element = element

return chosen_element

It's the same kind of self-correcting math as before, except now we can take bigger jumps rather than having to take steps of just 1 every time.

We now have a loop that runs in time proportionate to the total number of elements, and also uses memory proportionate to the total number of elements. That's pretty good, but there's another optimisation we can make.

Using a tree

If there are a large number of elements, it can still be a pain to have to iterate over them all just to select one. One solution to this problem is to compile the weighted elements into a weighted binary tree, so that we need only perform O(log) operations.

Let's take a larger weighted set;

element	weight
A	1
B	2
C	3
D	1
E	1
F	1
G	1
H	1
I	1
J	1

which we can express in tree form;

where each node has the cumulative weight of its children.

Tree-solutions like this lend themselves easily to recursion;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

import random

elements = [ { 'name' : 'A',
'weight' : 10 },
{ 'name' : 'B',
'weight' : 20 },
{ 'name' : 'C',
'weight' : 30 },
{ 'name' : 'D',
'weight' : 10 },
{ 'name' : 'E',
'weight' : 10 },
{ 'name' : 'F',
'weight' : 10 },
{ 'name' : 'G',
'weight' : 10 },
{ 'name' : 'H',
'weight' : 10 },
{ 'name' : 'I',
'weight' : 10 },
{ 'name' : 'J',
'weight' : 10 } ]

# Compile a set of weighted elements into a weighted tree
def compile_choices(elements):
if len(elements) > 1:
left = elements[ : len(elements) / 2 ]
right = elements[ (len(elements) / 2) : ]

return [ { 'child': compile_choices(left) ,
'weight' : sum( map( lambda x : x['weight'] , left ) ) } ,
{ 'child': compile_choices(right) ,
'weight' : sum( map( lambda x : x['weight'] , right ) ) } ]
else:
return elements

# Choose an element from a weighted tree
def tree_weighted_choice(tree):
if len(tree) > 1:
total_weight = tree[0]['weight'] + tree[1]['weight']
if random.randint(0, total_weight - 1) < tree[0]['weight']:
return tree_weighted_choice( tree[0]['child'] )
else:
return tree_weighted_choice( tree[1]['child'] )
else:
return tree[0]['name']

tree = compile_choices(elements)

And there we have it! A pretty good trade-off, we get sub-linear selection time with a relatively small overhead in memory.

A huffman(ish) tree

I should have included this is in the first version of this post, and Fergal was quick to point it out in the comments, but there is a further optimisation we can make. Instead of using a tree that is balanced purely by the number of nodes, as above, we can use a tree that is balanced by the sum of weights.

To re-use his example, imagine if you have elements that are weighted [ 10000, 1 , 1 , 1 ... ] (with a hundred ones). It doesn't make sense to bury the very weighty element deep in the tree, instead it should go near the root - so that on average we minimise the expected number of lookups.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

def huffman_choices(elements):
if len(elements) > 1:
total_weight = sum( map( lambda x : x['weight'] , elements ) )
sorted_elements = sorted(elements, key = lambda x : x['weight'] )[::-1]

left = []
right = []
observed_weight = 0
for element in sorted_elements:
if observed_weight < total_weight / 2:
left.append(element)
else:
right.append(element)

observed_weight += element['weight']

return [ { 'child': huffman_choices(left) ,
'weight' : sum( map( lambda x : x['weight'] , left ) ) } ,
{ 'child': huffman_choices(right) ,
'weight' : sum( map( lambda x : x['weight'] , right ) ) } ]
else:
return elements

None of these techniques are novel, in fact they're all quite common and very old, yet for some reason they don't have broad awareness. Reservoir sampling - which on its own is incredibly useful - is only afforded a few sentences in Knuth.

But dealing with randomness and sampling is one of the inevitable complexities of programming. If you've never done it, it's worth taking the above and trying to write your own weighted markov chain generator. And then have even more fun thinking about how to test it.

Period Pain

colmmacc — Mon, 14 Sep 2009 16:15:27 +0000

Imagine it was your job – along with 1,000 other people – to pick a number between 1 and 60. You can use any method you like (though you must use the same one), but if more than 30 of you choose the the same number, those of you who did would be shot. Would you let the group pick numbers at random?

Probably not, there’s always a chance it could go horribly wrong. And that chance? We could derive the correct p-value for having to shoot people easily enough from the uniform distribution, but forget that, let’s do it with code. It’s a lot easier to understand – and it’s a good habit too.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

import random

count = 0

# Simulate 10,000 runs of 1,000 people
# picking a number between 0 and 59.
for runs in range(10000):
numbers = [0] * 60

for people in range(1000):
r = random.randint(0, 59)
numbers[ r ] += 1

if sorted(numbers)[-1] >= 30:
count += 1

print count/10000.0

If you run this, hopefully you’ll get an answer that’s around “0.1″. In other words, about 10% of the time we expect to have at least one number being chosen by 30 or more people. Those don’t seem like great odds, and I wouldn’t play Russian roulette with lives like that.

Yet this is almost exactly what we do with a lot of automated tasks in some large-scale distributed systems. A pattern than can be observed over and over again is that someone writes a scheduled task that runs once an hour, day, month or whatever but then sleeps a random amount of time (typically between 0 and 3600 seconds) when it starts- in a very naive attempt to distribute impact across the larger system evenly.

The impacts can be pretty serious – it might be extended load on a dependent service, or it might simply be too many busy nodes in a cluster. Or it might be a two-day global outage of a popular Voip service. Too many things happening at once is usually a bad thing in distributed systems.

Security and anti-virus updates, apt/yum/ports repositories and auditing tools in particular seem to get this pattern wrong. Here’s a good example, from Ubuntu’s daily apt script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# sleep for a random intervall of time (default 30min)
# (some code taken from cron-apt, thanks)
random_sleep()
{
RandomSleep=1800
eval $(apt-config shell RandomSleep APT::Periodic::RandomSleep)
if [ $RandomSleep -eq 0 ]; then
return
fi
if [ -z "$RANDOM" ] ; then
# A fix for shells that do not have this bash feature.
RANDOM=$(dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -c"1-5")
fi
TIME=$(($RANDOM % $RandomSleep))
sleep $TIME
}

This is a very bad pattern – I’d go so far as to say that it’s actually worse than letting everything happen at the same time in the first place. It has 2 particularly dangerous qualities;

It increases the effective period of the task

If a task is running once an hour – it’s probably because we need to do something once an hour. That may seem tautological, but there’s subtlety. The clock-time hours we define as humans are arbitrary, by once an hour we should really mean “at least once in a 60 minute interval”, not “once between 1PM and 2PM”.

If we pick a random sleep time, we might end up running just under two hours apart. If at 1PM we pick 0 sleep seconds, and then at 2PM two we pick 3599 sleep seconds – look, we just ran two real hours apart!. Unsurprisingly the converse can happen, and we’ll run just 1 second apart, doing heavens knows what to load along the way.
The system as a whole has to cope with dangerous load spikes

Using our earlier example, If we allocated the numbers evenly, each value would get chosen only 16 or 17 times. We could plan for an impact of say 20 running at once. But as we’ve seen, if we pick random numbers every time, then 1 in every 10 runs, we’re going to have to cope with an impact of 30. That’s 50% more load, because of a one line error!

If this task is running every hour, then about 1 in every 7 weeks, we’re going to have to deal with an impact of 40 or more. And it will appear totally out of the blue, it’s a random occurrence. Nice! To take it to the extreme, there is a small – but finite – chance all all 1,000 systems choosing the same value. But avoiding this is why we are spreading the load in the first place, so why leave it to chance?

I used to run a busy Ubuntu mirror, and every day between 6:25 and 6:55 we’d see a gigantic wedge of load that could be distributed a lot more evenly. Though I think the worst of these problems have now been fixed.

The optimal fix for the problem is simple; coordinate – use a central allocator, or a gossip protocol, to ensure that every slot has at most N/M consumers.

This isn’t always possible though – Open Source security updates are usually polled from relatively dumb mirror servers, that don’t keep enough state to be able to divide load like this. But there are still two better approaches.

Firstly, you can pick a random number just once, and then reuse the same random number every time. This guarantees that the task does actually run once an hour, and it makes load predictable for the distributed system as a whole.

Secondly, you can use any uniformly distributed unique piece of data, local to the node, to choose your number. I like to use an MD5 hash of the MAC address, but even the hostname would do.

Probability-wise, the two approaches are identical, but in the real world you get fewer collisions with the latter – probably due to the similar state of entropy identical systems will be in when you mass boot a fleet of several hundred. In both cases, where aberrations emerge due to bad luck, they at least only have to be identified and fixed once. We’re no longer playing the load lottery once an hour.

But this is only half the story … what about other automated tasks with their own periods … how do we avoid colliding with them? And how do we architect timings in distributed systems in general to make them a lot easier to debug. That’s what the next blog post will be about. Stay tuned.

N+1

colmmacc — Wed, 02 Sep 2009 09:35:10 +0000

I rarely write introspective or meta blog posts, in fact I rarely even use the word “I” on this blog (one of the habits you should develop as a manager or team-member is to use the word “we” almost all of the time) … so I hope you’ll forgive this brief, obnoxious, self-centered, round-up of the last year. I’ll try to smatter it with some useful observations to make up for it.

Today it’s one year since I started working for Amazon.com, and officially moved in to my new place and hence back to Dublin. Before that, I’d been spending most of my time in Amsterdam and traveling. By just September last year, I’d managed 78 flights in 2008. In the year since, it’s been a much more comfortable, and relatively conservative, 24. Some visits to Seattle, New York, New Orleans, Barcelona, Bristol and London keeping me occupied and worldly.

I’ve learned a lot in the last year, a lot more and in very different directions, than I expected. Amazon turns out to be even more interesting than I had anticipated, and getting thrown in at the deep end early on – with a new Amazon Web Service to help build and support – has been an amazing experience. Before I started, I naively thought that scale was mostly about understanding distributed systems problems and performant designs. It is about those things, but not mostly.

In reality (well, in my opinion) the hard problems of scale are the incredible attention to detail and testing it requires, because when your rapidly-changing code is handling billions of requests – even a tiny fault that is triggered on one in every million requests will get you paged in no time at all. I’m not sure that there is any other way to learn it than having to support it for real. I recommend it.

I’m getting a chance to work on stuff I enjoy, building some big things and making some important and critical code go really really fast. It’s always nice to feel that features exist – that the universe is different in some way – because of work you’ve been involved in. That should always be the measure of progress; “How has the universe been changed?”, everything else is meta-work.

I am learning technical things at Amazon; when I started I had never written a line of perl, now I’ve written entire perl frameworks, a very basic perl compiler, a bytecode analyzer, perform low-level code-reviews, and teach perl once a week. The experience has reinforced my opinion that the notion of “knowing programming language X” is itself a broken anti-pattern.

That said, Perl was comparatively difficult to learn. Coming to it with about 15 years programming experience in many languages, it still took about 5 weeks to become proficient in it to the point that I really understood all of the magic symbols, operators and patterns in front of me at a fundamental level. At a guess, it took just 2 weeks to reach the same level of proficiency in python.

But that learning is self-driven, the real learning experience at Amazon is how things are organised and managed. How a huge multinational can structure and orientate itself such that things can happen incredibly efficiently and quickly is fascinating to observe and participate in. It’s like getting a free MBA. I can recommend that too.

On the college front – it’s been a strange year, as it was also my final year doing my BSc. in Computer Science in Trinity College, Dublin. It was tough going, mostly just to stick through it, but I think worth it in the end. Somehow came out with a first class honours degree. I’ve decided not to progress with a part-time postgrad just yet – it seems like a good time to try not doing so much college for a change, but maybe next year.

In the last year, I’ve learned two new instruments, banjo – for the fun of it – to a level where I can now keep up in sessions – and piano – to a lesser level but I can now arrange and play relatively intricate pieces. My place has worked out very well, and a year on I’m still very happy to be living here, it’s ideal. Ups and downs in my personal life in the last year have been extreme, but I’ve learned a lot from those too and am mostly the better for it.

But now that I have some more free time, I have to admit I’m not entirely sure what to do with it. I’m finding things to do with my evenings, and catching up with friends properly, but still have itches to be a bit more productive. I haven’t done as much Apache stuff as I’d have liked in the last 2 years, but the urge to write a webserver from scratch using a more functional programming oriented approach (though not necessarily a lamba-calculus derived language) is strong … and also pointless.

This past year also saw the final, conclusive, victory of sense in the Irish E-voting debacle. In short; we were correct all along, and the system has been completely abandoned. I have to admit it was a bit fun to be able to gloat about it on the radio.

So now … what next for my free time … well maybe you can help. To further compound the impression of arrogance and self-obsession I have no doubt created, it’s like this; I’m pretty clever. I’ve got above average maths, analytical and linguistics skills. I’m an expert programmer/developer and builder of things. I’m a fairly decent musician and photographer with some basic sense of style, design and composition.

Oh god there’s more! I’m politically knowledgeable, and I know how to manipulate the political and press systems and strategise (with a proven track record through two lobby groups). I’ve worked for two start-ups, and I know how to make things happen. I’m a quick learner, and I know that when something is worthwhile, realistic and interesting I can throw a huge amount of effort and pragmatism right at it. I like doing cool things for free, and earn just enough to be able to.

Now all that said, to moderate things I’m also basically an introvert and can be pretty awkward and quiet around too many unfamiliar people, am about the last person you’d ever want to take to a bar (I don’t drink for a start), know close to nothing about popular culture and have read maybe only 4 or 5 fiction books in the last 10 years. So there’s a strong counter-argument that I’m also an illiterate anti-social bore to be kept in mind here too. But thankfully, not a terrible one.

But with all of that in mind, what does need doing? Especially in Ireland/Dublin – because it would be nice to involve meeting people and getting better at that whole social thing. Any ideas? the bigger the better. Any technical, political, or social gaps that really need filling? anybody need some help? What would you, or indeed, jesus do? There are a few ideas knocking around already, I’ll be sure to update when they are more concrete. How should we change the universe?

Optimising strlen()

colmmacc — Sun, 01 Mar 2009 15:51:25 +0000

Optimisation has a bit of a bad rep these days, good advice such as Hoare’s dictum that “Premature optimization is the root of all evil”, has led to a stern outlook on adding obfuscated mess to gain efficiency. Sometimes going really really fast just makes you really really insane.

Economically, the convenience of developers has won out over efficiency. Very high-level languages are near ubiquitous. Very few people have to squeeze implementations into so many CPU operations, or such and such an amount of memory. On the contrary, click-dragging many millions of CPU operations and many millions of bytes of memory in the form of some library, form-control or icon in an even more inefficient IDE is the norm.

But as others have put far better, advocating the complete absence of optimisation is advocating a fallacy. Focused, intelligent, and crucially – well documented – optimisation is a very important skill that can save lots of time and money. To give three concrete examples;

1. Deriving the most optimal buffer sizes

At HEAnet when we were scaling ftp.heanet.ie to cope with over 50,000 concurrent downloads from a single host, we ran a lot of experiments to figure out the most optimal read buffer sizes.

This is very rarely done, most buffers are chosen arbitrarily, 4k and 8k being common for various reasons. Our experiments showed that the optimal buffer size was actually around 40k, and when we made this change within Apache we measured a 25% capacity improvement.

2. Using integers to compare strings

The objective of a contract I was involved in some years ago was to analyse a lot of HTTP data, from a load-balancer, to try and determine some statistics about it. Our input was billions of requests and getting the reports as quickly as possible was important (it fed into a health-check system, with a rolling average).

One of the key stages of the process was the very start of the request, because we would branch on the type of request (GET, HEAD, CONNECT, PUT, POST … etc). A branch here was bad for two reasons, it interfered with pipe-lining and the CPU caches, which is especially inefficient when most requests were GETs anyway.

Re-ordering the branches such that the GET case was a “fall-through” (that it didn’t involve a jump) helped a little, but there was still some inefficiency going on. The string comparison itself seemed to be wasting some of the L1 cache on us.

So, as a crazy solution, we treat the first 4 bytes of the request as an integer – and use integer comparison. On x86, “GET” == 5522759, “HEAD” == 1145128264 and so on. The first four bytes just happen to be unique in HTTP methods, and the check can happen in the CPU registers directly without having to deference pointers.

The app got about twice as fast, but this was probably due to some other variable now fitting in the L1 cache. Plenty of explanatory comments in the code made this an acceptable, if still crazy, optimisation to make. It also taught everyone involved exactly what SIGBUS really is.

3. Replacing inefficient calls

Sometimes the simplest optimisations are the most effective. At my current job, we managed to speed one of our external services up by about 20% just by replacing the top 3 calls that showed up in gprof with more standard implementations that are implemented in x86 assembly.

This is classic optimisation, run a profiler and go for a targeted attack, but it can still be amazing what it can achieve.

Unfortunately one of the problems with optimisation is that it’s hard. Being able to profile something is relatively straightforward and repeatable, but even the basic step of knowing that your application is slow is non-trivial. Sure there might be a high-level SLA, but you can always try throwing hardware at a problem.

Knowing that something is slow, or large, involves being able to make educated estimates about implementations, from first principals. Optimisers can approach a problem and think “well we’re accessing X much data, and performing about Y basic CPU operations, the fundamental limit is probably around Z”. That’s just step 1.

The other key part is that opimisers tend to have a deep, intuitive, understanding of systems at every level. They know not just a programming language, but the intricacies of how it is implemented. They’ll know how memory managers, file-systems, compilers, CPUs, networks and all sorts of things in-between actually work. This takes years, and quite a few “ahhhh” moments at 3AM in the middle of some nightmarish failure scenario.

strlen()

So where to start? One good place is to look is standard libraries, they contain many implementations of very basic routines that are called so often that they have to be optimal. Additionally, the really common routines are implemented in nearly every language, and you can look the trade-offs made within each. It’s a good way to learn a lot. One great case-study is strlen().

Before we go on, let’s get some things defined. For the most part here, we’re going to look at C style strings. A quick refresher; in C, strings are just a 0 terminated array of bytes (char is the C type for a character). They look like this;

1
2
3
4
5
6
7
8

/* A C string initialised as an array */
char hello[] = { 'h', 'e', 'l', 'l', 'o', 0 };

/* The same C string */
char hello[] = "hello";

/* A pointer to a C string */
char * hello = "hello";

strlen(), as the name suggests, returns the length of a string;

1
2
3
4
5
6
7

/**
* Return the length of a string.
*
* @param str The string to measure
* @return The string's length
*/
size_t strlen(const char * str);

Some important things; strlen(“hello”) returns 5, not 6. The zero at the end doesn’t count as part of the string. Zero is the only thing that defines the end of a string, all sorts of unprintable and control characters can be inside a string.

When someone is learning to program, a typical first attempt at a strlen implementation will be something like this:

Method 1: an iterative for-loop

1
2
3
4
5
6
7
8

size_t strlen(const char * str)
{
size_t len;

for (len = 0; str[len]; len++);

return len;
}

This method is easy to understand, and read, but it’s not very good. It runs in time O(n) and involves a lot of jumps.

The next thing programmers usually realise is that they don’t have to use array indices, that pointers are enough.

Method 2: sacrifice a variable and readability

1
2
3
4
5
6
7
8

size_t strlen(const char * str)
{
char * ptr;

for (ptr = str; *ptr; ++ptr);

return ptr - str;
}

Depending on how clever the compiler is, this code may be slightly faster, because there won’t be so many additions to the base pointer. We increment the pointer only, and use the difference between it and the base-pointer as the length. This code is less readable though, and probably counts as premature optimisation.

This method is still O(n) and still involves a lot of jumps. Let’s see what we can do about that next.

Method 3: partially unroll the loop

1
2
3
4
5
6
7
8
9
10
11
12
13
14

size_t strlen(const char * str)
{
char * ptr = str;

while(1)
{
if(!*(ptr++)) break;
if(!*(ptr++)) break;
if(!*(ptr++)) break;
if(!*(ptr++)) break;
}

return (ptr - 1) - str;
}

Now we’re into serious unreadability territory, and by the way, the above is how djb implements strlen. But we have actually gained some efficiency, now for every jump operation that the loop creates, we test for the 0 value 4 times. The choice of 4 times is arbitrary here, but an interesting exercise would be to vary this number and test each possibility.

More tests will interfere with pipelines, but there will be less jumping. Who knows where the sweet spot is. Still, though this implementation is faster, it’s still O(n). Unfortunately in C, we’re doomed to an O(n) implementation, best case, but we’re still not done … we can do something about the very size of n.

Method 4: word-wise checks

Just like the example I gave earlier, where we used integers to compare 4 bytes of a string all at once, we can do something even cleverer with strlen. We can construct a test, that with one small exception, can determine if any byte within a long is set to zero. This works for 2, 4 and 8 byte word-sizes.

It works by setting up a bit pattern like “01111110 11111110 11111110 11111111″, and then we add an arbitrary word to it. Anything except 0 in any of the bytes should cause an overflow into the neighbouring bytes. So by performing an addition, and then testing the “hole” bits, we can tell that there was likely a zero. There is one case where we’ll get a false-positive, which can happen when bit 31 is set – but that’s easy to check for.

It’s too long to reproduce here, but you can see the entire glibc implementation as a great example.

With this method we have divided our problem space, N is now N/4 or N/8, and the checks happen much more quickly. Though with really small strings, this method may actually be worse. The reason for that is that word-wise checks also have to be word-aligned. So if the string starts on a mis-aligned byte we may have to scroll up to 3-characters (for 32-bit words, it’s 7-characters for 64-bits) to become word-aligned.

Additionally when we do find a zero, we still have to perform 4 tests to figure out exactly which byte it was. If you had a lot of mis-aligned 7 byte strings, this method would be highly sub-optimal.

Method 5: Outsource the problem

In the many years war between RISC and “kitchen sink” architectures, the latter appears to have won. x86 is king, and that means we usually have plenty of instructions at our disposal. Two such instruction are “repnz” (repeat while non zero), and “scasb” (scan string) that combined allow to instruct the CPU to go find the next zero in a range of memory, using whatever tricks it likes.

The CPU can implement any of the approaches we’ve already talked about, but much more is possible. Modern memory, and SRAM in particular, has been designed to make parallelised searches possible. The chips can be electrically probing many sequences of bits all at once, and feed into a tree of gates that makes sure the earlier wins – O(ln) , in hardware. This is highly optimal, and if you’re using strlen() on an x86 host right now, that’s probably what’s happening.

But what else can we do? What if we make some trade-offs about the very nature of C strings?

Method 6: Add a cache

C is rightly criticised for how it handles strings, it’s pretty dumb, and C buffer overflows remain a major source of security problems. There really isn’t any excuse. Treating strings as arbitrary regions of zero-terminated memory has one main advantage; it allows strings to be seperated in-place. One can take “Hello World” and make it “Hello” and “World” with the very simple insertion of a zero byte.

But strings are not separated a whole lot, and it tends to be a one-time operation. If we throw away this “feature”, there is a much more optimal way to get string length – just keep recording it in a cache.

1
2
3
4
5
6
7
8
9

typedef struct {
char * str;
size_t len;
} string;

size_t strlen(const string * s)
{
return s.len;
}

Of course, we needn’t modify the API here, instead of creating a new type, we could just keep a static index of pointer values (say a hashMap) and record the lengths there. In fact, we could even have a compiler do this for us by defining the return value of the function as const.

But we do need to enforce the API. Now everything that deals with strings has to update this cache, and we can’t permit the programmer to mess with the string internally. Our string functions are the only valid way.

This is actually the norm. There are quite a few APIs for doing this in C, and it’s also what djb does in his code. Almost all modern languages take this approach, and have an internal record of every string length in their symbol table. Finally, we have a real O(1) solution.

What next?

But it doesn’t end there. Every solution above had trade-offs, and there are plenty of other ways of doing it.

Some text processors, for example, are designed to accommodate lots of string concatenation, so strings are optimistically allocated much more memory than they need, with lots of zeroes at the end. For these strings, if we know how much memory is allocated, we can implement strlen as a word-wise binary search, which will be O(ln).

Some XML processors, for another example, know that some very high percentage of the time that a closing tag will be exactly the same length as the name of the opening tag + 3 (‘< ', '/' and '>‘) – and take that as a best guess, and skip straight to that length and see if they find what the expect.

Other times, optimisers perform statistical analysis on their inputs and find out where the peaks are, and optimise the tests to go for those paths first. There really is an endless series of possibilities, and this is for something as simple as getting the length of a string. Underneath every little problem is a world of opportunity and a wealth of material from very clever people. (The Varnish source code, and the Architect Notes are also a great source of inspiration).

A few years ago, at Joost, we came across exactly this problem. An application that was spending about 10% of its time inside strlen. Within an hour or so of rewriting some primitives, it was down to less than a hundredth of a percent. You can’t drag and drop that kind of improvement in an IDE.

/~colmmacc/ » coding

Weighty matters

Non-random solutions

Random solutions

Reservoir sampling

Using a tree

A huffman(ish) tree

Period Pain

It increases the effective period of the task

The system as a whole has to cope with dangerous load spikes

N+1

Optimising strlen()

strlen()

Method 1: an iterative for-loop

Method 2: sacrifice a variable and readability

Method 3: partially unroll the loop

Method 4: word-wise checks

Method 5: Outsource the problem

Method 6: Add a cache

What next?