Archive for May, 2003

One quarter of Sourceforge

Posted on May 24, 2003, under legacy.

From the looks of things, Sourceforge seem to have slashed their list of official mirrors. As far as I can tell, HEAnet is now the only non-US mirror of Sourceforge and there are now a total of 4 SourceForge download sites.

We don’t mind, we certainly have the capacity to cope, but it is a little odd … maybe SourceForge are upgrading/replacing the other mirrors.

Firewalls Suck!

Posted on May 23, 2003, under legacy.

As you may know, I’m a network engineer for HEAnet, and one my jobs is system administator for ftp.heanet.ie, one of the busiest download servers in Europe. Running a big ftp server means there’s more than a few mails a week, usually about the odd corrupted download, a stale mirror … this and that. But yesterday we got a rather unusual problem mailed to us. Just one of our mirrors (in this case OpenBSD) wasnt working for the user. All of our other mirrors worked fine, and even OpenBSD worked fine in http, just not ftp.

Being the stupid type, I decided to actually try and figure out what was wrong. The reporter was extremely clueful, and had no fear of tcpdump, which is a good thing because I had to ask him to run it a lot. I’ll try and condense our experiences into something that is still readable, but remains horrific.

The user has a nice network, he has an OpenBSD machine that is being used as a Firewall (using packet filter) and NAT device for some Windows 2000 machines, as well as a FreeBSD machine or two in public IP space. He tested using Internet Explorer on the windows boxes, and command-line ftp on the FreeBSD and OpenBSD ones, we tried all combinations of filtering on/off and ftp passive mode on and off and so on with no avail. I checked using command-line ftp on a FreeBSD machine myself and Colin have in San Fransisco, and it worked fine for me.

No matter what, he failed to be able to use ftp://ftp.heanet.ie/pub/OpenBSD/ for some reason. Taking a gander around the various mirrors, I noticed that OpenBSD is unusual in that it has a larger than average

1
.message

file, this is the file that gives the nice OpenBSD Logo banner. So I deleted it, and voila the mirror worked for the user. All of the symptoms went away, weird, to say the least. So, some experimentation: We created an empty

1
.message

file, still worked. So we started shoving some dummy text into it, and it just kept on working, right up until we get to 1,288 bytes.

Anyone who knows IP well is probably shouting MTU, MTU, MTU! right now, but for those of you who don’t realise the signifance – in IP the Maximum Transmitable Unit size is usually 1,500 bytes. When you take away IP and TCP headers, this comes to a payload size of 1,288 bytes, a figure that is familiar (or should I say infamous) to network engineering types. And sure enough, if we lowered the MTU on either side, the size the banner could be before breakage also dropped. WEIRD-ASS. It very much looked like something was preventing FTP, a TCP based protocol which should have no problem with packet fragmentation from utilising more than one packet to transmit the ftp banner. Again, I say – WEIRD ASS.

So, we knew that something between our FTP server and our users machine was acting a little strangely. We ruled out routers, on the basis that it would take a really odd router bug to have this kind of an effect (not that most of NANOG wouldnt put that past Cisco ;). I called our users service provider, who happened to be a customer of ours to let them know about the odd problem, and see about finding a root cause.

Their network engineer confirmed that they could experience the problem aswell, which instantly gave rise to a desire to fix it, they know all too well that these small niggles can become major headaches far too easily! Again, their engineer was extremely clueful and again completely unafraid of the ways of tcpdump. So, straight into firewalls logs, as they were running checkpoint-ng, though had a nice liberal firewall policy as regards to outbound traffic.

And sure enough, right there in the logs were messages saying:

message_info: Port command ended without a new line

which corresponed pretty nicely with the kernel errors on our side:

conntrack_ftp: partial PORT 359824284+26

this, despite the fact that we were using passive ftp, which doesnt use the port command, and that ftp banners are exchanged on the control channel anyway .. but hey! So, after logging a call with his checkpoint support people, and a quick google on how to disable the check all is now working. FTP is working fine for the user, passive and active, with the full banner. Here’s the reason the check exists in the first place, and when you read it it kind of makes sense. The fact that the checkpoint people don’t seem to have checked it too well, or handle MTU boundaries satisfactorily doesnt give me much confidence in their abilities or testing procedures though.

But this definitely re-enforces my opinion that firewalls suck, they reduce the auditability of your network, and they interfer with things that arnt meant to be interfered with. If you want to limit access to servers, or limit outgoing connections … use an Access Control List, on your router. Don’t use one of these fancy mumbo jumbo firewall thingies, gah!

Actually I have much worse stories about firewalls (including one which reduced the security of a network my a few orders of magnitude) … but they’re for another day. In the meantime, hopefully I will have mentioned the words banner,checkpoint,ftp and mtu enough times in this articles for google to catch it for the next poor unfortunate
network engineer :)