Archive for May, 2003
Sloowwwwwwww
Posted on May 28, 2003, under legacy.
So, amid a new security issue, Apache 2.0.46 is Released and I set about upgrading all of the servers I’m responsible for. I start with prodigy, because skyhawk, our webmaster isnt on, I leave it running. My the time I’ve upgraded 3 Dells run Linux, and a FreeBSD server I have, Prodigy still hasnt even finished
1 | ./configure |
.
Another 4 servers later, I’m still waiting on Prodigy. Jesus, there’s a reason it’s called Slowaris.
Wowsa
heanet@canyonero:~$ ps -ef | grep -c httpd
1317
and that’s just http, there’s another 317 ftp downloads at the same time. That’s an awful lot of downloads.
Leathermen are great, today we actually used a Leatherman to cut 5 cm thick wooden floortiles, and it did it better than the hacksaw we tried to use first. Then, in a stunning overture the Leatherman made light work of rackmounting a Cisco 12400. What that photo wont tell you is that they are about 1.5 metres high and weigh about half a tonne.
Also, I got annoyed enough to post to IIU, something I havnt done in a while (though it is hard when you work in the industry).
One quarter of Sourceforge
From the looks of things, Sourceforge seem to have slashed their list of official mirrors. As far as I can tell, HEAnet is now the only non-US mirror of Sourceforge and there are now a total of 4 SourceForge download sites.
We don’t mind, we certainly have the capacity to cope, but it is a little odd … maybe SourceForge are upgrading/replacing the other mirrors.
Firewalls Suck!
As you may know, I’m a network engineer for HEAnet, and one my jobs is system administator for ftp.heanet.ie, one of the busiest download servers in Europe. Running a big ftp server means there’s more than a few mails a week, usually about the odd corrupted download, a stale mirror … this and that. But yesterday we got a rather unusual problem mailed to us. Just one of our mirrors (in this case OpenBSD) wasnt working for the user. All of our other mirrors worked fine, and even OpenBSD worked fine in http, just not ftp.
Being the stupid type, I decided to actually try and figure out what was wrong. The reporter was extremely clueful, and had no fear of tcpdump, which is a good thing because I had to ask him to run it a lot. I’ll try and condense our experiences into something that is still readable, but remains horrific.
The user has a nice network, he has an OpenBSD machine that is being used as a Firewall (using packet filter) and NAT device for some Windows 2000 machines, as well as a FreeBSD machine or two in public IP space. He tested using Internet Explorer on the windows boxes, and command-line ftp on the FreeBSD and OpenBSD ones, we tried all combinations of filtering on/off and ftp passive mode on and off and so on with no avail. I checked using command-line ftp on a FreeBSD machine myself and Colin have in San Fransisco, and it worked fine for me.
No matter what, he failed to be able to use ftp://ftp.heanet.ie/pub/OpenBSD/ for some reason. Taking a gander around the various mirrors, I noticed that OpenBSD is unusual in that it has a larger than average
1 | .message |
file, this is the file that gives the nice OpenBSD Logo banner. So I deleted it, and voila the mirror worked for the user. All of the symptoms went away, weird, to say the least. So, some experimentation: We created an empty
1 | .message |
file, still worked. So we started shoving some dummy text into it, and it just kept on working, right up until we get to 1,288 bytes.
Anyone who knows IP well is probably shouting MTU, MTU, MTU! right now, but for those of you who don’t realise the signifance – in IP the Maximum Transmitable Unit size is usually 1,500 bytes. When you take away IP and TCP headers, this comes to a payload size of 1,288 bytes, a figure that is familiar (or should I say infamous) to network engineering types. And sure enough, if we lowered the MTU on either side, the size the banner could be before breakage also dropped. WEIRD-ASS. It very much looked like something was preventing FTP, a TCP based protocol which should have no problem with packet fragmentation from utilising more than one packet to transmit the ftp banner. Again, I say – WEIRD ASS.
So, we knew that something between our FTP server and our users machine was acting a little strangely. We ruled out routers, on the basis that it would take a really odd router bug to have this kind of an effect (not that most of NANOG wouldnt put that past Cisco ;). I called our users service provider, who happened to be a customer of ours to let them know about the odd problem, and see about finding a root cause.
Their network engineer confirmed that they could experience the problem aswell, which instantly gave rise to a desire to fix it, they know all too well that these small niggles can become major headaches far too easily! Again, their engineer was extremely clueful and again completely unafraid of the ways of tcpdump. So, straight into firewalls logs, as they were running checkpoint-ng, though had a nice liberal firewall policy as regards to outbound traffic.
And sure enough, right there in the logs were messages saying:
message_info: Port command ended without a new line
which corresponed pretty nicely with the kernel errors on our side:
conntrack_ftp: partial PORT 359824284+26
this, despite the fact that we were using passive ftp, which doesnt use the port command, and that ftp banners are exchanged on the control channel anyway .. but hey! So, after logging a call with his checkpoint support people, and a quick google on how to disable the check all is now working. FTP is working fine for the user, passive and active, with the full banner. Here’s the reason the check exists in the first place, and when you read it it kind of makes sense. The fact that the checkpoint people don’t seem to have checked it too well, or handle MTU boundaries satisfactorily doesnt give me much confidence in their abilities or testing procedures though.
But this definitely re-enforces my opinion that firewalls suck, they reduce the auditability of your network, and they interfer with things that arnt meant to be interfered with. If you want to limit access to servers, or limit outgoing connections … use an Access Control List, on your router. Don’t use one of these fancy mumbo jumbo firewall thingies, gah!
Actually I have much worse stories about firewalls (including one which reduced the security of a network my a few orders of magnitude) … but they’re for another day. In the meantime, hopefully I will have mentioned the words banner,checkpoint,ftp and mtu enough times in this articles for google to catch it for the next poor unfortunate
network engineer :)
FluxFlox
Today I took the plunge, after *counts* 7 years of using WindowMaker I’ve decided to try something new, fluxbox. Don’t get me wrong I love WindowMaker but a small few things annoy me about it, the clip and dock just take up space as far as I’m concerned, and the complete dearth of decorative options for Title Bars was very annoying. So now I’m using FluxBox. It took a while to get keybinding in to a usable state for me, but now that I’m there I have to say I’m pleasantly suprised. This might just work out.
Last night I read through a review copy of Dave Malone’s and Niall Murphy’s IPv6 book, looks extremely good. I managed to mail a vew comments before the night was out, but I’ll sit down and give the book a proper read later this week.
Bringing a smile to my face, I managed to track down the Department of Health Press Release about the incoming smoking ban, it’s 4 months old, but still, it makes me very happy. I can’t wait for smoking to be banned.
Though with the state of law enforcement in this country, it might be a while yet before the benifits are felt in any real way.
Links I like
ArchEire ,
kuro5hin ,
percyschmeiser,
kame,
Astronomy Picture of the Day,
Céile House,
blogs.linux.ie,
Oasis,
Internet Traffic Report,
O’Reilly.com,
How Stuff Works,
Irish Music Magazine,
IIU,
ISOC,
RFC Editor,
NewScientist
ApacheWeek
LárBhealtaine
For the first time in weeks I got to watch my favourite TV show, West Wing, on Thursday. Man was it good, one of the best episodes ever. For more viewing pleasure, now that the 4th freely available Animatrix short is downloadable I downloaded them all in work (gotta love GigE to the office) and watched them too. It’s a mixed batch but I liked them on the whole.
This week has been a good one for me programming wise, I’ve managed to get my head around AF indepedent programming. Actually I impressed myself by getting far enough to point out omisions in that document, if you look at the latest revision it’s from 16/05/2003, that’s because I mailed Itojun suggesting what is now rule of thumb number 3. The Document didnt even mention ai_addrlen two days ago. Itojun’s a cool guy, and kame are good people.
I tried out Simon Tatham’s DoIt which looks very useful, but the windows part of it crashes on me and just seems fundamentally broken somehow. I’ve mailed a bugreport, we’ll see how it goes.
It’s nearly a week since the Dell was installed, people are logging in, and everything is going well, but I still hate the name deathray, it sucks, and it’s pretty agressive to have to word “death” as the first thing you see in your prompt. I definitely preferred carbon.
Midweek Madness
It’s midweek madness time again, yesterday was an unbelievably busy day in work, and as if that wasnt enough I decided to punish myself by migrating RedBrick’s /home to the new Dell, oh and there was a SAGE-IE meeting.
Actually the meeting was good, we got learn about some very mad laws that exist right now, courtesty of Adrian Colley, and some scarier proposed laws that are on their way. Nick Hilliard also expanded on his proposal to start a free Irish Certificate Authority.
FirstMonday eventually came out, on Saturday, but there were some intresting papers, my favourite was The Radius Project. I might even get around to reading the latest version of NewScientist today. That’s progress.
colmmacc@deathray (~) $
Well, we finally got the new machine installed. It didnt take all that long, though it did go through 3 hostnames .. postel, carbon and it’s current (and maybe even final) name … deathray. We also managed to get the RedBrick room into a more useful state, and building work is now getting the go ahead!
Now, for the rest of the move …
Go Monaghan!
At last, Armagh get their just-desserts for beating Dublin in the Semi-Finals last year. Monaghan beat Armagh, in the very first round of the championship. It’s going to be a great summer!