I guess a good day has to be followed by a bad day. It was going pretty decent, and then the network problems got bad enough it was a pain to do stuff. So Nick and I spent a lot of time sniffing traffic, analysing the resulting data, yanking machines off the network, and tracing cables. All fun while ssh sessions to different monitoring points randomly freeze up as there's some random connection glitch. Sometimes I wonder if it's time to yank everything out, reset it, and put it back together from known configs while testing every piece. At this point the guess is that something is occationally reflecting a few packets randomly (just long enough to cause a glitch, not long enough or predictable enough for a profile of what's happening to actually show up). That and the switches randomly dropping connections for no reason and the vendor still scratching their heads at it (worked fine up until this year). Oh well, we'll see what happens.