The week finished off pretty well. Nothing too exciting during the actual work week. Had some discussions with people about stuff, will probably have to follow up, but hopefully things will get better (the number of things that applies to from this weekend is amazing - probably everything you're thinking of and a few more). Also talked to Carlos again. That car wasn't stolen - it was just some idiot who didn't know how to treat a vehicle he owned properly. Oh well, I'm just glad I don't have to deal with it.

Saturday morning stuff blew up. Resulted in some power outages. Then more stuff blew up. Bethel ran off generators. Then they overheated, and there was another outage. That one wasn't bad, as it was less than an hour and we have a bit over an hour of runtime in each server room on batteries. But then there was another one for a few more hours, and that took out both server rooms. That outage occured Sunday morning. Brent and I both stopped in before church and got a few core things up and running. Doug apparently spent a bit more time and got most of the network up and running too. After church Brent and I spent a good chunk of the day getting everything else up and running, and it seemed to be going good. There were a few network oddities, but they were the type that go away over time as it figures stuff out.

At least until Sunday night. I kept noticing that my ssh sessions were lasting shorter and shorter times before dying (I tunnel all my email traffic through SSH normally - yea, call me a paranoid nutcase or whatever). Around midnight I finally went in because stuff wasn't working and the routers were seeing high load. It looked liked some sort of worm like last year, and now looking back on the patterns I'm guessing it's probably RBOT. Of course the fact that there was tons of broadcast traffic managed to upset the already confused switches even more, and DNS was effectively unreachable from portions of the network. This included one of the email servers resulting in some email starting to bounce. Managed to shut that down, but stuff just started to segment more and more. Finally just basically tried to get stuff mainly working, paged network services (or attempted to page - the pages got stuck in the system), and called it good. I think I got to bed around 4.

Monday afternoon Nick got back in town, and called me to say he was having a bonfire as well as ask what was up. I told him and he ended up picking me up and we went in to work. Ended up rebooting the entire core of the network a few times, removed some redundant network links to help improve the chances of stability, fixed the problem with most of the campus switches trying to get to the VLAN management server, and ended up rebooting the main DNS and DHCP server. Then we had to attempt to get everything to learn about everything else again, and switch to running in a normal state. Part of that involved trying to get all the switches to know the true location of the servers (since before they were seeing them all over). And Brent had to fix a couple things that were really broken too (and I had to power cycle one of those systems). Dave had to fix printing too, but that happened later. After the bonfire that Nick and I finished up just in time to leave for.

The bonfire was good though. We even got Paul to actually leave the Help Desk and come visit. Nick had to call him stuff first though. Burning wood, sticky half-melted fluffy-puffs, and a bunch of cell phones beeping constantly with pages we were just ignoring ("hey x is down now, that's too bad") have a way of helping everyone relax. Other than that I watched the first three DVDs of Alias Season 2 (minus the first episode I watched Thursday), and sat around the house. So how did you relax for labor day weekend? :)