I guess a good day has to be followed by a bad day. It was going pretty decent, and then the network problems got bad enough it was a pain to do stuff. So Nick and I spent a lot of time sniffing traffic, analysing the resulting data, yanking machines off the network, and tracing cables. All fun while ssh sessions to different monitoring points randomly freeze up as there's some random connection glitch. Sometimes I wonder if it's time to yank everything out, reset it, and put it back together from known configs while testing every piece. At this point the guess is that something is occationally reflecting a few packets randomly (just long enough to cause a glitch, not long enough or predictable enough for a profile of what's happening to actually show up). That and the switches randomly dropping connections for no reason and the vendor still scratching their heads at it (worked fine up until this year). Oh well, we'll see what happens.

Today was a good day. I killed a bunch of processes on homer, and removed it from Netsaint. Only a couple more decisions about DNS and it's ceremony at Lake Valentine time. Yes, I was the last one who needed to get stuff off there. I was waiting to push some stuff at another school, but got motivated by some spammer joe-jobbing a mailing list of all things. So now it's done. I hope we get those DNS decisions done soon. I'm getting enough crap bounceback from this thing...

I'm glad the election turned out the way it did. No contesting of popular vote vs electors, enough margin everyone isn't screaming, and conceeding the election rather than being a big baby and whining about it for months. Now I just hope the real election goes as well. Did you vote?.

Last night it rained a lot. It was really interesting this morning when I got up and there was enough fog in the air that everything which wasn't actually warm had a solid layer of condensation on it. Then it got nice out. This afternoon I decided to be a code monkey and recoded parts of IncidentBase. Mainly incidentdetail, startupdateincident, and myincidents, although some shared areas end up across the site. Andy convinced me to put some CSS with rounded borders around the User Info area, and that turned into me rewriting almost all the layout tables into divs. The menu bar up top is now a rounded thing that butts up against the top edge rather than being a square thing down a bit. And people can't post comments that make the page wide anymore either (it'll cause their comment to scroll within the normal space). incidentdetail is actually down to around 350 lines now, although a bunch of that was replaced with calls to 3 different IncidentBase library collections now. That was enough work though, I can't imagine how much of a hastle the rest will be to get to the point we can consider a real redesign (hopefully all in css). Quick glance says over 20k lines - most of which is hack upon hack from people asking for new stuff all the time. At it's all my code and will hopefully make sense the second time around.

Work as usual. Just tedious, although I think we're making good progress towards automating identifying bustificated lab machines, which should help everyone out a bit. And should be much better than having some poor student go around and attempt to log in to every one...

I got a New Camera today. Yea, it's not top of the line, but it also was dirt cheap (like $100 cheap). That's a good solid camera though. The goal was something I can actually leave in my bag rather and have with me rather than having to make a special point of carrying my camera or miss stuff because I don't have it. It's small enough I can slip it in my pocket and it's not that noticable either. I also managed to install the software and properly tag both my cameras with my name and stuff (which also gets embedded in all the pictures they take), which is a good thing.

I finally finished testing the domain stuff. Looks like we need to roll out more Domain Controllers to properly handle stuff. Oh well, I guess you do what's necessary. At least I don't have a domain controller and 4 desktop windows systems running on my powerbook anymore. Hmm. I think there's a total lunar eclipse coming up Wednesday night. I'll have to try and remember that or see if I can dig up info about it. Missed the one in the spring, and the next one is 2007...

Pretty usual week, thus the lack of posts. Friday night ended up rebuilding my powerbook since the partitioning tools from Apple suck. There's no good reason to not allow changing partitions on a system drive (I could see the system partition maybe, but not others). It seems to be running better for the most part. This time I actually went through custom and chose not to install the stuff I never used anyways (and discovered used way too much space). That includes iCal, iPhoto, iDVD, iMovie, GarageBand and the like. Unfortunately they still include too much crap. Like Address Book, Chess, iChat, Internet Connect, Mail, and Stickies. They already have a custom install, why make us still have to delete a bunch of crap afterwords? At least it still sorta resembles it's unix roots and I was able to drop in my home directory and have it work. Why it can't be on it's own partition you never have to touch when reinstalling without worrying about the partitioning issue if you want to move space around though...

Saturday was a busy day. Woke up in the morning to some pages from idiots who decided our blocking them from the mail servers meant they should try again in the middle of the night from a different machine (this one doesn't look quite as cracked). It was rather nice of them to include paging the sysadmins as part of the spam run though... Then there was a Soccer game followed by Football. It was cold and the lens didn't like the cold much. I had to help it focusing occationally, and still ended up with an unfortunate number of not-quite-focused shots. After the football game ended up taking some pics up at seminary since I was there and there was a good view with the colors and the lake. Took a look at the inter-building cable runs too since I hadn't been up there since they went in - interesting. After that was off to my parents' place for dinner since it was my mom's birthday. Pizza, followed by Radio. Good movie, good story.

Sunday was Salem and listening to Politics in church. That was interesting. Then some sitting around the house. Finished off the day by watching 3 movies. Started with Red Dragon. It's definitely an interesting story, seemed stronger and doesn't seem as held back as the other two. Second was The Shining. Seems like a failure compared to Kubrick's other films. I heard bad things about it, but tried to think it couldn't be so bad. So the movie starts off with the shadow of the helicopter they're shooting from visible in the shot - and doesn't get any better. Oh well, I guess they can't all be perfect. I ended up realigning my projector a bit during this one. Finished off the night with Conspiracy Theory. I liked that movie. It's one of those ones where there's action and they reverse the plot a ton of times. Some people like those, some don't. And now I need to get up early again tomorrow, so that's it.

People think I'm nuts because I like cold. Shortly after the last post, I got sick of sitting in the hot server rooms, and went to wander around outside. Ended up on the hill watching the wiring and cleanup happen. I was asked at least 3 times if I was cold. The temp was in the 30s, and I was standing there in my usual winter clothing selection (jeans and t-shirt). It's not like anyone else was wearing much else, most people just had a jacket and most didn't even have gloves (other than the big rubber ones when they started turning back on the 13.8kV feeds). Finally went back in the buildings about 2 hours later, and worked a bit inside. I got home around 10:30, and the window was open with the fan blowing in just like it has been for the past few weeks, and the house was around 60. Didn't bother me, just went to sleep for a few hours. Then spent more hours outside (soccer game), and home and sat in the basement for a few more hours (below 60 down there). Now I think I'm starting to realize why the cold doesn't bother me - somehow I just adapt. When I first came home from the game I noticed it was a bit chilly. The usual things, fingers and toes slightly numb, muscles tense, etc. A few hours later, I'm normal, all extremities feel normal, relaxed, and the thermometers say stuff is the same. Maybe I'm just weird or something. Oh well, all I care about is I don't need to bother with lugging around a jacket, just have to deal with how hot it feels inside for a few minutes when coming in until I adapt back.

Well, I'm sitting in one of the server rooms at Bethel right now, and it's actually sorta quiet. We shut most of the stuff down, and it's just a placeholder server (keep a basic "stuff is down" up for all web servers) and the network equipment up. Lots of complaining UPSes though. The funny part is walking down the ITS office is louder than the server rooms because of all the beeping (the server rooms have big UPSes with a distinctive beep that gets attention but isn't annoying, but the offices all have little ones beeping away like crazy). If all goes well things should be up in a bit over an hour, and hopefully we can get stuff up quickly and I can go home and get some sleep. Bill Buchanan just stopped in too. Sounds like it's a late but relatively smooth night so far - no issues have come up.

Nothing too interesting. No big attacks or anything. After work has basically been sitting around relaxing. Did rebuild 3 servers, two of them with Ross, so we got time to chat. That was nice. Did hit up Eddingtons for lunch on Friday, which was nice. Haven't been there for a long time.

Well, it's getting hot in here (most equipment is shut down, but with no AC it still gets hot quickly even with the door open). Over 80 on the cool side of the room. I think I'm gonna go walk around campus for a bit or something.

------------------------------------------------- This mail sent through IMP: http://horde.org/imp/

Another week complete. Nothing too exciting during the week. Got advisor mailing lists up and running, and some other miscellaneous stuff. Sesame Beef, but they apparently didn't give Wing all that great of meat this time. On Friday stuff crashed, people didn't tell people, and Nick and I figured it out when tracking down other problems. Then more stuff died, we blocked stuff, and it worked. Then the router paniced loosing the block rules, and I hacked root and added them back, along with a startup script in case it crashed again. Did get monitoring up though and since then nimda has created a couple dozen Incidents for machines which are misbehaving. 1.4GB of blocked SYN traffic so far - wow. Nick and I finished off the night by going to Taco Bell and ended up eating outside because the inside was closed. Oh well, it was a nice night.

Saturday I got up early to attempt to make it to the Bethel football game. Go out to my car and it just clicks when I turn the key. After a bit of troubleshooting and poking with meters and vaporizing metal, determined that it wasn't a dead battery, but actually the battery was broken. Luckily figured that out just in time before Betsy and Pete came over to give it a jump (thanks for being willing to help out), saving them the trip. The thing was apparently leaking acid (there was a puddle of probably .75 liters of acid in the battery tray only visible once the battery was out), and it had caused the post to corrode and come off the battery. At first it only looked like it had broken, thus the attempts to bridge it and the vaporized metal (ah, the fun chemistry of lead acid). Ended up with a trip to buy a new battery (under $40), going through a few boxes of baking soda do neutralize the acid, and a jump to get it over to my parents place where we cleaned it out, got the old battery post properly removed from the cables, and washed and siphoned out the tray, and it's back up and running. More excitement than I wanted for the day. Only got one acid burn from it and it's pretty small.

Mixed in with all that were two sports games. The first was Bethel's at Hamline. Even saw Mr. Boyum there, and got a picture of him avoiding looking at the camera. My dad got his 20d Friday so I got my camera back and got to use his 28-70mm USM IS lens that came with it (he was using a 70-200 USM IS 2.8L). Nice lens, and very fast focus. Probably should get a little closer than the stands with it though. Another photographer there loaned him a 1.4x teleconverter for the day, so his photos should be interesting. After that went to my youngest sister's soccer game, and ended up swapping lenses and using the L for a bit. That one's even nicer, although having a lens that weights 3x as much as the camera and is worth 2x as much feels a little odd. Probably can't justify buying it, but they can be rented in the cities for about $25/day (a weekend is a "day"), so may have to do that occationally.

Finished off Saturday (well, other than the car thing) by Betsy, Pete, brooke, Joe, and Travis coming over for movie night. We watched Rushmore which is a decent movie, although seems a bit rushed. Probably could have stretched it out to 2 hours and not had it been too boring. Sunday was church at Salem, which I was tired for. I must not have slept well since I was both tired and sore. Oh well. Closed up some stuff for work (stuff from Friday night), and just sorta relaxed for the afternoon. Watched Underworld, which was pretty decent. Think the Matrix movies without the disappointment of what happened to the story, mixed with something like The Crow. Lots of shooting and chasing/acrobatics, along with a scarily realistic soundtrack. You know how most movies sorta have lots of vibration when shots happen? This one you feel the shots. And there are lots of PoV shots when one person gets attacked by another - and you feel those impacts too. Fun movie.

Last week was interesting. We could sometimes reach our servers, and sometimes couldn't. I think as of today they have it pretty well fixed, but I'm sure something will come up with the number of changes that had to be made. Like all the pages I've been getting for the past 20 minutes probably mean something, especially since it was so quiet for the past day. I'm just glad that I can basically ignore it since my systems themselves stay up and I can't do anything about their connectivity.

I did get a lot of other stuff done though, like a bunch of mailing lists moved to the new server. With most of them people aren't even noticing the move, which is nice. Also lots more miscellaneous stuff, like reprogramming an alarm, talking to lots of people about stuff again, and generally dealing with the day to day stuff. I like Dave's response to what he does (as the same applies to both of us) - sit in an office and type all day. It sure is a lot simpler to explain.

Friday after work brought my sister over to get a bike and also went shopping at Sam's club. I picked up a few weeks worth of Dew, so shouldn't be running out for a bit. Saturday was sleep in and then try and ignore pages. Stuff broke again. I hear Doug worked most of the weekend, hopefully he gets a break now. Ross called me to invite me to a barbeque/bonfire, but I didn't feel like doing much of anything other than sitting around - I was still tired from the week. I did finish up Alias Season 2 though. Now I just have to watch Season 3 and I'll be caught up. I suppose I should get myth working again before Season 4 starts. I now have the box in the same closet as the HD tuner, so should be able to get better quality captures.

Sunday was church followed by sitting around the house all day. Basically trying to stay awake. I watched Identity, which was an interesting idea, but badly done. They start setting it up one way and it's really obvious it's gonna end with a reversal and yea... Can't really recommend based on the plot. If you're watching on a good system though, the sound is good, as well as the video cuts. While not having a great surround field, it has a lot of punch on the "surprises". That was followed by bringing in 4.5 gallons of Dew into the office so I don't have to carry it in later and mailing back the movie so I can get more sooner. Hmm. Mountain Dew Ice Cream. That actually sounds good.

Monday was sleep in day. Although it seems everyone forgot, at least at first. It was nice though. Stuff broke, and they fixed it, and I got to ignore it. Except when I couldn't get my email, that was sorta annoying. The mailing list server unsubscribed a ton of people today from one list I moved over. They had been bouncing ever since the migration. Apparently the old server just threw away the bounces and never bothered actually disabling or unsubscribing people. Hopefully this way we'll have less junk flying around the mail system long-term. Finally completely removed and reinstalled fink on my powerbook, so now I can install stuff. Got gimp up and running so I can edit pictures again. I have to say that it doesn't perform as well on large images as my linux box did. Then again, I normally seem to have problems with memory management on this box. There was a responsiveness glitch though. When changing layers and stuff and it had to swap things in from disk it'd take forever and the X11 interface doesn't seem to handle async very well. I wonder if it's based off older X code. Unfortunately that happened quite a bit. It warned me though that the image file was too big when I created it - I probably just need to tweak the settings a bit. It doesn't like when you get more than 11 million pixels or so in the image. Finished off the evening by watching Count of Monte Cristo, which was OK, followed by Rising Sun. It's been a bit over 10 years since I read the book (back in my reading every Michael Chichton book I could get my hands on phase), but I have to say when they say "based on the book" in this case it's pretty loose. They kept the general plot, but seem to have dropped half the investigation. Oh well, after seeing what they did to Jurassic Park, Congo, and Sphere, and especially Lost World, shouldn't be too surprised... I wonder what's in store for tomorrow.

The week finished off pretty well. Nothing too exciting during the actual work week. Had some discussions with people about stuff, will probably have to follow up, but hopefully things will get better (the number of things that applies to from this weekend is amazing - probably everything you're thinking of and a few more). Also talked to Carlos again. That car wasn't stolen - it was just some idiot who didn't know how to treat a vehicle he owned properly. Oh well, I'm just glad I don't have to deal with it.

Saturday morning stuff blew up. Resulted in some power outages. Then more stuff blew up. Bethel ran off generators. Then they overheated, and there was another outage. That one wasn't bad, as it was less than an hour and we have a bit over an hour of runtime in each server room on batteries. But then there was another one for a few more hours, and that took out both server rooms. That outage occured Sunday morning. Brent and I both stopped in before church and got a few core things up and running. Doug apparently spent a bit more time and got most of the network up and running too. After church Brent and I spent a good chunk of the day getting everything else up and running, and it seemed to be going good. There were a few network oddities, but they were the type that go away over time as it figures stuff out.

At least until Sunday night. I kept noticing that my ssh sessions were lasting shorter and shorter times before dying (I tunnel all my email traffic through SSH normally - yea, call me a paranoid nutcase or whatever). Around midnight I finally went in because stuff wasn't working and the routers were seeing high load. It looked liked some sort of worm like last year, and now looking back on the patterns I'm guessing it's probably RBOT. Of course the fact that there was tons of broadcast traffic managed to upset the already confused switches even more, and DNS was effectively unreachable from portions of the network. This included one of the email servers resulting in some email starting to bounce. Managed to shut that down, but stuff just started to segment more and more. Finally just basically tried to get stuff mainly working, paged network services (or attempted to page - the pages got stuck in the system), and called it good. I think I got to bed around 4.

Monday afternoon Nick got back in town, and called me to say he was having a bonfire as well as ask what was up. I told him and he ended up picking me up and we went in to work. Ended up rebooting the entire core of the network a few times, removed some redundant network links to help improve the chances of stability, fixed the problem with most of the campus switches trying to get to the VLAN management server, and ended up rebooting the main DNS and DHCP server. Then we had to attempt to get everything to learn about everything else again, and switch to running in a normal state. Part of that involved trying to get all the switches to know the true location of the servers (since before they were seeing them all over). And Brent had to fix a couple things that were really broken too (and I had to power cycle one of those systems). Dave had to fix printing too, but that happened later. After the bonfire that Nick and I finished up just in time to leave for.

The bonfire was good though. We even got Paul to actually leave the Help Desk and come visit. Nick had to call him stuff first though. Burning wood, sticky half-melted fluffy-puffs, and a bunch of cell phones beeping constantly with pages we were just ignoring ("hey x is down now, that's too bad") have a way of helping everyone relax. Other than that I watched the first three DVDs of Alias Season 2 (minus the first episode I watched Thursday), and sat around the house. So how did you relax for labor day weekend? :)