Last week was interesting. We could sometimes reach our servers, and sometimes couldn't. I think as of today they have it pretty well fixed, but I'm sure something will come up with the number of changes that had to be made. Like all the pages I've been getting for the past 20 minutes probably mean something, especially since it was so quiet for the past day. I'm just glad that I can basically ignore it since my systems themselves stay up and I can't do anything about their connectivity.
I did get a lot of other stuff done though, like a bunch of mailing lists moved to the new server. With most of them people aren't even noticing the move, which is nice. Also lots more miscellaneous stuff, like reprogramming an alarm, talking to lots of people about stuff again, and generally dealing with the day to day stuff. I like Dave's response to what he does (as the same applies to both of us) - sit in an office and type all day. It sure is a lot simpler to explain.
Friday after work brought my sister over to get a bike and also went shopping at Sam's club. I picked up a few weeks worth of Dew, so shouldn't be running out for a bit. Saturday was sleep in and then try and ignore pages. Stuff broke again. I hear Doug worked most of the weekend, hopefully he gets a break now. Ross called me to invite me to a barbeque/bonfire, but I didn't feel like doing much of anything other than sitting around - I was still tired from the week. I did finish up Alias Season 2 though. Now I just have to watch Season 3 and I'll be caught up. I suppose I should get myth working again before Season 4 starts. I now have the box in the same closet as the HD tuner, so should be able to get better quality captures.
Sunday was church followed by sitting around the house all day. Basically trying to stay awake. I watched Identity, which was an interesting idea, but badly done. They start setting it up one way and it's really obvious it's gonna end with a reversal and yea... Can't really recommend based on the plot. If you're watching on a good system though, the sound is good, as well as the video cuts. While not having a great surround field, it has a lot of punch on the "surprises". That was followed by bringing in 4.5 gallons of Dew into the office so I don't have to carry it in later and mailing back the movie so I can get more sooner. Hmm. Mountain Dew Ice Cream. That actually sounds good.
Monday was sleep in day. Although it seems everyone forgot, at least at first. It was nice though. Stuff broke, and they fixed it, and I got to ignore it. Except when I couldn't get my email, that was sorta annoying. The mailing list server unsubscribed a ton of people today from one list I moved over. They had been bouncing ever since the migration. Apparently the old server just threw away the bounces and never bothered actually disabling or unsubscribing people. Hopefully this way we'll have less junk flying around the mail system long-term. Finally completely removed and reinstalled fink on my powerbook, so now I can install stuff. Got gimp up and running so I can edit pictures again. I have to say that it doesn't perform as well on large images as my linux box did. Then again, I normally seem to have problems with memory management on this box. There was a responsiveness glitch though. When changing layers and stuff and it had to swap things in from disk it'd take forever and the X11 interface doesn't seem to handle async very well. I wonder if it's based off older X code. Unfortunately that happened quite a bit. It warned me though that the image file was too big when I created it - I probably just need to tweak the settings a bit. It doesn't like when you get more than 11 million pixels or so in the image. Finished off the evening by watching Count of Monte Cristo, which was OK, followed by Rising Sun. It's been a bit over 10 years since I read the book (back in my reading every Michael Chichton book I could get my hands on phase), but I have to say when they say "based on the book" in this case it's pretty loose. They kept the general plot, but seem to have dropped half the investigation. Oh well, after seeing what they did to Jurassic Park, Congo, and Sphere, and especially Lost World, shouldn't be too surprised... I wonder what's in store for tomorrow.
The week finished off pretty well. Nothing too exciting during the actual work week. Had some discussions with people about stuff, will probably have to follow up, but hopefully things will get better (the number of things that applies to from this weekend is amazing - probably everything you're thinking of and a few more). Also talked to Carlos again. That car wasn't stolen - it was just some idiot who didn't know how to treat a vehicle he owned properly. Oh well, I'm just glad I don't have to deal with it.
Saturday morning stuff blew up. Resulted in some power outages. Then more stuff blew up. Bethel ran off generators. Then they overheated, and there was another outage. That one wasn't bad, as it was less than an hour and we have a bit over an hour of runtime in each server room on batteries. But then there was another one for a few more hours, and that took out both server rooms. That outage occured Sunday morning. Brent and I both stopped in before church and got a few core things up and running. Doug apparently spent a bit more time and got most of the network up and running too. After church Brent and I spent a good chunk of the day getting everything else up and running, and it seemed to be going good. There were a few network oddities, but they were the type that go away over time as it figures stuff out.
At least until Sunday night. I kept noticing that my ssh sessions were lasting shorter and shorter times before dying (I tunnel all my email traffic through SSH normally - yea, call me a paranoid nutcase or whatever). Around midnight I finally went in because stuff wasn't working and the routers were seeing high load. It looked liked some sort of worm like last year, and now looking back on the patterns I'm guessing it's probably RBOT. Of course the fact that there was tons of broadcast traffic managed to upset the already confused switches even more, and DNS was effectively unreachable from portions of the network. This included one of the email servers resulting in some email starting to bounce. Managed to shut that down, but stuff just started to segment more and more. Finally just basically tried to get stuff mainly working, paged network services (or attempted to page - the pages got stuck in the system), and called it good. I think I got to bed around 4.
Monday afternoon Nick got back in town, and called me to say he was having a bonfire as well as ask what was up. I told him and he ended up picking me up and we went in to work. Ended up rebooting the entire core of the network a few times, removed some redundant network links to help improve the chances of stability, fixed the problem with most of the campus switches trying to get to the VLAN management server, and ended up rebooting the main DNS and DHCP server. Then we had to attempt to get everything to learn about everything else again, and switch to running in a normal state. Part of that involved trying to get all the switches to know the true location of the servers (since before they were seeing them all over). And Brent had to fix a couple things that were really broken too (and I had to power cycle one of those systems). Dave had to fix printing too, but that happened later. After the bonfire that Nick and I finished up just in time to leave for.
The bonfire was good though. We even got Paul to actually leave the Help Desk and come visit. Nick had to call him stuff first though. Burning wood, sticky half-melted fluffy-puffs, and a bunch of cell phones beeping constantly with pages we were just ignoring ("hey x is down now, that's too bad") have a way of helping everyone relax. Other than that I watched the first three DVDs of Alias Season 2 (minus the first episode I watched Thursday), and sat around the house. So how did you relax for labor day weekend? :)
My guess is that car was stolen, given that it was in neutral, the button on the gear shift was broken, the doors were unlocked, and it had an open ignition. Security was putting a boot on it as I left, but it was gone the next morning too. Cops probably came and got it. The dent in my car was gone by the next morning when I went to work, so at least I don't have to deal with it.
I think this is the calmest school startup we've had since I've been at Bethel. Seriously, none of my stuff has really broken. The biggest issue we've seen is some switches flaking out, and Doug and Brent and I fixed that at least in the short term with not much work (we just effectively rebooted the network). Things started looping and confusing each other since buffers filled up and things started looping. That's fixed now though, at least on the network. There's still the maximum packet size at north village, but that's just helping them get their exercise since they have to either know what they're doing or go to the main campus. Other than that, I managed to eliminate one email list server so far - only two more to go. I also wrote a bounce handler to deal with bouncing emails for a lot of things, so none of the bounces for lists get to me now. The Help Desk probably will notice that they're not getting bounces for Incidents anymore soon too.
Last night was movie night with Butterfly Effect. Way too many of you didn't show up. Claim bad scheduling or whatever, but I didn't pick the day. And Doug even showed up. It was definitely a good movie. It was guys against girls on which ending was better (we watched both). Doug and Ross and I liked the director's cut, while Meghan and Steph and Lindsey liked the theatrical version. I wonder if that's trend holds (Hint: notice the "Add Comment" link below)? The movie was definitely different than what I was expecting. It's not a scary movie at all, but it is designed to shock and surprise you (like everything flashes and you feel an impact shock). It's an interesting one to think about the ideas involved in it.
Today was interesting. Lunch involved going Ross and I going to my place to eat leftover pizza from movie night watch Mr. Rogers. There need to be more shows like that on TV. Then before I leave work today Curt Koehn stops by and we chat for quite a while. He says hi to all you Bethel people I work with since I'll probably forget to tell you in person. Then I finish off the night by fixing email lists as someone is trying to use them, followed by watching Episode 1 of Season 2 of Alias. I'd say it's a good day.
As I write this I'm sitting in the SRC Lot at Bethel, waiting for a tow truck to arrive. No, my car is fine. It appears someone doesn't know how to handle their car though, and parked it pressed up against my bumper (or it rolled into my bumper since it's slightly sloped). So of course call Bethel security, and Carlos comes up to visit. They have no clue whose car it is and call the towing company. The fun part is I've been here since 9am so we have no idea when this car parked here. I'm guessing it was later. Actually I just called Raadt since he was parked next to me and the idiot wasn't here at 2pm when he went to lunch, so it was some time after that. Well, that's enough about this, I'm gonna go find something else to do while I wait. Hopefully they'll show up soon.
Thursday was interesting. We set up "Where's George", a webcam for the Bethel community to see the location of George. This was possible due to his standing outside greeting the freshmen and their parents as they moved in. It was pretty funny, although we did end up with a no-more-webcams-without-permission decree... Oh well, we were allowed to keep it up for the day, and it was fun while it lasted. Lunch was interesting, as a bunch of us just needed to get off campus, and had a fun time trying to get back on. We ended up eating lunch while sitting in the car listening to tunes in a long line of cars...
Work for the Thursday and Friday ended up mostly being list stuff. Setting up lots of lists, evaluating selection criteria for others, and migrating things to the new server. I managed to get most of the ones off one of the servers, and should hopefully be able to decommission that this next week after I hear back on some things. Saturday I was on call for the afternoon, and since I had to wake up and be available anyways, I figured I should put the time to use and did more lists stuff. There is definitely a more noticable latency doing X forwarding to Bethel now than during the summer though.
Saturday night a bunch of people came over and we watched Kill Bill: Vol. 1 followed by Kill Bill: Vol. 2. Good movies, although I had some issues with part of the ending on Vol. 2. I also discovered my walls have an interesting reasonance I'm gonna have to deal with. Sunday was Salem which was crowded due to a bunch of freshmen from Bethel being there for a service project. After that did a little grocery shopping so I can eat food at home for a while again. Then started watching The Man Who Wasn't There. That's an interesting movie. Billy Bob is an interesting actor. Don't see many black and white movies from this century either. Then the family went out to dinner for my brother's birthday at Green Mill, which was good. Then I finished the previously mentioned movie, and now am just sitting around. Realizing tomorrow is back to work. And hopefully not having to deal with a lot of stuff...
Saturday was decent. I didn't feel much better, but it was Meghan's going away party so I was outside a decent part of the day, which might have helped. After the party I fixed my dad's computer a bit, and that was it. Sunday was church at Salem, and I took a nap. I don't remember much else.
Monday was the first day of the fun known as training and welcome week. Basically since then I've been busy with work stuff constantly. Tuesday I started off the day with making some progress on the list server. I finished off the day doing a mass migration of email lists to the new server and rushing to get temp LDAP entries in place so we could go live since another system was broken. A bit over 4000 email lists in a day, that works. We'll find bugs in production I guess - so far all the new server restrictions seem to be working as planned though. Then Wednesday I finished up a script to add, modify, and delete LDAP aliases automatically, which should save hastle. That was also new student pizza and movie night which was fun with Ice Age. Although a bunch of us sat on our computers during it. And Mike Vedders leaves too many programs up and running while playing movies on the projector... That was followed by setting up my sister's computer (Macs have some cool features to help fix PCs), and meeting all her roommates and getting attacked by Austin in the process. 4 people at the movie ended up in her room, it was sorta weird.
Then I got this email forwarded to me. It was spam. No big deal, it's happened before, someone gets a list of Bethel people, the email servers don't like them anymore. The pattern in this one looks really odd though. Start poking a bit more, and see a distinct thing meaning the list came from one of two places. Look on one and someone obviously trying to get around the filters to harvest addresses. A marketing company, different from the spamming company. The emails are coming in at a rate slow enough that it's obvious they're trying to avoid detection. What a pain. Harvest from web (over 2 days, also likely to lessen chance of detection), demunge, and sell. At least the emails admitted they're an advertisement. How low can you go though - even targeted to the organization name...
Thursday was good. Had my last major Windows change made on the DHCP server today, which should be nice. It seems the broadcast traffic on the network has already dropped significantly. It'll be interesting to see later this weekend when more machines have renewed their DHCP leases. Also it was ice cream day, and it's hard to go wrong with that.
Then it was time to go to bed. That sucked, mainly because as soon as I was laying down the slight stuffiness turned into my entire head being stuffed up. My nose was completely plugged, and there was stuff running down my throat making both it and my stomach not happy after a while. Short summary is I didn't sleep much, and eventually was laying in bed at 5:30 wide awake and not able to sleep and send a message to my boss saying it was unlikely I'd be in. I slept in only a little bit and decided to see how I felt. A shower sorta cleared it up, but only for a few minutes. So armed with my cocoa spiked with red pepper (it's amazing how effective that is) headed off to go buy some medicine. After loading up on that and only being partially relieved after several hours, I went and bought some stronger stuff. Luckily that's doing a little better, but I'm still tired due to not sleeping much, and can't sleep during the day well normally. Hopefully tomorrow I'll feel better - being sick sucks.
Oh yea, someone on nanog was nice enough to give me a gmail account today. Looks cool, I'll have to see what I think long term though. I should have tried to get one earlier or something to get just jeremy for my username, but such is not the case. It's longer than that. One thing that impressed me right off the bat is the lack of a listed maximum password length and seeming acceptance of lots of non-alphanumeric characters. One more service realizing that the future of user-knows-only authentication being passphrase rather than password.
Work has been a little better with the boss back, but the work keeps piling on and he's still catching up and I'm still learning new an interesting ways to confuse the email server, so... I'm trying to get out at around 8 these days. Yea the days are long, but the hours at the end are when stuff seems to actually get done. I've started to get a lot of things automated, now I just have to get people to realise that's the case. I get halfway through looking at a request for access to something and wondering why it doesn't work since stuff is there and realize that they didn't bother checking and just requested it because it used to be that way. Oh well, such is life. I probably sent it in a long email too.
Speaking of long email, today I got drafts of some documents for training. I feel bad since I sent a really long list of stuff to change (like 15K plain text - for comparison my average "long" email people say is too long has been around 5-6K). Apparently I'm like the first person to look at it though out of a decent handful, so hopefully that'll be the most they have to deal with.
Last night some Jeff and Sarah decided to drop by my office since they were on campus anyways and it was good to see them. Although Jeff and I seemed to get on a lot of geeky tangents. It was pointed out to me that I didn't have any music playing and that knowing me it's not due to a lack of cables available. They were right and it actually turns out that I had everything I needed in my office, so the powerbook now has one more cable connecting it while on the desk. Now I just need to remember to turn the volume on the powerbook to almost off before connecting it to the amp so I remain friends with my 2nd floor coworkers...
One week and about 9 hours until projects need to be ready to go. Can we make it? And how many of you didn't realize that's when Welcome Week starts and we get swamped? At least we still sorta get a half day idea from student workers moving in the day before - should be interesting.
Copyright ©2000-2008 Jeremy Mooney (jeremy-at-qux-dot-net)
