A long week...

Tuesday was Sesame Beef day, and we even got it for free. Hard to beat that. As for work, Tuesday morning the web server crashed. We got that up. Then later one of the alarm systems had a bad battery and started beeping in classrooms, which the faculty didn't like too much. So that was disconnected. The generator was also fixed but they didn't want to test it in the middle of the day for some reason. Wednesday morning that was tested, and apparently all went well. Wednesday afternoon the root partition on one of our database servers filled up. The lack of usable /tmp created problems using the system. So we cleaned up a bit, moved that elsewhere, and attempted to scan the partition to figure out why it seemed there were unlisted open inodes or a miscalculation in free space. Turns out the partition was bad and the system promptly crashed. Rebooted off a restore and rebuilt with a combination of backups and copies from other servers, and brought it up. All seemed well.

Turns out something was missed. The sync scripts to the ERP and LDAP didn't handle it well, and the next morning the database listed nobody as having accounts. The scripts that reference that to keep Active Directory up to date started checking group memberships, found the accounts in the groups were unknown and took action to remove them. Luckily I had stuff that had synced earlier which had the data and imported it all again while schdav figured out the script syncing issue. Of course the AD scripts are designed to handle normal changes, not a complete initial load. As such they take some actions to ensure consistency which don't normally around each account, except when trying to do many thousands of them. So it took a couple hours for it to get everything updated again. Then people had to log out and back in to get access to things other than their home directories. That was a fun morning. There was also a switch interface issue taking out 2 buildings at the sem, but I didn't have to fix that one.

Friday the power cable fell out of our Internet connection provider's switch, taking out our connections to both Internet and Internet2. After not finding network services available to fix it, eventually went down, saw the lack of lights, and pushed the cable back in.

That's just the stuff that broke, lots of other planning and trying stuff in there too. At least it happened now and not in the next two weeks or so. The weekend has been going better. Yesterday was sitting around doing not much. I did have to fill out way too much stuff and get way too many envelopes ready to mail though. Why can't all companies/government entities accept electronic stuff? At least the county appears to be decent about email (better be after their phone queue hangs up on people), even if everything is still paper. Watched another disc of 24. Still not sure what I think of it. Not the best ever, but enough to keep me watching it at a somewhat decent pace (although it's slowing down my Netflix queue rate a bit). Today was restocking the dew in the office, grocery shopping, and probably more 24 in a bit. Tomorrow I'm taking a personal day, so may result in even more of that. Maybe I'll see about getting new tires on my car. It'd be nice to be able to accelerate like normal without spinning them.