So I've found some fun things while upgrading from Exchange 2000 to an Exchange 2003 cluster. First is unable to categorize does not mean Journaling. Lots of "Messages Awaiting Directory Lookup", querying all the servers that the servers were using to figure out why they were complaining about it, only finding entries about Journaling and how to fix/disable it. Turns out it's not that it sees the msExchMessageJournalRecipient with bad data, but just it isn't able to access the object to read that attribute. The fix is to enable inherited permissions on the server object in ESM. I suspect the cause of the problem was adding another Exchange domain necessitated additional ACEs for the Exchange groups in the new domain. But the question of why it was broken in the first place is up in the air.
The second piece is I think why. I found after "fixing" the above that the admins couldn't get into any mailboxes. This is default behavior (it sets a Deny ACL on the mailboxes), but apparently was fixed at some point by breaking inheritance and removing the Deny. Proper fix if the server admins should be mailbox admins as well is to add an ACL to the Server object allowing full control for the admin (non-inherited Allow overrides an inherited Deny in DACLs). It'd also likely be possible to fix at the Organization or Administrative group level, but that requires work to make the Security tab visible, and is more likely to break if future versions change those permissions (or add additional ACEs in delegation and one has to figure out how to fix again). Probably better to just do it per server or store.
Third is that Exchange follows MX records for intra-domain routing. This was discovered when changing this to get names set up for externally established DNS names. This is the most pressing argument I've seen for split DNS (if mail should go through a central hub), which is unfortunate since there's not a clean workaround. That is other than a front end server architecture (everything seems to push that way in the end). It'd be nice if they just made an OWA URL that would be used for server referrals.
Thats it for now. Between those problems and replacing a failed drive in another machine, I'm calling it a day.
So yesterday one of our Active Directory DCs failed due to running out of memory. Some of the fun error messages that ensued:
CN=Configuration,DC=<subdomain>,DC=<domain>,DC=<tld>
<SiteName>\<DCName> via RPC
DC object GUID: <GUID>
Last attempt @ (never) was successfulSource: <SiteName>\<DCName> ******* 305 CONSECUTIVE FAILURES since (never)
The effects of KCC and dynamic replication partners...
So apparently sometime early this morning the server that hosts this site along with several others became unable to allocate memory. I'm not sure of the cause, other than after I forced a reboot of my box it came back (around 11:30CST). The load on my box was nothing but the host load was high. Host load dropped to medium but is back to high, and my load is still low. So I'm thinking it's not me, and I just got hit as an unfortunate side effect. I've tweaked a couple things which will hopefully allow the system to run a bit better under load, but I'm not sure it'll help with heavy external interactions. Sorry if your site was affected, and now you know why.
Email can be annoying. Today apparently an email sent to 5 AOL members and marked as spam by one of them (which I didn't receive the scomp copy of until the issue was resolved) was enough to push one of the mail servers at work over to getting 421 tempfails from AOL's servers due to "[spam] complaints and/or high volumes of e-mail." Yea, it's wonderfully vague. It's amazing how much mail builds up to them in just a couple hours from a general (non-geek) population, and how much people expect email to be instant. Of course 10 minutes after I change the mail routing tables and forced a queue run to push the queued mail to a different server that isn't blocking mail, I notice a message to one of their alias domains go through. Figures. Yet another reason that spam (and the fighting thereof) is annoying...
On the good side, I did get a lot of old accounts that were just throwing mail around before waiting 5 days and bouncing it cleaned up. Over the past week I've changed them to do recipient checking at SMTP time, and after some log testing to also do sender checking. My mail queues are getting pretty clean now.
Someone tried to sell a security vulnerability on eBay. Of course Microsoft complained and eBay pulled the auction.
Finally, something that's more of a mess than our server rooms. Looks like they had cable management, but it filled up and they didn't do anything to fix it. Those free hanging 20ft runs of CAT5 have to be great for reliability.
Found this tool today, which is cool. Seems Windows Server 2003 SP1 finally caught up with Netware 4 on a nice feature.
Oh yea, forgot to mention. I've been planning on moving to a new virtual machine system for a while to reduce cost and also to upgrade to a newer system version. This week my virtual machine has rebooted like six times due to an issue with its host machine and getting it resolved. This prompted me to go look to see if there were free nodes at linode again, and they did so I have a machine to start poking at (so far have just removed a pile of useless packages from their "small" install - who uses an X server on a remote machine?!?). I've seen good things about them whenever the topic of virtual machines comes up. Before I move everything over though, anyone have any direct or close indirect experience with them? They use uml where the current one I think is vmware gsx, so I'm expecting some differences there. I'm more looking for info on reliability or any issues running specific apps. Anyone?
Tuesday was Sesame Beef day, and we even got it for free. Hard to beat that. As for work, Tuesday morning the web server crashed. We got that up. Then later one of the alarm systems had a bad battery and started beeping in classrooms, which the faculty didn't like too much. So that was disconnected. The generator was also fixed but they didn't want to test it in the middle of the day for some reason. Wednesday morning that was tested, and apparently all went well. Wednesday afternoon the root partition on one of our database servers filled up. The lack of usable /tmp created problems using the system. So we cleaned up a bit, moved that elsewhere, and attempted to scan the partition to figure out why it seemed there were unlisted open inodes or a miscalculation in free space. Turns out the partition was bad and the system promptly crashed. Rebooted off a restore and rebuilt with a combination of backups and copies from other servers, and brought it up. All seemed well.
Turns out something was missed. The sync scripts to the ERP and LDAP didn't handle it well, and the next morning the database listed nobody as having accounts. The scripts that reference that to keep Active Directory up to date started checking group memberships, found the accounts in the groups were unknown and took action to remove them. Luckily I had stuff that had synced earlier which had the data and imported it all again while schdav figured out the script syncing issue. Of course the AD scripts are designed to handle normal changes, not a complete initial load. As such they take some actions to ensure consistency which don't normally around each account, except when trying to do many thousands of them. So it took a couple hours for it to get everything updated again. Then people had to log out and back in to get access to things other than their home directories. That was a fun morning. There was also a switch interface issue taking out 2 buildings at the sem, but I didn't have to fix that one.
Friday the power cable fell out of our Internet connection provider's switch, taking out our connections to both Internet and Internet2. After not finding network services available to fix it, eventually went down, saw the lack of lights, and pushed the cable back in.
That's just the stuff that broke, lots of other planning and trying stuff in there too. At least it happened now and not in the next two weeks or so. The weekend has been going better. Yesterday was sitting around doing not much. I did have to fill out way too much stuff and get way too many envelopes ready to mail though. Why can't all companies/government entities accept electronic stuff? At least the county appears to be decent about email (better be after their phone queue hangs up on people), even if everything is still paper. Watched another disc of 24. Still not sure what I think of it. Not the best ever, but enough to keep me watching it at a somewhat decent pace (although it's slowing down my Netflix queue rate a bit). Today was restocking the dew in the office, grocery shopping, and probably more 24 in a bit. Tomorrow I'm taking a personal day, so may result in even more of that. Maybe I'll see about getting new tires on my car. It'd be nice to be able to accelerate like normal without spinning them.
The weekend was long, but good. Saturday involved getting annoyed at my alarm at 2am, followed by leaving for the parents' place shortly after 2:30. Given that I usually sleep about 12 hours later than this on Saturday, this was a stretch. We then left for up north at around 3:30, and arrived a bit before 8. Of course at this point we realized that due to the unknown weather and roads we had left some extra time, we were a bit hungry, and we neglected to note that non-packaged food is a bit harder to find in rural MN than in the metro area. So after consulting google it was 30 miles of backtracking to find some food and a place to change, and then head back to the wedding at 10. What's an extra 60 miles when you're going 500? :) That was good, and had an interesting mix of American and Zambian cultures (the family has lived there up until a few years ago). At some point we finally ended up going back, and after making much better time on the way back (although still had to slow down to 45 in most towns, and 30 in some like LP - crazy small town MN, that's slower than the street I live on) and arrived back around 6. Then watched a movie with the family, and finally got to go home and sleep around 11. Long, but probably worthwhile day. I do wish I could have spent more time listening to music or something rather than driving though.
The movie was Beyond the Gates of Splendor, about the Waodanis and the missionaries who they killed, of which Nate Saint and Jim Elliot are the only two that are probably generally recognizable. It's been like 13-14 years since I read their biographies, so I don't remember much about how much was covered there vs in the movie, but it was definitely interesting. It did focus more on the later changes of the tribe where I remember the books I read focusing more on the initial contact and gaining trust before the seemingly sudden change. Anyways, it's a good movie, and I recommend it (even though I was falling asleep through it). There's another movie called End of the Spear which looks like it should be good too. I think one of the more interesting parts is how their culture has gone one direction, while they see the US as going in the opposite direction (why they allowed the movie to be made about them).
Back to the weekend, Sunday didn't provide a chance to catch up on sleep. Church, and then family birthdays in the afternoon. Did get home and have a chance to relax a bit, but then work comes early in the morning. That was a bit smoother than last week (as we had already run through the stuff last week), but still rather busy. Dave and I are spending plenty of time figuring out strategies and priorities. I'm glad I spent the time pestering to so I have a workable understanding of most everything though, and pushed things just far enough so that I'd have a reference of where it was on the to-do list and prerequisites, as the process could be a lot worse. I think we're making good progress though at cleaning up loose stuff, some of which just sat because it hadn't been revisited yet after things had been resolved, and some just gets the "too bad, this won't be fixed because the whole system will probably be scrapped at some point". I like those because it's less work for us. :) We'll see what tomorrow brings. Besides Sesame Beef that is.
So today we took Brent out for a going away lunch. Right as we're getting ready to go, the power flickers, and a few seconds later the lights go out. A short time later there's a bit of a flicker as the generator tries to kick in, but it struggles for a few seconds delivering way less than necessary, and then goes out. We figure what luck, but our server rooms should have kicked over to a different power feed, which has an independent generator. About this time the VP walks into the hallway to say we should make sure they're up and running OK. So we split up, figuring a few minutes later we'll be on the way to lunch, and hopefully the campus will be back up when we return.
Unfortunately we arrive to find the rooms on battery power, apparently meaning the second (independent) generator system also failed to start up. A few minutes later, we know that it's an area wide thing, both generators almost immediately shut down due to overload, and the fire alarm alert thing is getting old quickly. Why can that thing work great when there's no power, but has issues holding door magnets sometimes when stuff is up? Anyways... So the lunch plans are on hold. We have a crowd of ITS people around the server room just sorta hanging out and wondering when we can go to lunch or alternatively when we need to start shutting stuff down. Some of us are shutting down unnecessary stuff to keep the heat in the room down (AC is on power transfer so can get all 3 indefinite power sources, but not battery). Bethelwulf has long since shut itself down (aka its UPSes ran out of power).
A few minutes later one of the server rooms is below the safe runtimes on the UPSes (below 1 hour is unacceptable, it takes 15-20 minutes if we work very quickly and overlap when possible to get things shut down properly), so it's time to start shutting stuff down. Of course right about the time we're actually planning shutting down production machines that have redundancies (still don't want service interruptions at this point), we hear that the power is going to be back within a few minutes. So that goes to "let's wait 5 minutes" status.
Sure enough, a couple minutes later the AC kicks online and the UPSes off battery - one feed is up and running. A few minutes later the room lights come on, as the normal building feed comes up. Quick status checks, chats with the VP and EVP who are both wandering around wondering how much of our stuff is affected, and we're off to lunch. Of course most people were off to class or work, so were much less excited about the outage being over than we were. After the last big storms came through I almost forgot how annoying actually losing power can be.
Lunch was good, large crowd, although Arby's was packed. Of course leaving at a bit after noon rather than 11:30 will do that. After getting back we discovered that the only lost equipment (at least infrastructure, and so far) was apparently one switch card, which is pretty good. We decide that another battery chassis for the one server room needs to be investigated (especially as a run to power another wiring closet is being run from it soon), and decide getting the room lights on the transfer equipment, even though it wouldn't have helped in this case, could be very handy. In the afternoon I get a call from the electrician wanting to talk about the issues and testing to make sure it doesn't happen again, checking how much delays in other stuff (like getting that feed to the closet in place) may have impacted us, and other stuff. We meet tomorrow after the cause of the second generator failure is determined (the one is unfortunately normal due to the surge load on startup, they have to bring the campus up building-by-building for it to handle it. This is why we have multiple feeds for our stuff now). Should be good to get that figured out.
So that was my day. Anything exciting happen in yours?
So the X5L mentioned previously arrived last week. So far I like it a lot. I had to do some renaming initially as I copied stuff over, but I had to do an equivalent with the old thing anyways since ogg doesn't have track numbers but the metadata did. Anyways, I have a decent amount of stuff transfered over, and the rest has to wait until I boot up other machines (I didn't have much on the powerbook since I always had that with). Anyways, for starters the sound quality is awesome. The response curve is better than the old one, which I thought was better than a lot of other things. With a well encoded file it actually starts sounding rather similar to the original. The noise floor is awesome too. I initially didn't here it, but then I realized the background noise from the room through my earphones was just louder. There's nothing from the screen or drive though, just a very quiet hiss and a pop with the power switch. The UI takes a few seconds to get used to (a coworker said it made sense after about 30 seconds), and seems workable. The screen is bright and crisp, and easily readable from a decent distance or any angle. It doesn't take thinking and it's instant to turn the backlight on or off without affecting other things, which is nice. My only complaint is in audio mode it only updates a few times a second so the audio level graphs are kinda pointless. The thing feels slightly heavy for its size (but still reasonable), but I think that's probably due to the battery. I can pretty much unplug it in the morning when I go to work, use it all day (drive to work, work, drive home, evening at home), and not bother turning it off at all (like when I leave the office or go to lunch), and still have it barely drop off the top of the battery gauge. That is such a nice feature that way too many things lack. Stupid battery technology... Anyways, I have nothing but good things to say about it, so if you're in the market for a digital audio player, you should at least consider it.
As for the week at work, it was busy. I got power distribution for a rack moved to new circuits in one server room, and in the other one added 2 (sorta 3) new circuits. Should allow us to get everything up and running before school starts. Stuff also sorta blew up, but luckily none of it was my fault and I can't really do anything but sit back and wait for it to be fixed. I almost got another new server up, but I'm waiting for it to be able to reach stuff so it can be used. I think I finally got all the new student stuff figured out, and the system assigned permissions this morning. I still sent the list of changes it made to the student supervisors for verification in the morning since I don't know if they got me everything. I also got permissions for student workers for the rest of the school tied to contracts, which should hopefully save time for everyone (they say the access when requesting the job, don't have to think about having it added/revoked since everything's tied to contract dates and signing/termination). It'll be interesting to see how well that works after what's happened though - apparently they couldn't get contracts printed and signed in May, but had to wait until August to request them, so people may have a couple days waiting to get access after getting back and signing them. Such is life I guess, and maybe can be resolved next year.
The weekend was good, with sitting around and relaxing. First movie was Elektra which was better than expected (I had low expectations going in). Other than the intro when I put in the DVD that is. I have something funny if people want to see it. Anyways, the plot wasn't all that great, but it had fighting and martial arts and some interesting effects. Second movie was Wild Things 2. It was one of those where when I heard of it I was wondering how they could possibly do a sequel, so I had to see it. It's basically the same plot as the first one. Yes, same plot, although some of the characters have different relationship roles and some minor details are different. So yea, not such a great movie, unless you want to laugh at the movie. Third movie was Swimfan, which was better than expected. The plot was interesting, although nothing special. The soundtrack was interesting and well done technically, and they used some really interesting editing techniques. Like rather than draw out the scenes to simulate the normal emotion like most movies, there's multiple angles cut together with overlap sorta like a replay. That description actually doesn't do it justice, and it works way better than it'd seem at first. It's sorta combines a this is important with the character replaying what happened in their head to sorta make you look at it from their perspective. Yea, I'll stop before I totally mangle it. You should see if you haven't though.
Copyright ©2000-2008 Jeremy Mooney (jeremy-at-qux-dot-net)
