2004/06/02
23:58:26
So I'm sitting here watching a document roll itself up, move a few inches, and then unroll itself. This is all happening in a small grey box. This part is only like 35GB (it totals 52GB, but I got the rest already), but it's the part I'm worried about. Windows is apparently flaky with large files - this is only two files, one 24GB and one 13GB. It kept complaining the source wasn't there anymore, or the path was two long, crazy stuff. Always in the middle of the copy. Seems more stable now though. The data that's being copied is part of the storage for an exchange server. This past Monday the Information Store (process that handles storing all the data in these funky database files) "terminated unexpectedly". Odd, but MS stuff seems to do things like that occationally, so just started it and let it go. It was acting a little odd, but just let it go, especially due to the other email issues it probably was getting pounded with email traffic. Then it happened again. And again. And got really slow randomly. Yea it was occationally spitting out error messages, but the documentation when you looked them up said stuff like permissions not right or that the Information Store isn't running (imagine that, seeing as the error was generated when it crashed). So finally (in the sense that only people who have done lots of computer troubleshooting understand) today it gets a few errors which point in the direction of database corruption. The fun part is that the backup guy's been fighting with exchange and the tape drives trying to get a complete backup for way longer than we'd like (so restoring would be bad). Of course it's the middle of the day and it's still running, so gonna leave it up. I immediately created a couple other stores and started migrating boxes over (the enterprise versus standard license is probably worth it just for that ability if you're serious about using exchange). It's taking forever, but the more mailboxes out of the half broken database the better. But we decided better safe that sorry, so at 8pm shut the exchange stuff down totally until 7am tomorrow and are just doing a copy of all the important exchange files. At least we'll have a backup then. After that I think I'll try to have it migrate all the boxes to other stores and then we can unmount and repair the other one without downtime. I don't normally mess on that server much so didn't realize how much they had on them. I think I'm gonna push for a few smaller stores rather than one big one. The idea of a 35GB Jet database on an NTFS filesystem doesn't give me the most piece of mind... As long as this rate of people posting to the outages list doesn't keep up. Although I still find the one about someone taking out one of our office buildings by knocking an uplink cable off in the closet with big orange road cones sorta funny. Almost like the people who kept accidentally unplugging the switch which sounded like a industrial blower by practically sheering off the plug at the outlet and then not noticing and wondering where their network went a couple years ago (yes, it happened like 3 times too, and we did tell them why between times - shared closets suck). Other things that happened today include we packed up the NAS that we're sending back. Now you can move in the front half of my office and that's about it. It should ship out Friday though since tomorrow is the staff retreat, so don't have to live with it too long. Last night was more AD cleanup and emailing people about their accounts. Hopefully not much more of that left. Woohoo! The 13GB file just completed successfully, now on to the 24GB file... We announced yesterday our plans for Windows and this summer, and no complaints yet. That's a good thing. To wrap this up, today for lunch Ross and Nick and I went to Cattle Company. They messed up our order though, so we had to wait for new food to come out. Sorta annoying, but it was the 3 of us so we just sorta shrugged it off as don't go back to work as quickly. We did get more free bread out of the deal though.
That NetBackup exchange agent is pretty cool...