SAN Nightmare, Part 4

Note: This is part 4 of an 8 part series. Read them in order, it'll make more sense. Part 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

Part 4: Recovery

Wednesday, May 21

Restore operations have been ongoing since Saturday night, and the last batch of lab restores completed by the end of the day. A full week of downtime for some labs... and their files are being restored from the night of May 8. Ouch. The IRT and Public volumes are still offline. Restore operations at this point have taken a back seat to making sure the backup operations run properly.

Friday, May 23

After working with Apple over the course of the week, it was determined that a software patch was necessary to prevent recurring issues. In addition to the catastrophic failure of our "san1" volume, we had been experiencing other problems throughout the week with our other SAN volumes. Specifically, the would crash on a very regular basis, usually at least once a day. These crashes required frequent server reboots to address the issues. We were provided with a patch that contained a known fix to the known bug we were experiencing, which I installed in the evening while the server was offline for scheduled maintenance.

Almost immediately after installing this patch, I confirmed that the previous problem we had (random fsm crashes) seemed to be fixed... but that a new, much more serious problem had been introduced: random segmentation faults that corrupted the entire operating system. Oops. After running a few more diagnostics, we reverted the software back to the original less-buggy version and went from there.

Tuesday, May 27

The long holiday weekend gave me a chance to get the last volumes restored from tapes and back online. At this point, I finally get access to my files again -- I had been without them for 12 days, and work was piling up. By noon, another one of our SAN volumes (san5) had become very unstable -- three crashes in under an hour. Each time it crashed, it hung the file service processes as well -- file servers get angry when you disconnect the disks that hosted files live on without warning. We had to take the four labs living on this volume offline for the remainder of the day. By 9:00 PM, all data on the san5 volume had been moved to another Seagate FreeAgent Pro disk.

SAN Nightmare, Part 3

Note: This is part 3 of an 8 part series. Read them in order, it'll make more sense. Part 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

Part 3: Disaster Strikes

Thursday, May 15, 2008

At 10:00 AM, the "san1" volume, which is our largest volume with 5TB in total capacity and 11 of our 17 research divisions' (and IT) data, crashes and refuses to re-mount on either server. After a series of reboots and continued failures attempting to get the thing to mount, another call to Apple was in order. By noon, Apple determines that we should probably start restoring the data from backup tapes, since the prognosis does not look good. I start running some diagnostic tools for Apple while Michael loads the most current tapes in the tape library to begin the restore operation. At this time, we discover two things: (1) The volume "san1" is the only one that has not properly backed up in the last week (since the upgrade to xSan 2.0) due to the backup process crashing while running, and (2) The robotic arm on our tape loader has picked this exact moment, of all possible times, to fail. So, we have a drive that won't mount because it's corrupt, backups of that drive that are one week stale, and no way to read the stale tapes anyways because the robotics on the tape loader have failed. The first order of business, then, is getting the tape library working so we can read the tapes.
We put in a call to Quantum support about getting the library repaired and were told that since the device is out of warranty, they won't even talk to us until we purchase a support contract. The support contract department is out of the office for the day, so we should expect a call back tomorrow.

Feeling completely helpless, I decide to go home by 4:00 PM to get some dinner and some rest, because I have to be back in the office at 8:45 PM to start taking equipment offline for a large planned power outage. The impeccable timing of this disaster plus the planned outage kept me at the office until 2:45am.

Friday, May 16, 2008

By 8:00 AM, I'm back at the office running on a little less than 4 hours' sleep. Efforts to contact the folks at Quantum are unsuccessful all morning, so I leave to run other errands while Michael continues to try to talk to the contract folks. I head to Fry's to buy two 1TB Seagate FreeAgent Pro drives so we have somewhere to put the data once we start restoring it. (Side note: The FreeAgent Pro drives are eSATA/USB2/1394 and are awesome. I highly recommend them.) Also on the way back, a trip to Costco was in oder to pick up beer and desserts for the IRT-hosted Happy Hour that was scheduled for 4:00 PM. We had already booked and paid for the catering, so we couldn't cancel the thing. Another case of impeccable timing.

Shortly after returning to the office, we finally manage to get in ouch with Quantum regarding the support contract. We ended up paying a bit over $3000 for the "Gold" maintenance contract which entitles us to 24/7 on-site support. They diagnose the problem as a bad picker hand and schedule a courier to deliver the part by 4pm, and a technician to install the part by 6pm. Convenient: The IRT Happy Hour ran from 4-6pm.

The Quantum tech shows up, installs the new picker hand incorrectly, and continues to get the same error message as before on the Library. Then, he tweaks something and manages to run over the umbilical cord that connects the hand/picker to the rest of the library's electronics at about 8:30pm. Since it shot sparks, Quantum decided to send another umbilical out to us. I've been in the office for quite a while at this point, so I send the service tech home with instructions to come back in the morning. Quantum delivers the part to my apartment at about 11pm, just as I'm finishing up watching the season finale of The Office on my DVR.

Saturday, May 17, 2008

9:00 AM: Arrive at Office.
10:00 AM: Replacement umbilical cord installed. Same error.
10:30 AM: Service determines the problem is in the (already replaced) picker.
10:35 AM: Closest picker is in Irvine, it is being sent by courier to arrive at 2pm.
11:00 AM: Lunch.
2:30 PM: Picker arrives late due to bad traffic in Oceanside/Del Mar.
3:30 PM: Same error. All field-serviceable parts have been serviced. Quantum replacing entire chassis.
3:45 PM: Closest chassis is in downtown LA. Estimated arrival: 8:00 PM.
3:46 PM: I send the service tech home. I don't trust him anymore. I'll swap the chassis myself.
7:01 PM: Courier must have broken all sorts of speed laws to get chassis to us by 7pm.
10:30 PM: Restore operations begin. I go home.

Sunday, May 18, 2008
9:00 AM: Service tech returns to pick up the bad unit. The new one is working fine, thanks.

SAN Nightmare, Part 2

Note: This is part 2 of an 8 part series. Read them in order, it'll make more sense. Part 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

Part 2: Imminent Failure

Monday, May 12, 2008

In the morning, our LDAP server at work manages to get its internal account database corrupted. This server issue has absolutely no bearing on the xSan project other than its timing -- I ended up spending most of the day on Monday running around trying to fix other systems that were affected by the LDAP outage instead of paying attention to the backup scripts I'd started on Saturday afternoon.

Tuesday, May 13, 2008

Our weekly IRT meeting focused mainly on the LDAP failure from Monday, and how to better communicate things like downtime in the future. A brief wrap-up from the SAN migration over the weekend was presented, with the verdict that things looked good to this point. After the meeting, I checked on the backups and noticed the first problem: instead of speeding up the backup window, xSan seemed to be lengthening it dramatically. We had six SAN volumes, each are supposed to back up nightly. We had one dedicated computer to run backups and the ability to add a second if necessary which would give us two simultaneous backup processes at most. The "san5" volume was single-handedly taking about 27 hours to run an incremental backup... on its own. As a result, our other volumes are being skipped over for backups because the process is taking so long. I make a few changes and set up the "san1" volume to start a backup operation.

Later in the afternoon, we start having odd problems with some of our share points on the server. It turns out that people in specific labs aren't able to connect to their files, because the volume their data resides on has un-mounted itself from the file server. The odd thing was that I couldn't get it to re-mount on that computer -- but, it would mount on the "spare" server just fine. Over the course of the afternoon, I re-configured al the server share points onto the new hardware and moved the DNS records over. This allowed everyone to connect to the new box using the same server names. Everything seemed happy.

SAN Nightmare, Part 1

Note: This is part 1 of an 8 part series. Read them in order, it'll make more sense. Part 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

Part 1: The Upgrade

Friday, May 9, 2008

On the evening of Friday, May 9, I scheduled downtime on the file server to upgrade our version of xSan from 1.4.2 to 2.0. I had hoped the new version would fix some random, nagging problems we'd been having with the software such as occasional unannounced server reboots and problems with certain types of files. The random reboot thing was happening more or less on a weekly basis and seemed to coincide with some larger backup operations we were doing. The upgrade was also (hopefully) going to help our backup server more effectively back up the data on the SAN by improving read/copy speeds.

I downloaded the migration guides and read them over prior to starting the upgrade. The guide mentioned the need to wait a period of time (sometimes a few hours) for the volumes to update their metadata to the new 2.0 format before they would be available. I installed the software and noticed the volumes were showing up in the GUI admin tool as being available after a few minutes, and didn't think much of it until I went to try to start/mount some of them and began to receive errors. After a bunch of times where I froze up the GUI by trying to start a volume and had to force reboot the server, I decided to call Apple. The support rep on the phone mentioned something that was not notated anywhere in the migration guide at all: you have to just let the upgrade run its course before trying to start the volumes. Problem: there is no progress bar that tells you (a) whether the update has started, (b) whether it is running, or (c) when it is done. This update is all done silently in the background, and can take "hours" depending on what exactly you're storing there. To determine whether the RPL update is done you have to go hunting through the system logs for a very specific (undocumented) file and search for the string that denotes entries related to the upgrade. Thanks for documenting that, Apple.

Saturday, May 10, 2008

After the support call with Apple from the previous night, I decided to just let the "RPL update" run its course overnight, and come back in the morning to see how things looked. With all the RPL updates done and all our volumes mounting properly, things seemed to be in good shape. I brought the server back online at about 2pm, re-configured the backup routines, and told them to start backing up the server.

SAN Nightmare, Part 0

Note: This is part 0 of an 8 part series. Read them in order, it'll make more sense. Part 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

Part 0: Introduction

This series of entries timelines select events that happened between May 9 and June 23, 2008. The purpose of this series of entries is primarily so I can remember how bad the last two months have really been. In the process, maybe someone will stumble along this and decide that xSAN has as the potential to have as much of a "distaste for your environment" (Apple's words) as it did for mine.

A bit of background: We had been running xSAN 1.4.2 software since late November, 2007. Due to some issues we had with it (explained later on) we decided to upgrade to the new version in hopes of a fix. When we originally implemented the xSAN solution, we did so because we were intrigued by the idea of allowing multiple servers to share a single large pool of disk space. This would, in theory, allow us to do things like sharing a single public folder across several servers or move groups of people from one server to another for load-balancing reasons without having to move their data. Furthermore, it allowed us to set up a model in which a computer on the SAN was dedicated specifically to being the computer that Retrospect sent all its backup requests through. Structuring backups in this way freed up a considerable amount of CPU space on the file server itself to do things such as serve files in a timely fashion.

New Apartment, Part 3

As promised in the previous post, pictures have been posted of the apartment with all the furniture where it now lives. Additionally, photos and art have been installed on (most of) the walls. Things are looking better, but still need some work.

Also included are a few quick snapshots of the garden I planted last weekend. Enjoy!

Link to Photo Gallery

New Apartment - Furniture

After a very long delay, I have finally gotten around to uploading pictures of my new apartment with actual, real furniture in it. Between moving, unpacking, decorating, a crazy two months at work, and trying to actually be social, I haven't had time to bother getting these things uploaded until now.

Most of this gallery consists of photos taken during the "unpacking and settling in" phase, so a lot of the furniture is no longer arranged as shown in the photos. Another set of pictures will follow with the "final" furniture layout.

Click Here to link to the photo gallery.

Anza-Borrego Pictures Uploaded

See the entry below for pictures of my trip to Anza-Borrego back on Easter to take pictures of wildflowers.

Padres Game

For being such a great customer, Cox gave me a stack of tickets to the Padres' Friday Night game of Opening Week against the Dodgers. In the Cox suite. These are some pretty nice digs for a baseball game. If you've never been to Petco Park or sat in the Garden Level suites... they are right behind home plate and at a perfect height to see the entire field (and catch fly balls.)

View from our seats.

After the unfortunate (blowout) game, there was a special fireworks show for what the Padres were calling "Military Opening Night." Some of the photos that I took of that ended up coming out alright, as well.

Postgame fireworks display

In the end, a good time was had by all. Hopefully, I'll get tickets to another game soon!

New Apartment

I went over to the new building at Crossroads on opening day, a week before I was supposed to get my keys and move into my new place. The maintenance staff hadn't completed inspecting and finishing up my apartment... and accidentally left it unlocked. I took the opportunity to go in with the camera and tape measure and take lots of pictures (and lots of measurements.)

Photos of the (empty) new apartment are here.

Photos of the place with furniture will come later.

Anza-Borrego

I took a nice Easter drive out to Anza-Borrego today to do some wildflower viewing. Also did some off-paved-roading in my 2-wheel-drive low-clearance car and almost got stuck in some sand. Saw a Mustang that did get stuck in some sand. Despite the fact there were several vehicles that stopped to help, one of which had towing ropes... the ladies who were driving the Mustang refused help of any sort and insisted on waiting for a tow truck to come from an hour away to help them.

Update 24-Apr-2008: Pictures are online! Click on the pic below for the link to the gallery.

Small Victories

Time Warner updated the operating software on all of their cable boxes in my area this week. It happened to the standard-def box I have in my bedroom first, and I was quite annoyed when it happened because it managed to delete (seemingly randomly) about 2/3 of the scheduled series recordings I had set up.

This morning, I noticed my HD box in the living room had been updated. Same annoying problem. But, a new behavior out of the cable box: on non-HD channels, instead of putting up stupid, ugly gray vertical bars on either side of the 4:3 picture to make it fit in the 16:9 window, it now outputs solid black bars. I might actually be able to watch non-HD programming on that expensive TV without wanting to throw things at it. Like I said, it's the small victories...

I'm Moving!

On April 7, I get the keys to my new apartment. This is important for several reasons. In no particular order:

- I will be freed from the mold-ridden, flood-prone craphole that is my current apartment.
- I will no longer be living on the first floor, and hence less prone to water issues.
- I will be moving into a 1-bedroom apartment, and therefore once again living alone.
- The new apartment is in a brand new building. I will be the first person to ever live in my unit.
- The new apartment is big at 862 square feet, and cheap for comparable units in the area.
- I will still be close to work. Only about 2.5 miles each way.

Here's an official floor plan, borrowed directly from their website:

Official Floor Plan

Get a good look at that? Good, because that's not at all what my unit will look like. I'm going to have a mirror-image flipped version of what you see above. I took the liberty of flipping the image horizontally so you can see what my unit will really look like. Note that the text in the image isn't so kind as to magically stay put.

My Floor Plan

Now that this is settled, I can start furniture shopping. I've only been putting that off for, oh... 3 years. Should be painful and expensive.

Hoover Dam

On my last day in Vegas, I decided it would be a better use of my time to head out to Hoover Dam and take a tour, as opposed to spending another crowded, sweaty day insaide the show halls at CES. Plus, it would be very un-like me to rent a car and put less than 50 miles on it... I average a lot closer to 2,000 miles per rental, after all.

The drive is about 20-30 minutes or so, even when going the speed limit in my bright Orange cop-magnet of a rental car. I managed to walk right up to the Visitor's Center at 10:50 on a Wednesday morning, and immediately get onto the 11:00 Dam Tour. Normally, people have to wait all day to get one one of those things. Ah, the advantages of traveling alone...

The gallery contains about 100 pictures from inside and outside of the Dam. Enjoy!


Nevada Generation Room

1930's-style waterproof light switch

The Hoover Dam Complex.

My Cop-Magnet rental car

CES 2008

Inside the gallery are a few pictures from inside CES 2008. Unfortunately, none of the pictures I was able to snap off adequately convey just how insanely packed the Convention Center was -- I am fairly certain I have never seen so many people in one place in my entire life. And with over 3.2 Million square feet of space inside the convention center, stuff was still spilling out into the parking lot and into other hotels! Pretty insane.

R2/D2 Home Entertainment Center

People, People Everywhere...

Las Vegas at Night

This gallery contains all the pictures I took when I was in Vegas for CES... but not actually on the show floor at CES because it was nighttime. The first night I was in town, I took a stroll on foot starting at my hotel and working my way through the Hooters Hotel and Casino, followed by the Tropicana and MGM Grand. From there, I took the Monorail to Bally's and walked through that and Paris. I then went across the street to the Planet Hollywood Hotel and Casino and the attached Miracle Mile shops. After all that walking, I was beat and headed back to the hotel early to get some rest for the long day ahead of me on the show floors at CES on Tuesday.

Paris Las Vegas.

Christmas Tree and Skylight

Pictures in this gallery:

- The Dog.
- The new enormous fake Christmas tree.
- The skylight that blew off the roof.

All pictures taken on either 1/1/08 or 1/5/08.

Enormous Fake Tree.

Christmas Lights

On Christmas Eve, Mom, Nicole, and I went for a drive to go looking at Christmas lights. There's a court nearby in Citrus Heights that goes pretty overboard with them, so of course we had to go there. This gallery contains the pictures that came out OK.

The Grinch!

Iron Mountain Hike

On December 22, 2007 I went on a hike with John, Kristine, and John's parents at Iron Mountain. Other people were supposed to come too, but they chickened out... or something. I forget why exactly they didn't make it.

View from the top.

2007 LIAI Holiday Party

Photos from the LIAI 2007 Holiday Party, which was held at the Prado in Balboa Park. In November.

Me, getting ready to eat dinner.

Flooding and Wildfires

Some photos from the aftermath of the most recent La Regencia "Flood" and the soot dropped from the San Diego wildfires. Photos taken between 10/19/07 and 11/10/07.

I <3 La Regencia Maintenance.

Nasty post-fire air filters.

Mission Trails Hike

Pictures from a hike I took on September 30, 2007 at Mission Trails. Nothing really exciting here, but it's time to start flushing out the backlog of photos.

Amusing sign.

New Ball

I bought a new bowling ball. My old ball is too light - only 10 pounds. I wanted a 13 lb. ball, but this one was on sale:

Yeah, the name is tacky, but at least the rest of the ball isn't ugly. The ball is 14lbs, and is going to take some getting used to. It was on clearance, which means I got it for $50 off and for just a couple dollars more than the el-cheapo model. Hooray for marketing product cycles! The new model is identical except they made it uglier and changed some wording. I also bought a new bag to go with it that holds my shoes.

10.5 Server Pain

So, I tried to upgrade a server at work today to the newest most awesome OS X 10.5 Server from Apple. This particular server is not currently in live production but will be soon -- it is a meta data controller for the SAN we're about to install. If you're not a tech geek I don't expect you to know what that means, but bear with me. The point is, it's not a machine that people will call panicked when it stops working -- nobody knows about it yet. The hardware itself is recycled: it's an old file server that was removed from active use a year ago when we moved into the new building. As such, it's about three and a half years old.

I pop the 10.5 install Disc into the drive and boot. Computer gets angry at me and spits it back out, and thus I encounter the first problem: stupid old cheap servers have CD-ROM drives and can't read DVDs. The install media is a DVD. Can't install off of a disk, so we go to plan 2.

Plan 2 is to create an image of the disk and attempt to network boot the server off of said image and run the installer. Halfway through the boot process, the server decides "nope. i don't feel like it" and just flat out turns itself off. I have never seen this behavior before, but it insists on doing it several times. No worries... onto plan 3.

Plan 3 is to find an external DVD-ROM drive and hook it up to the computer via FireWire, then boot off of it and install the software. I try the front firewire port on the computer... not working. Try both ports on the back... still not working. I must have a bad drive. I try a different drive in all the ports, and it doesn't work either. Then, I remember: The FireWire controller on this particular computer is broken. I am starting to get annoyed. This is why we don't normally use old hardware -- it's too much of a pain in the ass. On to plan 4.

Plan 4 is to find a replacement internal optical drive for the server that will read DVDs. I debate stealing one temporarily from another (live) server and think better of it because the server is in fact live... then head back to my office to see about purchasing said drive. The Internet says they cost $200... not a viable option. Then, I notice: They are the same part number as drives that are fitted into Apple Laptops!

Time to cannibalize a drive from an old laptop. I find one and install it. It works perfectly. Victory, at last. And what a pain in the butt. Ugh.

$58.26

I received my check for Jury Duty today. Comes out to $4.42 per day for mileage and $15.00 per day for my time... for three days. Expenses incurred didn't exceed this amount, so I guess that's good.