SAN Nightmare, Part 6

Note: This is part 6 of an 8 part series. Read them in order, it'll make more sense. Part 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8

Part 6: More Suffering

Monday, June 9

My boss announces that we are no longer allowed to use his corporate credit card to make purchases, because his assistant has transferred to another department and will no longer be reconciling the bill for him. Later this afternoon, someone decided to try to copy 200+ GB to their lab volume, which is hosted on one of the external drives. The drive fills to 100% capacity, and suddenly we are in need of an additional FreeAgent Pro. This minor crisis is the justification for having Accounting assign me my own corporate credit card.

Thursday, June 12

The 'san4' volume starts crashing repeatedly mid-day, much in the same way 'san5' did on May 27. This particular volume was the exclusive host of the file server data for the company we share the building with, and is therefore a critical part of our network. It becomes necessary for us to take this volume offline and move the data on it to a new drive as soon as possible. With no more FreeAgent Pro drives available, I had to use a lesser FireWire drive. The lack of eSATA slowed things down considerably. Meanwhile, we placed a rush order for another FreeAgent Pro drive, another eSATA card, and two internal hard drives for the server that we're turning into a dedicated server for this group.

Friday, June 13

The data copy was complete by about 1:00 PM. We had decided to take this chance to move this particular set of files to a dedicated server to avoid future problems like this, and because it makes things much cleaner for us from an administration standpoint. I began the migration to the new server at 6PM.

At 4:00 PM, the same instability issues previously experienced in the san5 and san4 volumes have spread to san6. We are forced to take two more labs offline to move them to the new FreeAgent Pro drive purchased the day before.

Both copy operations are successful and wrap up faster than anticipated, leaving some spare time over the weekend to come up with a more comprehensive plan for final removal of xSan from our network.