The move was scheduled for the beginning of the US Memorial Day weekend, to give IT staffers an extra day in case they had problems bringing everything back online and because most of their clients who work normal daytime, weekday hours wouldn't be around. It was scheduled to take place in the middle of the night, at 2am, during the lightest user load on their system.
This meant some extra planning: "At two in the morning, you aren't going to find an extra 5ft cable, so we bought many extra supplies just in case," says Gorham. "Happily, we can return the unused ones to our vendors."
Now, I am not a night person. Usually, I am in bed by 9.30pm but to get this story, I set my alarm and arrived shortly before 1am on Saturday, May 26.
Despite the shortness of my commute, I am probably the last person to get to REJIS's office. There are already about 50 people crowded into the common break room. And, already, something has gone wrong. The new elevator in the new building wasn't working, and Frank, the elevator technician on duty that night, had been called in to debug it.
The total distance for the move from the old to new buildings is less than 100 feet (30 metres) — but it is up two flights of stairs and down a couple of halls. Hence, the need for the working elevator. When Frank heard where he had to go, he was a bit worried. "I am going to pull up to this building in the middle of the night and every cop in St. Louis is going to be there," he told me as he was sitting on the floor of the service room, trying to debug the elevator circuitry. "Sure enough, here I am at this high-tech cop shop."
The reason he is sitting on the floor is because there is this huge pile of manuals next to him. "The manuals aren't well written, and the documentation on the error message that I am getting is very obscure." Sounds a lot like what most of us have had to do with computer manuals.
While Frank puzzles over the manuals, Gorham is downstairs in the break room, giving everybody their instructions for the night. We are sitting next to a large stack of our own — a large collection of pizzas, cookies (carefully home-baked and labelled by flavour by some volunteer), sodas and plenty of coffee. It is going to be a long night ahead and the movers need their fuel. "If you are going to ask people to come to work in the middle of the night, you want to make sure that they are going to be fed and keep them happy while they are working," says Gorham.
To get people juiced, Gorham draws a few random names for prizes — this being St. Louis, more than two dozen people have donated pairs of Cardinals baseball tickets, and he has also purchased several restaurant gift certificates. Then we all go to work.
At 2am, Frank gets the elevator going and the moving crew begins to take apart the control consoles that are needed in the new building. The crew starts to roll the IBM e-Server and virtual tape system across the floor and upstairs into the new building. Everything will sit on top of WorkSafe Technologies' ISO-Base seismic isolation platforms that can dampen vibrations in case of earthquakes.
As each machine is taken off the old rack, it goes by Chassidy's station. She is in charge of blowing out and clearing the accumulated dust with a reverse vacuum.
REJIS isn't moving everything out of the basement; some of it's being recycled or tossed. Staffers are dismantling two robotic tape libraries that have outlived their usefulness, along with 10,000-plus tapes. "Our new library has the same storage with just 1500 tapes and is compatible with the state data centre formats, making it better for disaster recovery purposes," says Gorham. And REJIS plans on recycling a huge pile of bus and tag cables, and other old equipment to EPC Recycling.
Within a couple of weeks, the basement will be completely empty of all equipment, cabling, and even the raised floor will be removed, too. "We don't have any current plans for the basement, but are thinking about a few ideas for it," says Gorham.
By 2.30am, "All of our red servers are in place," says Gorham. "We thought it was going to take longer." REJIS tagged each server by colour, with red being the most critical (law enforcement servers, for example) that needed to have the least downtime. REJIS has a lot of non-law enforcement clients, such as doing the general data processing for a local city government — running payroll jobs and processing other HR data — that was triaged as not quite so mission critical.
By 3.58am, the huge IBM eSeries servers are connected and back online. "I'm happy about the limited downtime," says Gorham. "We were really only down for 118 minutes for mission-critical services."
Seeing how things are going well, I head home for a few hours' rest. I am back by 6.30am and walk into the server room, to see only one cable laying across the floor: "We just had one bad connection that we needed to fix," says Gorham. That was it; all other systems are up and running.
By 7am, as breakfast is served in the break room, most of the crew is finishing up and getting ready to head home. But not before a few lucky ones are picked at another random lottery to get some Cardinals tickets as their reward for a good night's work.
Looking back with the benefit of a couple of weeks of hindsight, Gorham says he is very satisfied with the overall move. "I would have done some of the building's system tests earlier on — such as making sure that major electrical and air conditioning systems were working, and maybe we should have had the elevator guy on site," he says. "And it would have been nicer if we had more new racks to spread out our servers."
But that's about it. All in all, it was a good night for a move.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.