Eileen Strider learned early in her career what it's like to work in a 24/7 IT environment. As an IT staffer with a Midwest defence contractor, overnight systems administrators called Strider on her honeymoon to trouble-shoot a system error.
"It's two in the morning and I hear the phone ringing and I'm thinking, 'There's only one reason that phone can be ringing right now, and I know I don't like it,'" she says. "I ended up talking to the guy for over an hour and we still didn't solve it. Thank God my husband slept through the whole thing."For companies with systems online 24 hours a day, planning round-the-clock staffing requirements tests even the craftiest CIO's wits. If you leave your system unattended overnight, a minor glitch could bottle up operations until morning. If you opt for a third-shift skeleton crew, hiring -- and retaining -- your late-night employees could be more trouble than it's worth.
So what do you do when you've got operations that never rest and people who must? Improvise. Innovative reorganisation strategies, creative compensation packages for late-night workers, progressive alternatives to the traditional IT environment and explicit procedural documentation for the future are all ways in which you can turn your 24/7 operation into a model of efficiency.
For some, like Strider, former CIO of Universal Underwriters Insurance Co., a system that works can be found overnight. For others, like Rich Ogata, former director of network operations and technical services at CyberCash Inc., the process can be painful and agonising, yet educational. Both Strider and Ogata emphasise, however, that when dealing with a 24-hour grind, you have to think of your own needs as well as the needs of your staff. Managing your employees' stress requires clever scheduling and organisation, they say; managing your own stress is entirely different.
Autonomy Is Key
As one-half of Kansas City, Missouri-based Strider & Cline, Strider serves as a consultant to companies with continuous IT operations. But her career began almost 30 years ago as a low-level IT analyst for the McDonnell Douglas Corp. in St. Louis. She climbed the ladder from third-shift trouble-shooter to second-shift programmer to daytime manager and, ultimately, to one of three positions directly under the CIO. In 1993 Strider moved across town to Universal Underwriters Insurance, where she was CIO until May of last year.
Not long after she arrived at Universal, circumstances made it imperative for Strider to initiate 24/7 operations there. Selling insurance to automobile dealerships across the country was not a round-the-clock business, but agents in the company's Kansas City headquarters processed orders from four different time zones. Mounting backlogs every evening forced the agency to run its systems all the time.
In the months immediately following the switch to 24/7, Strider had no formal plan for staffing the overnight shifts. A programmer was chosen randomly to be on call each night. When problems arose, whoever detected them phoned the programmer on call. In any given week, Strider estimates, one of her people was called upon at least five of seven nights. Morale, however, remained high, because of what Strider affectionately terms the "saviour" syndrome.
"Did they like coming in to work in the middle of the night? No. But in the beginning, they loved being heroes," she says. "If they could fix the system time and time again, they thought they were proving they could save the company from total disaster. It was a macho thing. They saw themselves as indispensable, and they loved every minute of it."As the system grew, the 24/7 trouble-shooting solution of the past began to falter, causing more late-night crashes, Strider says. Some staffers who had worked consecutive nights were too tired to perform their daytime duties properly. Others fell ill from physical exhaustion. Then, in early 1994, the company moved its data centre from Kansas City to Chicago. With the centre more than 500 miles from the IS staff, working without an overnight strategy would have cost the company far too much money. Strider says she had no choice but to completely rethink the way she had been dealing with 24/7 operations.
First, Strider ordered extra phone lines for the homes of almost 30 people on her IT staff. When staffers were called upon to fix problems after hours, they could dial in to the system from home rather than come in to the office, and they could trouble-shoot over the phone with systems analysts in Chicago about the repairs they were making as they made them. Universal picked up the long distance bills and paid the staffers a 15 percent premium for the work they put in from home.
While these initial steps didn't eliminate her staff's frustration with the 24/7 operation, Strider says they were important because they showed her people change was on the way. "These preventative actions gave my people hope that their lives would get better," Strider says. "I saw maintaining this sense of hope as a top priority, because without it, I knew there was no way I could prevent burnout [among them]."Strider herself worked some overnight shifts and had an extra line installed at her home office so that she could help her staffers with the most difficult cases.
Months after the last of the home phone lines were installed, Strider unveiled a reorganisation strategy that revolutionised Universal's approach to staffing in every department. She let her IT staffers divide themselves into teams, and each team was assigned a group of systems to monitor based on their expertise and their input. In all, there were five teams, one for each set of systems.
Strider chose an IS manager to head each team, and she looked upon those leaders to schedule work rotations as they saw fit.
"As long as every team was monitoring its system 24 hours a day, seven days a week, I didn't care how they did it," she says. "My philosophy was to give people as much choice as I could. The more autonomy they had, the less put-upon it seemed they felt."Strider set up ground rules designed to prevent burnout and improve productivity. All employees had to work a minimum of 40 hours each week, and no employee could work more than two consecutive overnight shifts in any four-day period. Telecommuters were required to supply their own hardware and to spend at least one day in the office each week.
Some staffers worked 10-hour days and others worked 12- or 15-hour days. Others instituted rotating on-call schedules or opted to work from home, telecommuting on the same phone lines installed to help maintain the 24/7 operations in the first place.
To her surprise, Strider says that few staffers had trouble adapting to the new system. The first time an employee required discipline, Strider let her team leaders handle the issue. If the employee's behaviour didn't improve, Strider handled the situation herself. When delinquent workers failed or refused to report for their overnight shift, Strider instructed the leaders to plan their schedules for them. Strider says she remembers only a handful of discipline problems in three years. Not once, she adds, did she fire somebody for breaking the rules.
"They didn't ever abuse the system because it was a system they created," she says. "They were so proud of the way we had responded to the 24/7 question, there was no way they were going to do anything to ruin it."Healthy competition between teams prompted everyone in IT to do better, and within a month or two the number of positive customer service reports from the data Centre in Chicago and from internal clients across the country went through the roof. Systems performance improved tenfold, particularly between midnight and 6 a.m. Perhaps most important, Strider says, department morale soared.
"It was the hero mentality all over again," she says. "Trying to run a system 24/7 is tough, no matter how you look at it. It's tough for your people and as a CIO, it's tough for you. But when you give your people the chance to do it their way, not your way, they tend to cope a lot better. It gives them a sense of ownership, a sense of control. And I guess it makes those desperate calls at one in the morning a bit more bearable."Get It in WritingManaging 24/7 operations for a company like Universal Underwriters, a US$650 million concern with a 160-person IT staff, is certainly a challenge. Imagine, then, managing round-the-clock operations for a startup with fewer than two dozen IT employees. Rich Ogata says that task was overwhelming and deleterious for CIO and staff alike. When Ogata, former director of network operations and technical services at CyberCash, was hired in 1993 to build the company's 24/7 operations, he did most of the work himself. At the time, the company was operating off three servers in the CEO's basement. Almost three years later, more than 200 people worked out of the company's offices in Reston, Va., and a handful of them maintained the company's 24/7 operations under Ogata's management.
In the beginning, the company's IT staff was too small to warrant an overnight staffing plan. As it was, Ogata says, he'd work 14-hour days, often trouble-shooting well into the night. As the company (and the IT staff) grew, however, so did the need for some sort of staffing strategy. Ogata decided to hire a handful of IT professionals to cover 10-hour shifts in a four-day workweek; one person would work mornings, one would handle afternoons and evenings, and one would tackle the overnight. A fourth person was always on call.
Still, Ogata never had enough operators to cover the company's operations around the clock. Weekend shifts were particularly hard to fill, and what Ogata couldn't monitor himself, he relied on electronic monitoring systems to detect. It was not uncommon, Ogata says, for the monitoring systems to page his staffers-and him-three or four times a night.
"I won't pretend we weren't on edge because we were," he says. "You had to be dedicated to the point of insanity. I told my staff I was going camping one weekend and one of my managers said something like, 'I can't believe [Ogata] is leaving his babies.' As important as it was to walk away sometimes, you never wanted to stray too far."Burnout struck fast in this high-pressure environment, and Ogata says that at least two of his network engineers left because they couldn't deal with the stress. On the rare instances when other staffers failed to report for their shifts, Ogata had to pick up the slack himself, usually via his ISDN connection at home. With such limited resources, Ogata admits that aside from creative scheduling, there wasn't much flexibility he could offer his employees to alleviate feeling overwhelmed. In many cases, he says, impromptu pep talks were the best remedy.
Still, tempers flared regularly. Stress-related personality clashes occurred on a daily basis. One by one, Ogata's top four people, about a third of the IT staff, resigned over a seven-monthperiod. As his staffers walked out the door, Ogata discovered just how much essential information each of them was taking. Weeks after people had left, Ogata found himself calling them to ask about a program they wrote or about a recurring glitch they fixed once or about where they had put a server they reconfigured. Out of this total chaos, Ogata says he learned his most valuable lesson as a CIO: document, document, document.
"When it comes to understanding 24/7 operations, there mustn't be bits of information locked up in people's heads," he says. "You can't be calling somebody after they leave saying, 'Where'd you put that server?' because that information is too crucial to systems operation for you not to know. As a CIO, you've got to have a good handle on all the procedures and responsibilities for every position on your staff, and it's got to be laid out explicitly in an employee manual."With his staff severely decimated, much of the departmental work fell on Ogata's shoulders. Within a few months, the pressure became too great. After finishing an extensive manual of IT at CyberCash, residual stress from the resignations forced Ogata out last spring. Ogata tried a brief stint in the IT department of a nationwide bank; ironically, he found the pace too slow. Later this year he signed on as vice president of network applications at Emaginet Inc., another startup where he was hired specifically to develop the company's 24/7 operation from scratch. Now, he says, he's documenting everything as he goes, constantly trying to assess actual workloads. Already, he's seen that to run the system efficiently, he'll need to outsource the IT data centre and establish a separate monitoring staff for all round-the-clock systems. Already, Ogata says, the lessons he learned in his last days at CyberCash are paying off.
"Sharing responsibilities-getting them out in the open and having your people understand them from the beginning-reduces the stress factor for everyone," he says. "When you're dealing with operations that go on all the time, people need to know what needs to get done and how they can do it. If you can't tell them that, if you can't get enough a grasp of your own system to put it to writing, there's no way you can expect your people to do things right. In the middle of the night, the last thing you want is a work-related phone call." Is Your IT Staff Overworked?They live on coffee and peanut-butter cups, the bags under their eyes could hold groceries, and still they put in 15-hour days. Still not sure? Look for these five top signs of staff burnout.
-- Declining health
The more tired your workers are, the more susceptible they are to falling ill.
-- Absentmindedness
Exhausted, stressed people are prone to careless errors. They overlook things they normally remember to inspect.
-- Flaring tempers
Can't we all just get along? Not on three hours of sleep, we can't. When your people bicker over simple things-lines of coding, a server password-you know it's time to break it up and give somebody the afternoon off.
-- Procrastination
Just as it's easy to stay on task when you're rested, it's easy to be distracted when you're not. Especially if your staffers have Web access, the possibilities for procrastination are endless.
-- Substance abuse
This is, without a doubt, the most serious of all the symptoms. Stress, both at home and in the workplace, can lead even the most stable people to drug and alcohol abuse.
(Matt Villano is a freelance writer based in New York City. He can be reached at whaleman@tiac.net.)
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.