In a fertile valley in the heart of the country's most populous state, visionaries and entrepreneurs replaced the apricot, plum and olive trees with semiconductor plants, supercomputers and data centers, driving the country to record levels of prosperity.
Then the lights went out.
That's the short version of how California and Silicon Valley - nation's economic engine during 10 years of record growth - were broadsided by an energy crunch that has shaken the state and sent ripples all the way to Washington, D.C. For many of Silicon Valley's high-tech giants, the rolling blackouts have provided a shrill wake-up call: Local governments and utilities can't reliably provide the energy needed to keep critical IT systems and data centers up and running all day, every day.
Even for businesses miles from California's energy crisis, with energy needs considerably smaller than those of the power-hungry Internet server farms that dot Silicon Valley, the West Coast-style woes could be looming on the horizon.
Energy experts say increasing demand and an aging electrical transmission grid mean that power shortages could soon affect neighboring Western states, and even certain parts of the East and Midwest. "Other parts of the country, such as New York City and Long Island, are saying they are short of capacity and getting shorter," says Steve Taub, director of distributed resources at Cambridge Energy Research Associates, an energy consultancy in Cambridge, Mass. The simple problem, Taub says, is that prices for natural gas (on which utilities rely) soared as the economy and demand for electricity took off.
Although most regions are not facing a California-type power crisis, experts have advice for IT leaders dependent on local utilities and legislatures: Better think outside the grid by dusting off Y2K contingency plans and investing in backup systems that will cover you in case someone pulls the plug on your power supply. And better look closely at what some companies have been doing for years to guard against power failures.
"Most of us took it for granted that power was always going to be there," says David Cooper, CIO at the Lawrence Livermore National Lab in Livermore, Calif. "From now on, power will always be a concern. As CIO's, we need to focus on risk planning and backup scenarios."
Less than two years ago, energy lagged far behind the staffing crunch, housing prices and gridlocked traffic as major concerns for high-tech companies trying to lure talent to Silicon Valley. Unemployment rates hovered just over 1 percent in the cities and towns between San Jose and San Francisco, and modest bungalows sold for more than US$1 million. Slowly but surely, however, the state's flawed electricity deregulation plan combined with rising natural gas prices to put a squeeze on power supplies, eventually leaving the state teetering on the brink of disaster.
In January and again in March, the unthinkable happened. Blackouts similar to those that occur regularly in some developing nations hit the nerve center of modern technology. Two days of rolling blackouts, in which power was cut in progressive waves across parts of the state, cost California businesses about US$1.7 billion in lost productivity, according to the Electric Power Research Institute (EPRI) in Palo Alto, Calif. High-tech and low-tech companies alike had to shut down. Now, as summer approaches, many warn of further energy problems as state reserves continue to hover at dangerously low levels.
"There is a very good chance of increased rolling blackouts this summer," says Frank Wolak, a professor of economics at Stanford University and chairman of the California Independent System Operator's market surveillance committee. "The problem is becoming so dire that lost economic output could have long lasting effects on the national economy." (The Independent System Operator was established after deregulation to run the transmission network in California.) The Big "On" Switch for Dotcom America Even before California's energy crisis hit late last year, some high-tech leaders invested in elaborate backup systems to keep the juice flowing in case of catastrophe. One of those companies, Exodus Communications in Santa Clara, Calif., which runs Internet data centers around the world, serves as a prime example for those who can't afford even a flicker of downtime.
Exodus, which has 10 Internet data centers (IDC's) in the immediate area and 42 in the world, helps power websites for more than 4,000 corporate customers, including General Electric, McDonald's, Merrill Lynch, Microsoft and Yahoo. Forty percent of the Internet traffic on the world's top 100 websites zaps through an Exodus facility.
Just as the company maintains 24-hour security for servers that run the websites, the Web-hosting company also has invested millions in elaborate uninterruptable power supplies (or UPS, backup power sources designed for computers), onsite diesel generators and a brand-new electrical substation. While some Silicon Valley companies make noises about leaving the area, Exodus recently announced plans to build its own onsite plant that it hopes will supply power for several new IDC's.
Although many of California's dilemmas are unique to the state, companies across the country could learn some crucial lessons from Exodus and other Web-hosting companies that are shouldering some of the responsibility for providing energy themselves, experts say. Many Silicon Valley companies, for example, are eagerly investing in various types of distributed generation, which include diesel generators, "co-generation" plants - in which heat produced by electrical generators is recycled for a secondary industrial use such as heating or cooling - and other newer technologies that provide backup power when the local utility can't hack it. Silicon Valley Manufacturing Group member companies have also agreed to cut energy consumption by 10 percent by installing more efficient air-conditioning systems, dimming their lights and taking other conservation measures. "There is no silver bullet to this problem," says Michelle Montague-Bruno, spokeswoman for the Silicon Valley Manufacturing Group in San Jose. "It's a portfolio approach that includes distributed generation, conservation and the upgrading of transmission lines."
Lori DeMatteis, director of global hosting services at San Francisco-based Digital Island, another Web-hosting company, warns that many IT organizations are unprepared for the crisis. "Everyone's so concerned about [computer] security, that they forget about the basics like electricity," DeMatteis says. "It's not as easy as throwing up a generator in the parking lot. You have to go through monthly checks and pay vendor service contracts. You have to be able to flip from commercial to generator power."
Server Farms Take Off
When Exodus was founded in 1994, the current energy woes were far from Silicon Valley's radar screen. Exodus and competitors such as Digital Island and Digex, a WorldCom subsidiary in Beltsville, Md., lease space to house hundreds of servers kept locked in temperature-controlled data centers. The Web hosters sell services that include tight security, swift Internet connections and emergency power systems that guarantee reliability. Exodus, which boasts a network of data centers spread from Silicon Valley to New York City, Boston, Tokyo, Frankfurt, Germany, and London, holds a leading position in this expanding market. The Yankee Group, a Boston-based consultancy, predicts the U.S.-based server farms that took up 9 million square feet in 2000 will grow to as much as 25 million square feet by 2003. Indeed, Exodus plans to build more Internet data centers near its Santa Clara home as well as abroad.
All of that growth depends on electrical power, and a lot of it. Server farms consume copious amounts of electricity - 85 to 100 watts per square foot or more, according to John Roukema, assistant director of Silicon Valley Power, the city of Santa Clara's municipal utility. That's up to three times what the average chip manufacturing plant uses, at 30 to 50 watts per square foot. A standard office building uses 5 watts per square foot, and a typical home uses 1 watt per square foot. In Santa Clara, home to Intel, a Nortel Networks campus and other high-tech giants, 80 percent of new requests for power hookups come from data centers, Roukema says. If all those requests come online, the utility could be adding more than 400 megawatts of electricity, in an area that peaked at 450 megawatts of usage last summer. "This is a new industry, and we're trying to accommodate them as quickly as they would hope," Roukema says.
Moving Beyond the Grid
From the outside, the two-story building looks like dozens of others that line Silicon Valley's miles of high-tech office parks: squat and anonymous. Palm trees sway in a light breeze, and employees in khakis and leather jackets stroll down cement walkways. Inside, however, the atmosphere is more military bunker than corporate campus; bulletproof glass, cameras, security guards and biometric identification devices ensure that those without clearance won't get in.
Welcome to Exodus's SC-4 data center. Several miles from the company's sprawling glass and stainless steel headquarters in Santa Clara, the center's manager, Dario Zuech, proudly shows off the building's extensive battery-powered UPS system and five backup diesel generators. Inside, racks of servers and computer equipment hum as technicians measure performance and respond to customers' concerns. When the rolling blackouts hit the area in January, this center never actually lost power, Zuech says, but plenty of nervous customers called to make sure everything was up and running. "We were laughing because, for us, it was no big deal," says Duncan Scollon, one of Zuech's colleagues, who manages a nearby Exodus data center. "We live with this kind of backup planning every day."
With the likelihood that the current energy squeeze promises to remain a threat for years to come - along with a lack of new power plants to keep up with demand - Exodus executives are not taking any chances. The company is taking its contingency planning several steps further by building an electrical substation and its own power-generating facility. "People are waking up to the fact that power is not an unlimited resource," says K.C. Mares, director of energy and utilities at Exodus. "By building these systems, we can continue to meet our customers' needs at a high level of reliability, and we won't be constrained by the grid."
By building the electrical substation, which was expected to go online in early spring to feed electricity to two Santa Clara data centers, Exodus is answering criticism from some state lawmakers that say Silicon Valley companies, and IDC's in particular, guzzle more than their fair share of power. The substation, which might otherwise have been built by the local utility, effectively upgrades the local grid and allows Exodus to tie in to high voltage lines that improve reliability and load capacity, Mares says. "By taking electricity at a higher voltage, we're adding less impact to the distribution system and allowing other electricity users to gain access," he adds.
Plans for the onsite power-generation facility, which will be built by General Electric's Digital Energy unit, call for the new facility to provide 100 percent of the power for a new cluster of IDC's in Santa Clara. The permitting process for the new plant has yet to start and Mares won't discuss the cost of the project, but he voices confidence that the co-generation plant will allow the company to produce electricity at a competitive cost to what the utility is providing. In addition, the heat resulting from the co-generation process can help run air-conditioning systems crucial to data centers. These efforts now come with an additional price, however: California companies must pay "exit fees" (up to US$6.40 monthly per kilowatt hour) to utilities for generating their own electricity; some state lawmakers are backing an effort to trim or erase these fees.
Exodus isn't alone in its efforts to free itself from dependence on the local utility grid. San Jose-based US DataPort, which builds Internet campuses for use by Web-hosting companies, is planning to build a 188 acre campus in Prince William County, Va., which will include 10 to 20 data center buildings - and a 36 acre power-generating facility. The power plant, which they call a critical reliable energy center (CREC), will be capable of providing 250 megawatts of electricity with natural gas and steam turbine generators. Calpine c-Power, a division of San Jose-based Calpine Corp., which is set to build the power facility, is working on a dozen such onsite power projects for companies that need reliable electrical power. Jeff Byron, director of business development for Calpine c-Power, says companies realize they can't rely on local utilities. "The most important lesson from California's situation is that if businesses want to fix the energy problem, they have to take care of it themselves," he says.
US DataPort's CREC, which will cost about US$300 million to build, would clearly be a stretch for most companies. But many, including some old economy industrial businesses in California, are taking small steps to gain control of their energy flow. ARB Inc., a Lake Forest, Calif.-based construction company that makes oil pipelines, lost power in its Northern California offices for an hour during a January blackout, leaving staff members unable to connect to the company network. Spooked by the experience, Mel Reeves, ARB's CIO, promptly invested US$40,000 in a UPS system for the home office and two backup generators. "Further blackouts would affect our business quite a bit," Reeves says. He adds that project price quotes and bids go out over the Internet and that such crucial information could be lost, along with the business prospects that go with it, in an hour of downtime.
Cisco Shuts Down the Fountain
In response to California's energy crisis, some high-profile technology companies have said they plan to expand elsewhere, where power is less of a factor. Intel CEO Craig Barrett, for example, has been widely quoted as saying the company would build no new plants in California because of the state's unreliable and costly power supply.
But industry experts say that while certain regions are more susceptible to electricity outages, there is no safe haven. "All CIO's and IT organizations plan for disasters and loss of electricity. Now, you can't assume it's a once in 20 year event," says Dennis Fishback, senior vice president and CIO at Calpine.
Exodus and other Silicon Valley companies have made it David Cooper, CIO at the Lawrence Livermore National Lab, says that IT executives can't take power for granted and must focus on backup plans. clear they intend to remain in the area and continue to expand, despite the energy crunch. And many have committed to energy conservation programs as a part of their strategy. Members of the Silicon Valley Manufacturing Group have agreed to expand a program designed to permanently reduce energy use in Silicon Valley by 10 percent over the next two years, says Montague-Bruno. At Cisco Systems, for example, which has backup generators for its data center and network operations, fountains all over the San Jose campus have been shut off and lights have been turned down in hallways and cafés. Lights in all offices are controlled by motion sensors and about 500 energy-efficient motors control air-conditioning across the campus, says Steve Langdon, a company spokesman. According to the EPRI, a 10 percent reduction of energy consumption at peak usage times could reduce energy bills as much as 50 percent.
At Exodus, Mares says the company is working on installing efficient air-conditioning and lighting systems in newer Internet data centers. And while manufacturers such as Sun Microsystems and Hewlett-Packard are designing servers that will eventually run more efficiently, he acknowledges that the current servers are using more and more energy. In order to remain a leader in the competitive Web-hosting market, Exodus knows it has to build and fill the data centers before the rest of the pack catches up. And it knows that in this case, producing power will cost less than not having it.
"By taking the energy needs of our customers into our own hands, we won't be reliant on utilities or government regulations," Mares says. "By generating power, we believe we can help to improve some of the grid constraints."
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.