Data centers go underground in bunker mentality can be helpful for security, backup and business continuity
As Hurricane Ike bore down on Houston one Friday last September, the Continental Airlines' flight operations center, located on the 14th floor of a glass-sided downtown high rise, suddenly went dark. For the airline's pilots and flight crews, however, business proceeded as usual.
Here's why: At that same moment, 42 miles north of the city and some 60 feet underground -- in a hardened Cold-War era bunker built by a paranoid millionaire oilman to survive a nuclear holocaust -- Continental's backup data center took over. Throughout the ordeal -- from Friday morning, as the storm approached, through Saturday, when winds above the Westland Bunker in Montgomery, Texas, gusted to 125 miles per hour, until Sunday evening, when operations resumed in Houston -- the airline managed an 89% on time rating for its global flight schedule.
Locating a backup data center in an underground bunker may seem like overkill, even in a hurricane zone. But the facility met all of the airline's requirements -- including cost, says John Stelly, managing director of technology at Continental. The bunker, run by real estate partnership Montgomery Westland, has been converted into 33,000 square feet of rack-ready data center space complete with air conditioning, redundant network and power sources, uninterruptible power supply systems and backup generators.
Continental leases 2,000 sq. feet underground and another 12,500 sq. feet of office space above ground, in a hardened building complete with 3-inch-thick bulletproof windows. The airline can house its entire operations staff of up to 125 people at the backup site.
After Hurricane Katrina, Continental began looking for a fallback data center for use during hurricanes. Westland "was far enough away to be out of harm's way but close enough for folks to drive to," Stelly says. The blast-resistant facility is admittedly a bit much for even Continental's backup needs, but the four-feet-thick walls and high security entrance are nice extras, Stelly says.
Also, connectivity options at the Westland facility were a plus. The network and power feeds for the bunker were sourced from areas well away from Houston, while pricing was competitive with above-ground co-location facilities.
Rise of the underground
With a renewed focus on data center outsourcing and space in high availability facilities in short supply, investors such as Montgomery Westland have snapped up and renovated abandoned mines and military bunkers in the hopes of cashing in.
Since 2007, for example, Cavern Technologies has operated a data center 125 feet below ground in an abandoned limestone mine. The mined out area underground, which covers 3 million sq. feet, is 15 minutes outside of Kansas City, Mo. Unlike other mines, the Cavern facility was created with the idea of reuse in mind, so floor space isn't irregularly shaped like other underground facilities can be, says president John Clune. The area's relatively low electricity costs, at 3.5 cents per kilowatt-hour, help to make operating costs lower than those in other parts of the country, he adds.
Another facility, The Bunker, is a decommissioned, 50,000 sq. foot Royal Air Force bunker that operated until the 1990s. The facility is inside a hill near Dover, England and it now hosts data centers 100 feet below ground. "People get a picture of a hole in the ground. That's not the case. It's a state-of-the-art data center," says Paul Lightfoot, director of managed services for The Bunker. Clients range from businesses running mission-critical Web applications to a financial services firm that runs online trading systems. "We do everything from basic square footage to fully maintained systems," he says.
Iron Mountain is among the oldest and best known providers of underground storage and data center space. Known for storing everything from backup tapes to old movie reels in The Underground, its repurposed limestone mine in rural Pennsylvania, the company has seen its electronic storage and leased data center space business increase while its traditional paper record storage business has slowed. "It is now the fastest growing component of our business," says vice president Charles Doughty. In addition to leasing rack-ready space, the company offers data center design, hosting and management services.
But while interest is up, the number of actual customers leasing space in its underground data centers remains small. Iron Mountain counts five operating data centers in its underground facility, including its own. But with 60,000 square feet of available data center space and another 145 acres undeveloped in the facility, Iron Mountain has plenty of room for more.
Underground data center facilities fall into two categories: Abandoned mines, like Iron Mountain's, and decommissioned military bunkers such as InfoBunker, a subterranean facility just outside of Des Moines, Iowa. InfoBunker leases data center space to organizations ranging from a local telephone company to government agencies about which Jeff Daniels, a vice president at the company, says he can't talk.
On the demand side, an increase in extreme weather events, heightened concerns about security since 9/11 and the need to provide higher levels of security to comply with regulatory requirements have made these spaces more attractive to some organizations. Underground facilities offer security and structural protections that would be cost prohibitive to build from scratch.
Meanwhile, the recession and credit crunch have made it harder to get funding to build new data centers, and organizations have become more accepting of the idea of using co-location facilities to house mission-critical data center operations. "Demand for computer space is stronger than I've ever seen it [and] the supply is so small, so inadequate," says Peter Gross, vice president and general manager at HP Critical Facilities.
Basic co-location space isn't the problem, says Jon Bolen, chief technology officer at Westec Intelligent Surveillance. The surveillance-monitoring service, based in Des Moines Iowa, serves clients such as McDonald's Corp. and Zales Corp., and recently completed a search for a hardened facility for its own backup data center. During this search, Bolen saw a general lack of high-end infrastructure, of enterprise-class data center space. "If you need space that is as good or better than the space you would build there's a shortage of places you can go."
This shortage has given the underground facilities an opening to pull in larger data center clients. Cavern says it is negotiating with Fortune 500 clients looking to lease spaces of 30,000 to 100,000 square feet. However, most clients are smaller organizations that don't require so much space; more typical would be a hospital that leases 1,500 sq. feet.
One might assume that IT organizations would have to pay a premium for bunker space. After all, the cost of building such a structure is high, and special venting and air-flow systems are required. But IT executives say they've driven deals where the total cost of ownership is competitive with above-ground facilities. Because they're repurposing existing space that the government or a mine operator paid to build, providers say they don't have to pass on the original construction costs for the structures and can afford to be cost competitive.
Before deciding to go underground, IT executives need to identify potential limitations, experts say. Even things as simple as ceiling height can be a challenge. Continental's data center space in the Westland bunker has 10-foot ceilings, and putting full-height racks on top of an 18-inch raised floor was a tight fit. "We had to come up with a design to allow us to use full-height racks while providing sufficient airflow," Stelly says.
Another concern: While computer systems may be protected in a bunker, critical infrastructure needed during a disaster, such as generators, fuel tanks and air conditioning cooling towers, may be above ground. That could be a problem if the catastrophe you need to worry about is a tornado, warns Westec's Bolen.
Bolen recounts how one company claimed that its hardened facility could withstand a direct hit from an F3 (158 to 206 mph) tornado. But the air conditioning and generators were outside. "When an F3 hits, those generator and HVAC units are going to come off their pads," he says.
Westec ended up taking space at InfoBunker, about 45 miles away from its offices, Bolen says. The 65,000 square foot Cold War command bunker, designed to withstand a 20-megaton nuclear blast, maintains all infrastructure, including generators, fuel and cooling equipment, 50 feet underground.
Another consideration is that these underground facilities tend to be in rural, out-of-the-way locations. The facilities may be too far away from a company's primary data center, and finding local lodging for staff in a disaster situation may be difficult. Continental had to find office space and lodging accommodations for more than 100 operations staff during Hurricane Ike. Fortunately, Montgomery Westland had hardened above-ground office space as well as access to local lodging.
Underground facilities do have a few other advantages. The limestone floors at The Underground have a virtually unlimited load rating, while the walls maintain a constant temperature of about 55 degrees and act like a heat sink for some of the waste heat that comes off data center equipment. The limestone walls absorb 1.5 BTUs per hour per sq. foot of wall space, Doughty says.
Cool stuff
The green aspect of going underground is what attracted Marriott International. It wanted to move from an outsourced "cold site" disaster recovery service to managing its own hot site backup data center. Management wanted a hardened, secure facility in a location that was within a day's drive from Marriott's Bethesda, Md., headquarters. And it wanted to make sure the facility followed the company's focus on environmentally friendly best practices, according to Dan Blanchard, Marriott's vice president of enterprise operations.
Last year, the hospitality business completed the build-out of a 9,000 sq. foot remote backup data center at The Underground. Blanchard says that although the extreme level of security, including armed guards, exceeded his requirements, the idea of reusing an old mine rather than breaking new ground appealed to Marriott. "It's a definition of recycling to use the space that was a mine and convert that fairly inexpensively to its next use, which for us is a data center."
Energy efficiency also factored into Marriott's decision, Blanchard says. While Marriott's data center uses a traditional chiller as its primary cooling system, the backup is a prototype free cooling system. That prototype, designed by Iron Mountain, uses an air-to-air heat exchanger, drawing 55-degree air from the 1,000 acres of unused space within the mine. "The air is the exact temperature of what you would bring in with mechanical cooling," Doughty says. Iron Mountain also is experimenting with a system that would pull cool water from an underground lake within the mine.
An abandoned mine may conjure up images of damp walls and dripping ceilings -- but that's not the case here. "You have pumps and a lot of protective devices," says HP's Gross, and all of the facilities claim that dampness is not a problem. Doughty says The Underground is naturally dry due to its location and the type of limestone above the mine.
Air quality also is good, he says. The air in the Iron Mountain facility is relatively clean and non-condensing, he says. "As soon as you put heat to it moves away from the dew point," and that makes it a good choice for cooling, he says.
Blanchard says the new Recovery and Development Center, which is used for software development until needed in an emergency, costs half as much as he previously spent on power. Some of that is attributable to relatively low cost of power in Pennsylvania (5.5 cents per kWh). The rest comes from efficiencies of design and the characteristics of the underground environment.
Gross cautions, however, that cooling efficiency gains specific to the location are probably not all that significant. A well-designed data center today can cut power consumption in half by using new energy efficient equipment that can run at higher operating temperatures, by optimizing airflow designs to allow intake air temperatures to rise as high as 85 degrees and still keep equipment within operating temperature limits, and by picking a location in a colder climate, where water- or air-side economizers can be used to take advantage of cool outside air as weather permits.
Security, Gross says, is the primary benefit of using an underground facility to host a primary or secondary data center. But for most of his clients, the ability to get people to the backup data center in a hurry, connectivity options, and finding a facility that meets budget are priorities. Underground facilities usually don't beat out above-ground sites in his clients' evaluations, he says.
Still, Continental and Marriott are among a small number of enterprise operations using underground facilities. Rakesh Kumar, an analyst with Gartner Inc., says he is unaware of any Gartner client that is currently leasing space in one. The primary benefit of such sites, he says, is that they are designed to be highly resilient -- often to military specifications. That's important for some government data centers. "But for most commercial enterprises, it probably will not be such a major requirement." IT executives considering underground data center space should check into expansion capability, energy efficiency and how electricity use is metered, he says.
The main entrance to InfoBunker's underground facility is a four-inch-thick, 4.5-ton main blast lock. It serves both as a secured entrance and as a very heavy-duty person-trap. Visitors must pass through two doors that are interlocked. One will not open before the other is closed and sealed.
The operations center at InfoBunker's underground facility. The facility has 65,000 square feet of data center-ready space. 
Main entrance at The Underground, a 145-acre underground facility operated by Iron Mountain in Butler County, Penn. Visitors are accompanied by armed escorts.
Workers in Iron Mountain's control room at The Underground monitor systems operations as well as external factors like the weather.
The Underground hosts five data centers, including Iron Mountain's own primary data center, shown here. It backs up data for more than 400,000 computers and 20,000 servers, and archives approximately 10 billion e-mail messages.
During Hurricane Ike last year, Continental Airlines evacuated more than 100 operations workers to this hardened office building, adjacent to the underground Westland Bunker, where data center operations continued uninterrupted 60 feet below ground. The office building has 3-inch bulletproof glass on the first floor and reinforced walls. 
Suites like this one at the Westland Bunker house customer data center equipment within the hardened, underground facility. The bunker is home to 13 primary and backup data centers. The Tier 3 suite shown is 4,800 square feet.
The Westland Bunker offers caged suites and rack space, shown here. The racks in this 12,000-square-foot room, located some 55 feet below ground, house servers and networking equipment owned by many companies.
Main entrance to The Cavern, operated by Cavern Technologies in Lenexa, Kansas. Originally a limestone mine, the facility now leases data center space to 26 customers.
A secure access corridor within The Bunker, a former Royal Air Force bunker in Dover, U.K.
The main entrance at The Bunker, located on a hilltop in Dover, U.K., doesn't look like much. But 100 feet below ground it hosts data center equipment for more than 50 customers.
One
of the data halls at Cavern Technologies in Kansas, which offers few
clues that the facility is 75 feet below ground. (Photo: Cavern
Technologies)
Behind The Bunker
