USSHC Ultimate Underground Data Bunker

USSHC was founded in 2002. However, the process of creating the ultimate underground data bunker actually started in 1999 with the needs of an Internet Service Provider. It was found that truly geographic IP transit could not be fulfilled by the local telephone carriers due to telephone tariffs. The larger Tier 1 Network Service Providers could not provide higher reliability because they depend on the local carriers to deliver the local access to their network. No matter how services were delivered via these traditional means, there was always a single point of failure.

There are solutions that use fiber rings to deliver services, but even fiber rings have single points of failure: The telephone exchange central offices.

The only way to have true IP redundancy is to have different connections from multiple IP transit providers from opposite directions, that use different local fiber networks, from entirely different companies. Most data centers try to meet these needs using fiber paths from carriers not bound by telephone tariffs. However, this “solution” was accompanied by a host of other problems.

In terms of physical security, every data center investigated shared a common building with other tenants. Either the data center was an afterthought and added to an existing building (server closet that grew in to dedicated space), or it was purpose built but with much office space and other common space in the same facility. Both of these types of shared structures increase the risk of collateral damage due to fires in the same building, and to security risks due to the large numbers of people sharing the facility. The best data center fire suppression system in the world doesn’t do a bit of good if the office above it building burns down on top of the data center.

Major shortcomings in physical security were also a recurring theme during the search for a data center. Many facilities share common space with other businesses. Despite partitioning off a building, the common mechanical facilities such as chiller plant, and electrical, are typically shared with other tenants. Obtaining building wide security is difficult not only due to the different tenants sharing common areas, but due to reception areas in buildings that were open to the public and entirely unsecured. Most “secure” server spaces were found to be secured from public areas by walls made of sheet rock! Some sheet rock walls contained windows! We desired something a little more secure than two layers of half inch thick sheet rock, or a single pane of glass.

Despite finding several facilities that all claimed to be “hardened” and able to withstand the force of a tornado with walls made of reinforced concrete, and at least one door made of steel with no windows to the outside world, further investigation revealed that at most, they were only partially below ground, (walk out basement) and all lacked physical plant equipment that was designed to operate during major contingencies. They also shared office space in the same building. Time and time again, it was found that 100% of the data centers had their heat rejection and standby power systems above ground. And in no case were the generators or air conditioning systems “hardened” at all. While the servers may survive if the data center took a direct hit from even a small EF-1 tornado, they would not remain operational for any length of time once the uninterruptible power supply batteries were exhausted. Even if the connectivity and building itself survived, and a generator was tough enough to operate after a storm or tornado, the external cooling would not. Even with power, the servers would quickly overheat, leading to downtime, and possible data corruption or loss.

Some data centers that claimed to be “hardened” were found to require a constant feed of municipal water for on site cooling. With all of the redundancy built in to the site, the whole data center could fail due to a non redundant source of cooling water that could be interrupted due to a pipe break, power outage, earthquake, or even simple maintenance. Or the whole data center could fail due to a water pipe break that would flood the facility with a high pressure torrent of municipal water.

Then there were the data centers located in flood plains. We were shocked at just how many data centers were located in flood plains. More alarming was the “head in the clouds” attitude that most had about the flood plain being entirely acceptable because the data center was on an upper floor.

The harder we looked, and the more we uncovered, the more discouraged we became. Eventually however, USSHC solved all of these problems, and then some.

The idea behind USSHC was to provide a safe, secure place to house an Internet Service Provider that would be immune from any form of disaster, “deep in an Iowa underground bunker” where the power would always stay on, and the servers would always stay connected, fully online, and fully operational, despite what was going on in the outside world.

Since it went live in 2002, the facility has been expanded to allow other companies to share the same level of redundancy, security, and performance.

In 2009, USSHC opened the GWAB (Geek with a box data suite) to offer an economical alternative to our premium data center colocation offerings.

Innovations for the Data Center

6 Cool Innovations for
The Data Center


1. Fiber optics with a twist

The success of the HDMI cable in consumer electronics has proved that having a common cable that works with Blu-ray players, HDTV sets and just about any set-top box helps remove clutter and confusion. Intel has developed Light Peak following this same approach. It's fiber-optic cable that will first be used with laptop and desktop computers to reduce clutter and to speed transmission, but it could also make its way to the data center as a way to connect servers and switches.

The 3.2mm cable, which is about as thin as a USB cable, can be up to 100 feet long. Intel has designed a controller that will sit inside a computer, and cables are currently in production. Third parties, including Hewlett-Packard and Dell, will start making computers with Light Peak fiber-optic cables in them by 2011, according to Intel.

For data centers, Light Peak presents a few interesting possibilities. Fiber optics have been in the data center since the early 1990s, when IBM introduced its Escon (Enterprise Systems Connection) product line; it connects mainframes at 200Mbit/sec. Light Peak differs in that it runs at 10GB/sec., and Intel claims that the components will be less expensive and lighter-weight than existing fiber-optic products.

"Intel claims Light Peak will be less complex and easier to manage by eliminating unnecessary ports, and deliver the higher throughput required by high performance e-SATA and DisplayPort systems," says Charles King, an analyst at Pund-IT in Concord, Mass. "If the company delivers on these promises, Light Peak could simplify life for data center managers plagued by installing, managing and troubleshooting miles of unruly optical cables."

Success here will depend on "how willingly developers and vendors" embrace Light Peak and build products around it, King explains.

2. Submerged liquid cooling and horizontal racks

Liquid cooling for data centers is not a new concept, of course, but Green Revolution Cooling has added a new twist. For starters, the rack is turned on its side, which helps with cable management and makes it easier for administrators to access equipment, and the horizontal rack is surrounded by liquid. A new coolant, called GreenDEF, is made from mineral oil that is nontoxic, costs less than other liquid-cooling methods and is not electrically conductive like water, according to a GR Cooling spokesman.

"The liquid actually moves through the floor and circulates up through all of the computing nodes," says Tommy Minyard, director of advanced computing systems at the Texas Advanced Computing Center, part of the University of Texas at Austin. This means more-effective cooling because heat is moved away from the processors via cables on the sides and under the rack, he explains. Minyard is installing GR Cooling systems in his data center and expects a 30% to 40% savings compared to traditional air-cooled systems.

Data center cooling device

Green Revolution uses horizontal devices for racks, along with a new type of coolant, to reduce energy costs in a data center.

Minyard says liquid cooling has made a rebound lately, recalling the days when Cray offered submerged cooling systems, and he notes that even IBM is moving back into chilled-liquid-cooling some compute nodes.

Pund-IT's King says a major issue is that enterprises have fought the return of liquid cooling in the data center because of the high costs of implementing the technology and because it is unproven as a widespread option.

"Liquid cooling usually costs much more to install upfront than air cooling," says Mark Tlapak, GR Cooling's co-founder. "Compared to air, every liquid cooling system has some added nuance, such as electric conductivity with water-based cooling systems. " But, he says, "spring a leak in the water systems, and you lose electrical equipment." Still, for Minyard, GR Cooling is an ideal fit: His data center gravitates toward dense, powerful systems that pack intense power into small spaces, such as IBM blade servers and the latest Intel processors. The Ranger supercomputer, for example, uses 30kw of power per rack.

3. Several broadband lines combined into one

Enterprises can spend many thousands of dollars on fiber-optic lines and multiple T1 connections, but at least one emerging technology is aiming to provide a lower-cost alternative.

Mushroom Networks' Truffle Broadband Bonding Network Appliance creates one fast connection out of up to six separate lines, a technique known as bonding. The Truffle combines the bandwidth of all available broadband lines into one giant pipe, with download speeds of up to 50Mbit/sec., the company says. Internet access may be through a DSL modem, cable modem, T1 line or just about any broadband connection.

This helps increase overall throughput, and acts as a backup mechanism, too. If one of the "bonded" lines fails, the Truffle connection just keeps running with the other available lines.

Steve Finn, a television producer in Kenya, uses Mushroom Networks' appliance for a program called Africa Challenge that is broadcast to eight African countries. He relies heavily on broadband to produce the content and at one time paid as much as $4,000 per month for connectivity. Speeds vary depending on latency and traffic, but he says the bonded speed is generally about four times faster (four lines times the speed of each individual line), at about half the cost of one equivalent high-speed line.

Frank J. Bernhard, an analyst at Omni Consulting Group, says Mushroom Networks fills a niche for companies that do not want to pay the exorbitant fees for multiple T1 or T3 connections but still need reliable and fast Internet access. Other companies, including Cisco Systems, offer similar bonding technology, but at a greater cost and with more complexity at install, which means the technique has not yet been widely used.

4. Multiple data centers more easily connected

In a very large enterprise, the process of connecting multiple data centers can be a bit mind-boggling. There are security concerns, Ethernet transport issues, operational problems related to maintaining the fastest speed between switches at branch sites, and new disaster planning considerations due to IT operations running in multiple locations.

Cisco's new Overlay Transport Virtualization, or OTV, connects multiple data centers in a way that seems really easy compared with the roll-your-own process most shops have traditionally used. Essentially a transport technology for Layer 2 networking, the software updates network switches, including the Cisco Nexus 7000, to connect data centers in different geographic locations.

The OSV software costs about $25,000 per license and uses the maximum bandwidth and connections already established between data centers.

There are other approaches for linking multiple data centers, a Cisco technical spokesman acknowledges, including those involving Multiprotocol Label Switching (MPLS) or, before that, frame-relay and Asynchronous Transfer Mode protocols.

But unlike some of the older approaches, the spokesman explains, Cisco OTV does not require any network redesign or special services in the core, such as label switching. OTV is simply overlaid onto the existing network, inheriting all the benefits of a well-designed IP network while maintaining the independence of the Layer 2 data centers being interconnected.

Terremark, a cloud service provider based in Miami, uses Cisco OTV to link 13 data centers in the U.S., Europe and Latin America. The company says there is a significant savings compared with taking a "do-it-yourself" approach to linking data centers, due to reduced complexity and OTV's automated fail-over system that helps multiple data centers act as one if disaster strikes.

"Implementing the ability to balance loads and/or enact emergency fail-over operations between data centers traditionally involved a dedicated network and complex software," says Norm Laudermilch, Terremark's senior vice president of infrastructure. "With Cisco OTV, Ethernet traffic from one physical location is simply encapsulated and tunneled to another location to create one logical data center."

Virtual machines from one location can now use VMware's VMotion, for instance, to automatically move to another physical location in the event of a failure.

5. Priority-based e-mail storage

Communication is what drives a business, but too often the bits and bytes of an e-mail exchange are treated in the data center as just another data set that needs to be archived. Messagemind automatically determines which e-mails can be safely archived onlower-cost systems.

The tool analyzes all company communication -- tracking which messages end users read, delete or save -- and then groups them according to priority level.

Data center administrators can use that information to store e-mail based on priority level, which in turn can save money. For example, instead of storing all e-mails in one high-cost archive, messages marked as low priority -- based again on the end user's clicking behavior -- can be stored in lower-cost storage systems. High-priority e-mail can be stored on higher-performance, and higher-cost, media.

That same behind-the-scenes analysis can be used outside the data center, rolled up into a dashboard that managers and end users can view to help them on projects. For example, business units can view e-mail diagrams that show who is communicating effectively on a project and who seems to be lagging behind and rarely contributing.

Pund-IT's King says Messagemind is an intriguing prospect because e-mail has become such a wasteland of broken conversations and disconnected project discussions. Managing e-mail becomes even more painful if a company is subject to litigation, and e-mail becomes part of the legal discovery process.

"Even the best e-mail solutions require employees to manage their messages," says King. "If it works as advertised, I could see this catching hold in enterprises. By managing e-mail more effectively -- and automatically -- Messagemind's solution could take a great deal of weight off the shoulders of data center admins struggling under ever-increasing volumes of stored messages."

6. User accounts virtualized for easier restoration

Virtualization has become the buzzword of the past decade, but it usually involves abstracting an operating system from a server or data from your storage allocations. AppSense is virtualization software for user accounts. It extracts user profile settings from Windows applications and maintains them separately. That means that if an application is updated or changed, the user information is still available. If user settings are corrupted or lost, administrators can restore the settings with a minimum of bother.

Landon Winburn, a software systems specialist at the University of Texas Medical Branch in Galveston, Texas, uses AppSense to virtualize user-account profiles for his 3,000 students. Winburn says the university used to manage user settings manually, taking about 40 to 60 calls per week related to log-ins. The university also had five to 10 corruptions per day related to user settings.

"Before AppSense, the only solution for a corrupt profile was to delete the profile and have the user start again from scratch for all applications," says Winburn.

But now, with AppSense's ability to restore these settings, the university doesn't have to directly address the problems, since they are handled automatically. By virtualizing accounts, the university could also increase the number of XenApp Server accounts from 40 user profiles per server to about 80.

John Brandon is a veteran of the computing industry, having worked as an IT manager for 10 years and a tech journalist for another 10. He has written more than 2,500 feature articles and is a regular contributor to Computerworld.

Data Centers to Expand in 2010, 2011: Campos Research & Analysis said's demand will be high for data centers

Data Centers to Expand in 2010, 2011

Rich Miller

More than a third of large corporate data center users in North America plan to expand their footprint in 2010, and many are expanding because they have run out of power, not space. Those were the key findings in survey data released Wednesday by Digital Realty Trust. The survey of senior decision makers with responsibility for their companies’ data center strategies was conducted by Campos Research & Analysis for Digital Realty. Among the key findings:

• 83 percent of respondents are planning data center expansions in the next 12 to 24 months;
• 36 percent of respondents have definite plans to make those expansions during 2010;
• 73 percent of respondents plan to add two or more facilities as part of their data center expansions;

It’s not surprising that Digital Realty believes demand will be high, since the company is in the business of building and leasing data centers. But the customer survey’s major points were echoed by multiple panelists at Wednesday’s New York event.

Financing is a Factor
“Demand has been pretty steady,” said Dan Golding, Managing Director at DH Capital, an investment bank specialized in hosting and telecom deals. “The story has really been supply. It’s been very, very difficult for people to finance new data centers.”

At the national level, the pending demand for data center space may be three times greater than the available supply of quality space, according to Jim Kerrigan, the director of the data center practice at the real estate firm Grubb & Ellis. “All those deals that got shelved in 2009 because the CFO said no .. they’re going to happen,” said Kerrigan.

The end users at DataCenterDynamics New York included large firms in the financial sector, who concurred with the notion that cost-cutting has resulted in pent-up demand for data center space. “A year and a half ago we were talking about new data centers,” said Glenn Neville, Director of Engineering at Deutsche Bank. “Since then we’ve been talking about how long we can go with our current data centers. Our plans for growth are still there. Those plans are being postponed, but they’re not being cancelled.”

“It feels like someone closed a door, and things are backing up behind it,” said David Schirmacher, a vice president at Goldman Sachs.

Big Chunks of Space Grow Scarce
Kerrigan said the supply and demand challenges will be most acute for companies needing large footprints of contiguous space. That imbalance stands in stark relief to the requirements described in the Digital Realty survey, in which 70 percent of companies planning data center expansions say they envision large projects of at least 15,000 square feet in size or 2 megwatts or more of power.

“One of the most interesting pieces of data in this study is the lead role that power is now playing in these expansions,” said Chris Crosby, Senior Vice President of Corporate Development for Digital Realty Trust. “The need for additional power has become the main driver for data center expansion plans as companies seek facilities with adequate power and favorable utility rates to control operating costs.”

As a result, more companies are tracking their data center power usage and using the data in their capacity planning. The survey found that 76 percent of respondents now meter their power use, while the number of companies that meter power down to the PDU level increased by 29 percent over last year. “These are very positive signs that companies better understand their data centers’ energy use and can make informed decisions to reduce energy consumption,” said Crosby.

Digital Realty Trust: Data Centers to Expand in 2010, 2011 Written by Jeffrey Clark

Tuesday, 09 March 2010

A 2010 survey conducted by Campos Research & Analysis on behalf of Digital Realty Trust indicates that a significant portion of large North American companies are planning to expand their data center infrastructure in 2010 and 2011. This survey, conducted in mid-January, queried high-level company employees (executives or upper-level managers in information technology or finance) from 300 large companies. The participant companies were required to have a minimum of 5,000 employees or a minimum annual revenue of a billion dollars, and the individual respondents were required to be in charge of managing some aspect of the company’s data centers, whether operation, expansion, or implementation. The companies represented in the survey consist of a fairly wide cross section of industry, with slightly over a quarter dedicated to IT, the Internet, or telecommunications. About 15% of the companies were in finance, with the remainder in the “other” category. According to the survey, the companies average about four data centers each, with nearly 20% operating six or more data centers. Of the companies represented in the survey, 22% built or acquired a new data center in the 12 months preceding the survey, and 63% did so between one and three years prior to the survey. With ever-increasing demand for data services despite the recent economic downturn and with most companies having built or acquired additional data center facilities over a year prior to the survey, a significant increase in data center facilities in 2010 and 2011 seems to make sense, especially as companies hope for continued economic recovery. For 2010, 36% of responding companies indicated that they were “definitely” planning data center expansions; 46% responded that they would “probably” expand their data centers, leaving only 18% that were unlikely to initiate an expansion this year. The numbers for planned data center expansions in 2011 were virtually identical, with a total of 84% either “probably” or “definitely” planning an expansion. Most of the companies (63%) with definite plans to expand have only one or two locations slated for expansion; 14% indicated plans to expand four or more data center locations. The survey also indicated that data center budgets are on the increase, with 75% of the respondents expecting some increase; 35% expect an increase of less than 10%, and 30% expect an increase between 10% and 20%. According to the survey report, the average budget increase for 2010 is forecast at 8.3%—a change of +1.7% over the 2009 budget increase. IT budgets followed a similar trend, with an average expected increase of 8.1% and with about 73% expecting some level of budget growth. On average, the survey report indicated that 35% of represented companies’ IT budgets are dedicated to data center operations and development. At the top of the list of preferred locations for new or expanded data centers are several American metro areas, led by New York City, which is closely followed by Chicago, Los Angeles, and Dallas. Foreign locations that ranked high on the survey’s list included London, Singapore, Paris, and Tokyo. The survey also inquired about the respondents’ reasoning for their expected data center expansion in 2010; those that indicated on the survey definite plans for expansion rated the relative importance of various reasons for this expansion. The leading reason was power capacity, with 74% of the respondents indicating that this reason was “extremely important.” Next was disaster recovery and Sarbanes-Oxley (compliance, presumably) at 72%, followed by security at 69% responding with “extremely important.” Other possible reasons cited in the survey include energy efficiency, consolidation, cooling, redundancy, potential regulations, environmental concerns, and need for additional space. A potential indicator of the scope of the expected 2010 data center expansions is the expected change in power usage. According to the survey report, the average expected power usage increase among companies that have definite plans for expansion is 12.8%, with 40% planning between a 10% and 20% increase. For all companies included in the survey, the average expected power usage increase is 8.3%. Overall, the survey commissioned by Digital Realty Trust indicates no slackening in large companies’ increasing need for data center services. The expected expansions over the next two years are likely predicated on whether the economy begins to recover or whether the United States (and, to a lesser extent, the rest of the world) is in for a so-called double-dip recession.