Underground Data Centers New Trends: IBM Generator Failure Causes Airline Chaos

Saturday, October 17, 2009

IBM Generator Failure Causes Airline Chaos

October 12th, 2009 : Rich Miller

A generator failure Sunday at an IBM data center in Auckland, New Zealand crippled key services for Air New Zealand, prompting the airline’s CEO to publicly chastise Big Blue for the failure. The data center outage crashed airport check-in systems, as well as on-line bookings and call center systems Sunday morning, affecting more than 10,000 passengers and throwing airports into disarray.

The problem occurred during planned maintenance at IBM’s Newton data center in Auckland. A generator failed during the maintenance window, dropping power to parts of the data center, including the mainframe operations supporting Air New Zealand’s ticketing. IBM says service was restored to most clients within an hour, but local media reports say Air New Zealand’s ticketing kiosks were offline for up to six hours.

Air New Zealand chief executive Rob Fyfe is not happy.

“In my 30-year working career, I am struggling to recall a time where I have seen a supplier so slow to react to a catastrophic system failure such as this and so unwilling to accept responsibility and apologise to its client and its client’s customers,” Fyfe wrote in an email to IBM, which then became public.

“We were left high and dry and this is simply unacceptable,” Fyfe added. “My expectations of IBM were far higher than the amateur results that were delivered yesterday, and I have been left with no option but to ask the IT team to review the full range of options available to us to ensure we have an IT supplier whom we have confidence in and one who understands and is fully committed to our business and the needs of our customers.”

No comments:

Post a Comment

Comparison chart: Underground data centers

Five facilities compared

Robert L. Mitchell

Name	InfoBunker, LLC	The Bunker	Montgomery Westland	Cavern Technologies	Iron Mountain The Underground
Location	Des Moines, Iowa*	Dover, UK	Montgomery, Tex.	Lenexa, Kan.	Butler County, Penn.*
In business since	2006	1999	2007	2007	Opened by National Storage in 1954. Acquired by Iron Mountain 1998.
Security /access control	Biometric; keypad; pan, tilt and zoom cameras; door event and camera logging	CCTV, dogs, guards, fence	Gated, with access control card, biometrics and a 24x7 security guard	Security guard, biometric scan, smart card access and motion detection alarms	24-hour armed guards, visitor escorts, magnetometer, x-ray scanner, closed-circuit television, badge access and other physical and electronic measures for securing the mine's perimeter and vaults
Distance underground (feet)	50	100	60	125	220
Ceiling height in data center space (feet)	16	12 to 50	10	16 to 18	15 (10 feet from raised floor to dropped ceiling)
Original use	Military communications bunker	Royal Air Force military bunker	Private bunker designed to survive a nuclear attack. Complex built in 1982 by Louis Kung (Nephew of Madam Chang Kai Shek) as a residence and headquarters for his oil company, including a secret, 40,000 square foot nuclear fallout shelter. The office building uses bulletproof glass on the first floor and reception area and 3-inch concrete walls with fold-down steel gun ports to protect the bunker 60 feet below.	Limestone mine originally developed by an asphalt company that used the materials in road pavement	Limestone mine
Total data center space (square feet)	34,000	50,000	28,000 plus 90,000 of office space in a hardened, above-ground building.	40,000	60,000
Total space in facility	65,000	60,000	28,000	3 million	145 acres developed; 1,000 acres total
Data center clients include	Insurance company, telephone company, teaching hospital, financial services, e-commerce, security monitoring/surveillance, veterinary, county government	Banking, mission critical Web applications, online trading	NASA/T-Systems, Aker Solutions, Continental Airlines, Houston Chronicle, Express Jet	Healthcare, insurance, universities, technology, manufacturing, professional services	Marriott International Inc., Iron Mountain, three U.S. government agencies
Number of hosted primary or backup data centers	2	50+	13	26	5
Services offered	Leased data center space, disaster recovery space, wholesale bandwidth	Fully managed platforms, partly managed platforms, co-location	Disaster recovery/business continuity, co-location and managed services	Data center space leasing, design, construction and management	Data center leasing, design, construction and maintenance services
Distance from nearest large city	Des Moines, about 45 miles*	Canterbury, 10 miles; London, 60 miles	Houston, 40 miles	Kansas City, 15 miles	Pittsburgh, 55 miles
Location of cooling system, includng cooling towers	Underground	Underground	Above and below ground. All cooling towers above ground in secure facility.	Air cooled systems located underground. Cooling towers located outside	Chillers located above ground to take advantage of "free cooling." Pumps located underground.
Location of generators and fuel tanks	Underground	Above ground and below ground	Two below ground, four above ground. All fuel tanks buried topside.	Underground	Underground

*Declined to cite exact location/disatance for security reasons.

Glossary of Data Center Terms

AC - Alternating current

ACAE - Air conditioning airflow efficiency, the amount of heat removed per standard cubic foot of airflow per minute

AHU - Air handling unit

Air Mixing - The unintended mixing of cold and hot air

Airside Economizer - An economizer that directs exterior air into the data center when the air temperature is at or below the cooling set point

Aisle - The open space between rows of racks. Best-practice dictates racks should be arranged with consistent orientation of front and back to create ‘cold'and ‘hot'aisles.

AMS - Asset management system

ASHRAE - American Society of Heating, Refrigerating and Air-Conditioning Engineers is an international technical society organized to advance the arts and sciences of air management.

BACnet - A data communication protocol for building automation and control networks
BAS - Building automation system
Blanking Panel - A device mounted in unused U spaces in a rack that restricts bypass airflow, also called blanking or filler plates
BMS - Building management system, synonymous with BAS, AMS and other computer-based tools used to manage data center assets
BTU - British thermal until, a standard measure of cooling equipment capacity
Bypass Airflow - Conditioned air that does not reach computer equipment, escaping through cable cut-outs, holes under cabinets, misplaced perforated tiles or holes in the computer room perimeter walls.
C - Degrees Celsius

C/H - Cooling/Heating Cabinet - Device for holding IT equipment, also called a rack
CAC - Cold aisle containment system that directs cooled air from air conditioning equipment to the inlet side of racks in a highly efficient manner
CADE - Corporate average data center efficiency
CapEx - Capital expense, the cost of purchasing capital equipment
Carbon Footprint - A measurement of the volume in pounds of Carbon Dioxide generated by business operations. CFD - Computational fluid dynamics, scientific calculations applied to airflow analysis
CFM - Cubic feet per minute, an airflow volume measurement Close-Coupled Cooling - Cooling technology that is installed adjacent to server racks and enclosed to direct airflow directly to the rack without mixing with DC air
Coefficient of Effectiveness (CoE) - Uptime Institute metric based on the Nash-Sutcliffe model efficiency coefficient Cold Aisle - An aisle where rack fronts face into the aisle. Chilled airflow is directed into this aisle so that it can then enter the fronts of the racks in a highly efficient manner.
Cold Spot - An area where ambient air temperature is below acceptable levels. Typically caused by cooling equipment capacity exceeding heat generation.
CR - Computer room
CRAC - Computer room air conditioner (pronounced crack) that uses a compressor to mechanically cool air
CRAH - Computer room air handler (pronounced craah) that uses chilled water to cool air
Critical Load - Computer equipment load delivered by PDU output
CSI - Cold supply infiltration index, quantifies the amount of hot air mixing with cold inlet air prior to entering the rack. Cutout - An open area in a raised floor that allows airflow or cable feeds
CW - Chilled water
DC - Data center
DCiE - Data center infrastructure efficiency is an efficiency measure that is calculated by dividing the IT equipment power consumption by the power consumption of the entire data center. This measure is the inverse of PUE.
Dead Band - An HVAC energy saving technique whereby sensitivity set points of equipment are set more broadly to improve coordination of the equipment and avoid offsetting behaviors, also dead band control strategy
Delta T - Delta temperature, the spread between the inlet and outlet air temperatures of air conditioning equipment, measured as the maximum achievable difference between inlet (return) and outlet (supply) temperatures. This is not the astronomical
measurement of Terrestrial Dynamical Time minus Universal Time.
Dewpoint - The temperature at which air reaches water vapor saturation, typically used when examining environmental conditions to ensure they support optimum hardware reliability
D/H - Dehumidifying/Humidifying
Dry-Bulb Temperature - The temperature of the air measured using a dry-bulb thermometer, typically taken in conjunction with a wet-bulb reading in order to determine relative humidity
EFC - Equivalent full cabinets, the number of full cabinets that would exist if all the equipment in the data center were concentrated in full cabinets
ESD - Electrostatic discharge, more commonly ‘static discharge'
F - Degrees Fahrenheit Ft2 - Square feet or foot
GPM - Gallons per minute
HAC - Hot aisle containment system that directs heated air from the outlet side of racks to air conditioning equipment return ducts in a highly efficient manner Harmonic Distortion - Multiples of power frequency superimposed on the power waveform that causes excess heating in wiring and fuses Heat Exchanger - A device used to transfer heat energy, typically used for removing heat from a chilled liquid system HDG - Hot dipped galvanized Hot Aisle - An aisle where rack backs face into the aisle. Heated exhaust air from the equipment in the racks enters this aisle and is then directed to the CRAC return vents. HPDC - High-performance data center, a data center with above average kW loading, typically greater than 10kW/rack Hot Spot - An area, typically related to a rack or set of racks, where ambient air temperature is above acceptable levels. Typically caused by heat generation in excess of cooling equipment capacity. Hp - Horsepower
Hr - Hour HVAC - Heating, ventilation and air conditioning system, the set of components used to condition interior air including heating and cooling equipment as well as ducting and related airflow devices
In-Row Cooling - Cooling technology installed between server racks in a row that delivers cooled air to equipment more efficiently Inlet Air - The air entering the referenced equipment. For air conditioning equipment this is the heated air returning to be cooled, also called return air. For racks and servers this is the cooled air entering the equipment. IP - Internet protocol, a communications technology using the internet for communications. IR - Infrared spectrum used by thermal imaging technologies
JVM - Java virtual machine, Java interpreter. Software that converts the Java intermediate language into executable machine language.
kBTU - Kilo British thermal unit, one thousand BTU, a unit of measurement for the cooling capacity of a CRAH kCFM - Kilo-cubic feet per minute, 1000 CFM kV - Kilovolt kW - Kilowatts kWc - Kilowatts of cooling, alternate unit of measurement for the cooling capacity of a CRAH kWh - Kilowatt hours kVA - Kilovolt amperes = voltage x current (amperage) KVM - Keyboard, video, mouse; an interface technology that enables users to access multiple servers remotely from one or more KVM sites. More obscurely, can also mean K Virtual Machine: a version of the Java Virtual Machine for small devices with limited memory.
Latent Cooling Capacity - Cooling capacity related to wet bulb temperature and objects that produce condensation Line Noise - Distortions superimposed on the power waveform that causes electromagnetic interference Liquid Cooling - A general term used to refer to cooling technology that uses a liquid circulation system to evacuate heat as
opposed to a condenser, most commonly used in reference to specific types of in-row or close-coupled cooling technologies Load - The kW consumption of equipment, typically installed in a rack. Also, the heat level a cooling system is required to remove from the data center environment
MAH - Makeup air handler, a larger air handler that conditions 100% outside air. Synonymous with MAU. Make-Up Air - The conditioned air delivered by a MAU or MAH MAU - Makeup air unit, a larger air handler that conditions 100% outside air. Synonymous with MAH. Maximum Temperature Rate of Change - An ASHRAE standard established to ensure stable air temperatures. The standard is 9 degrees F per hour. MERV - Minimum efficiency reporting value , ASHRAE 52.2, for air filtration measured in particulate size.
N+1 - Need plus one, a redundancy concept where capacity is configured to include planned capacity plus one additional device to enable continued operations with the failure of one system in the configuration. This presumes immediate detection and remediation of the failed unit. NEBS - Network equipment-building system design guidelines applied to telecommunications equipment No. - Number Nominal Cooling Capacity - The total cooling capacity of air conditioning equipment, includes both latent and sensible capacities. Due to humidity control in data centers, the latent capacity should be deducted from nominal capacity to determine useful capacity
OpEx - Operating expense, the ongoing expenses related to operating the data center Overcooling - A situation where air is cooled below optimum levels. Typically used in reference to rack inlet temperatures.
PDU - Power distribution unit Pole - A row of power receptacles with power supplied from a PDU Pole Position - A power receptacle on a pole Pressure Differential - The difference in pressure between two locations in the data center used to analyze air flow behaviors
PH - Phase, electrical phase 1-3 Plenum - A receiving chamber for air used to direct air flow PU - Packaged unit, an air handler designed for outdoor use. PUE - Power usage effectiveness, a measure of data center energy efficiency calculated by dividing the total data center energy consumption by the energy consumption of the IT computing equipment. This measure is the inverse of DCiE.
Rack - Device for holding IT equipment, also called a cabinet RAH - Recirculation air handler, a device that circulates air but does not cool the air Raised Floor - Metal flooring on stanchions that creates a plenum for airflow and cabling, synonymous with RMF Recirculation - Chilled airflow returning to cooling units without passing through IT equipment, also referred to as short cycling Return Air - The heated air returning to air conditioning equipment RFI - Radio frequency interference Rh - Relative humidity RMF - Raised metal floor, an alternate term for the more commonly used term ‘raised floor' ROI - Return on investment, a measure of the amount of time required to recover an investment RPM - Revolutions per minute, used to measure fan speeds RPP - Remote power panel RTU - Rooftop unit, an air handler designed for outdoor use mounted on a rooftop. A typical application of a PU.
S+S - System plus system SCFM - Standard cubic feet per minute, the volumetric flow rate of a gas corrected to standardized conditions of temperature, pressure and relative humidity Sensible Cooling Capacity - Cooling capacity related to dry bulb temperature and objects that do not produce condensation Sensitivity - An equipment setting that bounds the set point range and triggers a change in device function when exceeded. Most commonly referring to CRAC/CRAH temperature and humidity set points. Set Point - Typically used in reference to air conditioning equipment thermostat temperature and humidity settings Short Cycling - Chilled airflow returning to cooling units without passing through IT equipment, also referred to as recirculation
STS - Static transfer switch Sub-Floor - The open area underneath a raised computer floor, also called a sub-floor plenum Supply Air - The cooled airflow emitted from air conditioning equipment
TCE - Triton Coefficient of EffectivenessSM, synonymous with UCE Thermistor - A type of resistor with resistance varying according to its temperature
U - Rack mount unit, the standardized height of one unit is 1.75" UCE - Upsite Coefficient of EffectivenessSM, synonymous with TCE UPS - Uninterruptible power supply, device used to supply short-term power to computing equipment for brief outages or until an alternate power source, such as a generator, can begin supplying power.
VFD - Variable frequency drive
W - Watt(s) Waterside Economizer - An economizer that redirects water flow to an external heat exchanger when the exterior ambient air temperature is at or below a temperature required to chill water to a given set point, simultaneously shutting down the mechanical chiller equipment. Wet-Bulb Temperature - The temperature of the air measured using a wet-bulb thermometer, typically taken in conjunction with a dry-bulb reading in order to determine relative humidity Wg - Inches of water column, an approximate unit of measurement for static air pressure Work Cell - The area of a rack and the related area immediately in front of and behind the rack. Standard racks are 2 feet wide and 4 feet deep. Standard aisles are 4 feet wide, so half of that space is workspace for a given rack. This results in a standard work cell of 16 square feet. Actual work cell size varies with data center design. WPSF - Watts per square foot

Underground Data Centers New Trends

Saturday, October 17, 2009

IBM Generator Failure Causes Airline Chaos

IBM Generator Failure Causes Airline Chaos

No comments:

Post a Comment

Comparison chart: Underground data centers

About Me

List of Underground Data Centers (click on name)

Glossary of Data Center Terms

Labels