Immersion liquid cooling small form factor and server-class systems
Small embedded systems can be cooled through submersion in new dielectric cooling fluids.
As long as computers have been in existence, engineers have struggled to cool their creations, partly because the methods of cooling have remained relatively unchanged. Recent advances have prompted some to explore a new spin on a very old method of computer system cooling: total liquid immersion.
Standing in our development and engineering lab during a recent experiment, I nervously watched while a computer system, powered on and lights blinking, slowly submerged into a clear, watery liquid. Technology professionals are conditioned at almost a primal level to believe that liquid + electronics = bad, so it was quite a novelty to observe a computer functioning perfectly, completely submerged in what appeared to be nothing more than water.
Immersion cooling, formerly reserved for only the most expensive and exotic supercomputers, can be effectively downsized and applied both to small form factor embedded applications and server-rack systems. The following discussion analyzes various forms of liquid cooling and explores how small embedded systems can be cooled through submersion in new dielectric cooling fluids.
Drawbacks of traditional cooling methods
First, let’s examine the shortcomings of air cooling and why designers might consider alternative methods of cooling their computer systems:
- Comparatively, air is an extremely inefficient heat conductor.
- Air cooling requires large open channels of space to direct airflow on the boards (increasing signal path lengths), inside the chassis, and between server racks.
- Fans consume a lot of energy, are noisy, and relentlessly pull dust and other contaminants into computer housings.
- Fans only cool the areas the airflow can effectively move across.
- Air cooling requires large heat sinks that add cost and weight to a system and waste a tremendous amount of space.
- Fans and air cooling move the heat to the surrounding atmosphere around the computer, where it is recycled into the system (further reducing efficiency) and dealt with via energy-consuming, oversized HVAC systems.
What about fanless small form factor machines? Such systems generally conduct heat directly to the surrounding atmosphere via the computer housing. The problem with this method is that the actual heat to transfer originates from only a very small surface area, greatly limiting efficiency. Dissipating that heat effectively requires careful engineering and thermally conductive materials to spread the heat over a larger area where it can radiate to the surrounding air. These conditions typically dictate that only lower-power, lower-heat processors can be used in such systems.
The answer to the shortcomings of air cooling high-performance systems has largely come in the form of liquid cooling. Active liquid cooling (pumped fluid) has gained more mainstream popularity in recent years, particularly in the high-performance computer gaming community. This generally involves a hybrid of air cooling, with liquid coolant applied via hoses to specific locations within a computer system.
However, liquid cooling for large-scale, hot-swappable server deployments has proven impractical and risky. Even when using expensive hose quick connects, engineers must deal with the potential for catastrophic leaks and the complexities of piping coolant around the insides of multiple computer housings in a rack. Thus, it’s understandable that IT professionals have not given this technology much thought.
Where small form factor systems are concerned, there simply isn’t room to contain liquids, radiators, hoses, and pumps in such confined spaces. Therefore, liquid cooling is ruled out almost immediately.
New fluids for immersive cooling
The concept of cooling circuitry via submersion has existed for years. However, this traditionally is limited to the most high-end supercomputing equipment. Immersion cooling has recently gained niche popularity as something to experiment with for high-performance video gaming systems, using mineral oil as the cooling fluid. While an interesting novelty, these mineral oil submerged systems still demand spot cooling for specific chips, thus requiring pumped fluid via hoses directly connected to the board. Removing components from the oil bath is an ungainly process, not only when dealing with the hoses and pumps, but also considering the sheer mess associated with dunking and withdrawing motherboards and computer systems in and out of tanks filled with slippery oil.
Back in the lab with the “drowning” computer (see Figure 1), engineers have found a solution to the oily mess using new dielectric cooling fluids. With these fluids, immersion cooling is not only a possibility, but also highly practical. Imagine a liquid that a designer can specify to a variety of boiling points. It looks just like water, but in actuality is nothing of the sort. Nonconductive and noncorrosive, the liquid allows a computer board to be submerged indefinitely without heat sinks or fans and cools it far more effectively than air. To hot-swap a system, simply withdraw the board from the liquid, and it dries almost instantly upon removal, with no residue or aftereffects.
Small form factor systems such as body-worn computers, vehicle computers, or computers in hazardous environments might use a passive single-phase system where the liquid stays in liquid form. Buoyancy-driven flows cause the fluid to circulate due to the differences between the device and the housing temperatures. The fluid touches every surface of the interior housing, so the heat transfer process can be very efficient. This also allows more powerful processors to be utilized than what was previously possible with traditional passive-cooling solutions.
For server-class systems, a two-phase system can be highly efficient, permitting a significant amount of heat to be dissipated by allowing naked chips to boil the fluid around them and thus carry heat away. Because the boiling point can be low, the onboard chips remain well within operating parameters and form nontoxic, completely inert bubbles around the hottest components. Those gases rise up, condense, and recirculate back into the liquid bath. A nonrefrigerated system of water or other heat-conductive fluid can be pumped through a radiator system over the bath to condense the gases. Alternatively, it could flow around the outside of the tank holding the computers, thus carrying the excess heat to be vented into the atmosphere outside the facility.
Measuring immersion cooling efficiency
So how efficient is this cooling method? Research suggests that power densities are limited more by the electrical bus than by the capabilities of the cooling fluids.
Estimates reflect that a full server rack assembly using 80 W of fans per kW of server power amounts to $2,800 a year in energy costs just to operate the server fans. Fluid cooling would reduce that cost $123 per year, not taking into account the savings from reduced HVAC energy usage.
Furthermore, by eliminating heat sinks, fans, and the airspace normally required for air cooling, a server can become very compact and dense, packed into a confined space along with many more identical servers. Under normal to nearly ideal conditions using traditional air cooling, one would expect the temperature of a processor to be 20 °C to 30 °C higher than ambient temperature. Using submerged fluid cooling with reasonable surface area for heat dissipation, this delta drops to a consistent 10 °C over ambient temperature. This efficiency gain is also significant for small form factor applications.
Engineers in the lab performed one test that involved the thermal equivalent of cooling 30 full-speed Intel i7 940 processors (120 cores) in a less than 8 x 8 square-inch area. To simulate this, engineers mounted an array of twenty 19 mm x 19 mm heater assemblies on a circuit board, each generating 200 W of heat. This 4,000 W (4 kW) assembly was submerged in 200 cc of fluid with a 49 °C boiling point. Water circulated around the outside of the closed tank, entering circulation at 37 °C and leaving at 47.1 °C, and dissipated the heat to the air outside the building.
The water was only capable of air cooling to 37 °C, yet it maintained a consistent temperature of 58 °C on all the chips. This was accomplished at just a fraction of the energy usage that would otherwise be spent on an array of fans and HVAC climate control systems to manage the heat generation. While this example is on the extreme end with unrealistic heat densities, it illustrates the cooling capacity of the fluid in a submerged server or small embedded system.
This idea of submerging computer systems for cooling may eventually catch on with the commercial masses. The technology and its substantial benefits are available and ready to be implemented. Engineers with special cooling needs are encouraged to participate in this evaluation and exploration project, and perhaps in time we can determine if the cost and environmental savings can outweigh the primal urge to keep liquids away from technology.
In addition to the efficiency benefits, these new dielectric fluids provide some degree of entertainment value. Besides the potential for practical jokes (“accidently” dumping a glass of “water” on a laptop or iPhone), it’s somewhat relaxing, in a geeky zen sort of way, to see fans turning and lights blinking on a computer as it bubbles away, blissfully unaware of its strange, new liquid environment.