Basics

What is passive 2-phase immersion cooling (aka open bath immersion cooling)?

 

In this simple concept, hardware is immersed in a nonflammable dielectric liquid that boils on the heat generating electronics. Because the enclosures are not hermetically sealed and they are not under pressure (just like a freezer, the lid is simply closed when access is not needed), it’s also called open bath immersion cooling.

If it sounds a little radical, know that the boiling point is very low (ie. 34°, 49°C, depending on fluid) and that passive 2-phase (evaporative) immersion cooling has been around for decades to cool high value electronics in tens of thousands of systems including super computers, industrial applications and radar or microwave relays. The key difference is that all these systems are sealed and under pressure.

Passive 2-phase immersion cooling (also known as open bath immersion cooling) eliminates all these complexities and makes the system very practical.

Isn’t passive 2-phase immersion cooling very expensive?

 

On the contrary. The elegance of passive 2-phase immersion cooling lies in reducing the total cost of ownership (TCO), and this starts at the capital investment to acquire the technology.

But the advantages don’t stop a the direct cost, designing a hardware for an immersion cooling system is much less complex than for traditional systems. The advantages are numerous both from a technical and financial standpoint.

What does 2-phase mean?

 

2-phase simply means the heat is removed passively by boiling and condensation (the liquids used are very special as they boil at very low temperatures).

A phase change takes place when a substance changes from one state (liquid) to another (gas). The boiling action automatically causes convection and passively reduces hot spots without any internal pumps.

This is in contrast to single phase (or forced convection cooling), which takes heat away by actively pumping cooling liquid to and from the hot spot.

Why hasn’t anyone done this before?

Passive 2-phase immersion cooling is not new, but the simplified way how we use this technology for cost saving is new. So why hasn’t anyone done this before?

In short, the computer and chip technology has matured in the past few decades. What we couldn’t do just 20 years ago is now possible due to modern chip packages that incorporate heat spreaders with high performance interfaces are routine. Other factors are the increased demand for high density (ie. GPU computing) that drive our developments, and a few very simple things such as the availability of SSD for storage or new types of hard drives (Helium).

Here is an article with more technical details you might find interesting (starting with IBM’s research into this area in the 1970s):

http://www.allied-control.com/blog/why-hasnt-anyone-done-this-before

Do I have to change or refill the liquid?

Our systems are primarily designed to save money, improve efficiency and allow high density. Changing fluid or refilling is not necessary with well designed systems, because they are made to avoid or minimize losses (including vapor) or contamination. We found that spilling liquid while pouring in and out of containers is by far biggest source for losses, and we have integrated ways to avoid spills on initial filling, along with many other technical features that are designed to keep fluid in the tank.The fluids are inert and have a shelf life of 30 years. There will be no significant degradation of the fluid when it is used for electronics cooling (in a recent study, scientists looked at fluid from servers after 23 months in use). We still have fluid that was first used over 20 years ago.

Is 3M™ Novec Engineered Fluid™ as messy as oil?

No, absolutely not. 3M Novec Engineered Fluids are very clean. The liquids are definitely not oily and are much thinner than oil (thinner than water actually). There will be no residue on any parts, such as boards, chips or cables. There is no need to clean up, and in fact your hardware will be cleaner than ever before (3M Novec is also used as solvent to clean during the manufacturing stage, thing of the inside of hard drives such as on the discs themselves). It’s also non-toxic, doesn’t harm the environment, and is non-flammable. There is no need to pump Novec like oil, so there are no noisy pump stations in the datacenter.

What’s the disadvantage of immersion cooling?

You can’t call 1-800-DELL to order your hardware and set it up next week.Also, at this moment of time, most hardware has been designed with air cooling in mind. This means we are often spending a lot of time to remove bulky heatsinks and disconnect fans. Big power connectors that are fine for air cooling (there is plenty of space to spare with air cooling) might be in the way to stack hardware in a density that would be possible with immersion cooling.

The advantages far outweigh the disadvantage already now. But one can only imagine the true potential once hardware manufacturers discover how much smaller and more efficient they could design their hardware with full immersion cooling. Luckily for us, Intel and other manufacturers are looking ahead and start to offer hardware in denser form factors (ie. Intel® Xeon Phi™ DFF cards).

 

Safety

Is 3M™ Novec Engineered Fluid™ harmful or dangerous?

 

The liquid is very safe. In fact, there is no expected irritation if it is touched, makes eye contact, and no health effects are expected if it is swallowed. The liquid is considered a very clean fluid with no global warming potential and it does not deplete ozone.MSDS (Material Safety Data Sheets) and other information is available upon request.

While the fluids are generally considered safe, we do encourage anyone that handles it to follow appropriate personal and industrial hygiene practices.

Is the fluid flammable?

 

Novec fluid doesn’t burn. On the contrary, it is used as state of the art fire protectant in data centers, operation theaters, museums and archives worldwide.Having inherent fire protection built-in can have great advantages and cost savings (Immersion-1 does not need any additional fire protection in the facility and should a board ever malfunction, the fluid protects the hardware itself from any damage).

Can one get hurt from touching a hot tank?

Passive 2-phase immersion cooling tanks boil at very low temperatures (ie. 34°, 49°C, depending on fluid). Touching the boiling fluid or the tanks is generally not a problem, just as touching a hair dryer on the outside is not a problem. Most tanks are insulated for increased efficiency, so they don’t even feel warm on the outside.

 

Products

Are you selling any finished products?

 

Allied Control is primarily an engineering and consulting company. While we are building finished solutions, they are custom built for our clients to match their specific high density requirements. It is currently not possible to buy finished systems online, but we are offering an immersion cooling development system for international shipping.Within our test platform in Hong Kong, we are also able to provide experimental or demonstration setups for clients.

Your website mentions rack enclosures. Do you plan to enter the data center market for server cooling?

 

Our focus is currently on high density hardware and not on standard servers, which only come with a couple of CPU cores and a little bit of memory to be cooled. High density hardware such as HPC (high performance computing) systems, supercomputers, GPU clusters with NVIDIA accelerators, nodes enhanced with FPGA cores or parallel processors such as the Intel® Xeon Phi™ coprocessor. High density specialised electronics such as pure FPGA clusters are also ideal candidates for immersion cooling.Standard servers are very bulky and waste a lot of space for their heatsinks, fans and air channels. One of the major technical advantages of immersion cooling is that you don’t need all that. It allows true high density systems with hardware packed in very small spaces. Server and hardware manufacturers start to realize the potential of this technology and the growing demand for cloud computing hardware (a large number of identical boards) is definitely helping to make a case for immersion cooling.

That’s why we are looking forward to manufacturers making their systems more dense and to data centers becoming more energy efficient.

 

Technical

What are the limits of passive 2-phase immersion cooling?

 

As with any new technologies, the limits have yet to be discovered. It has successfully been demonstrated that 4000 Watt can be cooled with only 200cc of fluid in a volume of just 1 liter. To give you some idea, that could mean 40 top-of-the-shelf Intel Xeon II CPUs with 6 or 8 cores each running at max performance in a tiny space of 100x100x100mm (or hundreds of FPGAs in the same tiny little box).

Is hot-swapping of hardware possible with passive two-phase immersion cooling?

 

Yes, hot swapping is possible. Hardware can safely and elegantly move in and out of the tank because of the fluid’s low boiling temperature (ie. 49°C).When hardware is pulled out of the liquid and moves up through the vapor into the headspace with air, it is usually already dry. In other words, hardware leaves the tank dry.

Novec™ Engineered Fluids are extremely clean, they are not greasy, oily or sticky, there will be no residue.

How is 2-phase immersion cooling different from oil cooling?

 

2-phase immersion cooling does a much better job at cooling electronics, at much lower cost with less resources. Because of the physical limitations of oil and cooling, high density as in 2-phase immersion would simply not be possible.Oil cooling is a single phase process. Oil cooling systems are not passive, it requires hydraulic pumps to take hot oil away from hot spots because oil is a fluid with high viscosity. Passive 2-phase immersion cooling uses fluids with a very low viscosity just like water. This means the liquid circulates passively, taking the heat away efficiently without any pumps. Oil needs filtration and treatment and has a limited lifespan when used for electronics cooling.

The engineered fluids in a passive immersion system are inert, have a shelf life of 30 years and show no degradation after long term use for cooling. While the fluid is not particularly hot (ie. 34°C or 49°C when it boils), the constant boiling and condensation is a distillation process on its own. It’s also very clean (not oily or greasy), has zero ozone depletion, very low global warming potential, and it is the best fire protection one can imagine (Novec 1230 is state-of-the-art data center fire protection).

Using large amounts of oil in a data center brings non-technical implications since oil is classified as dangerous and flammable. Government regulations, fire safety standards and insurance risks all require careful considerations.

Oil cooling can reduce electricity bills and the burden on the environment. However, in an oil cooling application fans and air filters are replaced with hydraulic pumps and oil filters, while passive 2-phase immersion cooling simply eliminates these resource hogs altogether.

Technically and physically, passive 2-phase immersion cooling is way ahead, and practically we believe the same is also true.

How is 2-phase immersion cooling different from “sealed server” or “closed capsule” cooling?

 

Immersion cooled sealed electronics has existed for many decades, for instance in military or aviation applications. They are problematic and costly to produce, because pressure can build up in a sealed system.Sealed systems usually also focus on smaller pieces of hardware with a lower power density, leading to the duplication of parts in use.Example: 20 main boards would require 20 sealed enclosures, 20 heat exchangers, 40 water connectors (in and out), 20 power power connectors, 20 monitoring devices, etc. All these parts would be very difficult to service. In an open bath immersion cooling approach, with 2-phase cooling, the enclosure is semi-open, doesn’t have to be sealed, and said parts are all avoided because only one enclosure exists. Easier to service, less expensive and complex to build.

Similar to oil immersion cooling, “sealed servers” are employing a single phase cooling process. This is much less efficient compared to an 2-phase systems (the heat transfer coefficient is much higher) and requires more components. High density such as in passive 2-phase systems would simply not be possible.

From a semantic point of view, sealed servers don’t represent a cooling process on its own that can be compared with passive 2-phase immersion cooling. It’s more a product line comparable to blade servers.

It’s an approach in the right direction to save energy, but it’s more an infrastructure approach – in an optimal case, a customer buys a number of servers from a company and sticks them into cabinets it bought form the same company. Compared to blades, one of the downsides is that memory or CPUs can’t be replaced in the field easily. Servers need to go back to the vendor or the vendor needs to send somebody to service the hardware. Not so in our open approach.

Technically, sealed servers use a dielectric fluid such as mineral oil or a fluid similar to the liquids used in passive 2-phase immersion cooling. But the heat removal process does not happen through evaporation and condensation.

Sealed servers, compared to air cooling, replace one evil with another (ie. fans and air filters with hydraulic pumps and oil filters or unserviceable parts), while passive 2-phase immersion cooling simply eliminates all these evils. Technically, passive 2-phase immersion cooling is way ahead, and practically we believe the same is also true.

Can we use this with NVIDIA GPU systems?

 

Yes, absolutely. Passive 2-phase immersion cooling also works with GPUs like NVIDIA Tesla, NVIDIA Kepler or other GRID GPUs, AMD Radeon Sky Series and everything else.It’s a truly universal system. One and the same enclosure can hold any of these, or a mixture of. And when a new generation hits the market, the cooling system does not have to be redesigned.

Does it work with hard drives (HDDs)?

 

Passive two-phase immersion cooling works best with solid-state drives (SSD) or sealed hard drives (HDDs), such as Western Digital’s (HGST) helium filled Ultrastar drives (He6 series).