< All posts

Immersion-2 Tech Details: Cooling Hundreds of Kilowatt with 1500 Watt

I must have received a few dozen emails this week asking for more tech details of Immersion-2. Here is a quick update while Kar-Wing is en route to SC13 in Denver.

Immersion-2 is an immersion cooling platform design for 3M™ Novec™ Engineered Fluids such as Novec 7000 and Novec 649. The system features a modular design in a 19-inch server rack form-factor, with various custom shapes and sizes available on request (think oversized bathtub).

Cooling Hundreds of Kilowatt with 1500 Watt

At this moment, our systems are nowhere near capacity and we are effectively cooling hundreds of kW with just a few thousand Watt.

The following is a snapshot of system stats while I am typing this post:

 Outdoor Temperature:  24.3°C
 Relative Humidity:  70%
 Water-In Temperature:  34.4°C
 Water-Out Temperature:  39.9°C
 Room Temperature:  28.5C
 Radiator Frequency:  12Hz
 Pump Frequency:  25Hz
 Radiator Power:  0.4kW
 Pump Power:  1.1kW

Pump and radiator are running on German three-phase 400VAC motors, Hertz is the AC frequency.

To be fair, we are also running a tiny little window type air condition at the other end of the data center, as we have people working there. That AC unit consumes more power than our system altogether it this moment. It also keeps the Novec reserves nice and cool 😉

Also notice that 24°C is considered cold around here, winter is closing in. We definitely expect a few Kilowatt more during the heat of the summer, but with the current limit of 500kW we will probably never have to run more than 25Hz even when the HK observatory issues our beloved “very hot weather warnings” (around 10-20 days a year). A few weeks ago in hot weather, we were running on a PUE of less than 1.02 on full load.

Only 25% of Condenser Capacity in Use

The condensers are very efficient. The current hardware is only scratching the surface of single condenser capacity at water temperatures of around 35°C. Total capacity of a condenser is ~25kW at that temperature and we can install three more per tank. Our systems can also modulate water flow rates or increase water temperatures (hot water cooling) to get more out of them. We could easily move up to different fluids with higher boiling points to save more energy. Or we could automatically switch on or off our dry cooler modules. All this will have a large impact on performance and efficiency (we are not just talking a few percent here). And all these changes, except adding more condensers, don’t even require a screw driver. And having modular condensers also means we could always upgrade for something with more capacity.

b2ap3_thumbnail_BirdsViewTank.jpgMost of the real estate is headspace right now.

Only 30% of Space Used in Tanks

Only the lower 30% of the physical space in the tanks is full with hardware and fluid at this moment. The condensers are modular and there is a rail on top. Filling our enclosures with more hardware or arrange it differently (ie. two layers), is very easy if the client decides to move to next gen hardware, and so is packing things a little denser (given that they can get rid of the LAN wiring and integrate communications to a backplane).

19″ Standard Racks without Pumps or Heat Exchangers

Immersion-2 is very simple. Our racks are off-the-shelf and don’t need additional water pumps, heat exchangers or other parts. Immersion-2 does not have or need a raised floor. The water circuit spans along the aisle between two rows of racks. Depending on ceiling height, facility design or container, water outlets can come from above or below. The total system can be arranged in standard shipping containers, and water distribution is NOT attached to the racks so they can move freely (or be used for standard equipment any time).

Precision Manufacturing

We are working with a small number of precision manufacturing partners around the globe. Vital parts are made in Germany and the USA.

There are many other extremely important components that we are essentially developing and manufacturing in-house with our own SMT line or CNC machines, after a lot of experimenting and fine tuning during the past two years. We are also getting input from outside resources and we have partners lined up to scale up their own development efforts.

b2ap3_thumbnail_racks.jpgThis pioneering work allows us to acquire expertise and skills that we can ultimately pass on to our clients – and help them build their own immersion cooling systems.

Board to Board less than 10mm on 300A Backplanes

Just as with Immersion-1, we have arranged the hardware on custom backplanes with a distance of less than 10mm (center to center) between blades. In fact, we have arranged hardware in pairs, with two boards facing each other (components to components, back to back) to fit more nodes.

Each second board gets its power from a different power lane, with two 2000W @ 230V Platinum standard PSUs pushing a total of over 300A through each backplane. There are two of these backplanes in each layer. We have been working on designs with up to 16 of these PSUs in a row (GPU).

Most of the heat generating parts require less than a few millimeters for the fluid to take the heat away. We have minimized ‘empty space’ in order to save costs and create a denser package.

Populated racks of Immersion-2

b2ap3_thumbnail_PowerCooling.jpgTotal 500kW Hardware fits in the Size of 5 Racks

Practical considerations aside (serviceability, redundancy, etc), if we would consolidate all hardware of all our racks into one single tank that we fill with liquid, the space required would be less than 4 racks. That includes power supplies, backplanes and power distribution. Add condensers on top and a little bit of wiggle room and we’d be at around 5 racks.

The power density of the hardware in Immersion-2 could easily increase five to ten times if you add GPU accelerators into the mix. One thing that wouldn’t need to change on such an upgrade is our tanks or anything related to the cooling system around the hardware.

500kW Facility Power Limit != Cooling System Capacity

Installing Immersion-2 in an existing unit of a high rise building was possible for two reasons: it saves a lot of space and it saves 600kW of cooling power that would normally be required. With only slightly more than 500kW available at the facility, the air- or water-cooling alternative would have been to deploy only half of the critical IT load or look for a larger facility adding extra costs and wasted time.

600kW cooling power requirement if using CRACs or chillers is a true estimation by a well-known company specialized in data center thermal solutions that shall remain unnamed. Hong Kong, due to its hot and humid climate and no free cooling, has extremely high PUEs of 2.0 or more.

Oversized and ready for much more than 500kW.

Oversized Closed Loop Water Circuit & Electrical System

The 500kW installation of Immersion-2 was built into a facility that is currently limited to a three phase power supply of 800A 400VAC. In order to accommodate a possible increase in capacity, we have built a simple closed loop water circuit. When sizing the flow, pipe size and other properties of the circuit, our engineers intentionally went “a couple of sizes larger” than what is technically required. Water volume is around 1200 liters and it’s a closed loop water system without any chemicals, no water is being wasted.

The extra cost of using larger pipes is minimal, especially in view of the gained extra lifespan and room to for much higher IT loads. As a side effect, the efficiency of the water system is greatly improved, as larger systems run smoother, which leads to further energy savings.

b2ap3_thumbnail_500kWDryCooler2.jpgSilent Operation

Inside, Immersion-2 operates absolutely silent. The only noise that fills the room comes from a couple of 40mm fans in a handful air cooled Intel 1U low-end servers, which have nothing to do with the cooling system. Sooner or later, these 250W servers will have to move into one of the tanks just because the noise they generate.

Make no mistake, our system is silent on the outside too. Our system automatically slows the dry cooler to the minimum required fan speed, which is at this time of the year (around 25°C outside air temps) just 10-15Hz. The result is a dry cooler which operates below the background noise levels. Besides, the dry cooler employs fans that borrow their blade design from birds and aero-acoustic research. The same can be said for our water pumps. With only one of them online at a time, they rarely run more than 25Hz and the noise level when standing right next to them is less than that of a small 3HP outdoor air condition unit.

Inside view of one of our dry cooler units.

b2ap3_thumbnail_Dry-Cooler.jpgContainer Size Datacenter

The 500kW installation of Immersion-2 has a footprint of less than 15m² and was built with a ceiling height of less than 3 meters. Even with pumps and dry cooler, it all fits in a standard shipping container and can be put to work even in extreme weather conditions. No air is being pushed through the IT system and challenging or harsh environments are definitely not a problem for immersion cooling.

That’s all it takes to cool 500kW. German fans running on a few Hertz and an inexpensive Dry Cooler from China.