Oils ain't Oils
If you are Australian, and over 40, you might remember this as a nostalgic catchphrase from the 80's and 90's, advertising motor oil made by Castrol.
While typically referred to as Oil, immersion cooling uses dielectric fluids to cool IT load - a non-corrosive, non-conductive liquid and either non-flammable or very high flashpoint, that has the viscosity of a mineral or baby oil.
Types of Immersion Cooling
Single-phase and two-phase immersion cooling are two different approaches to cooling electronic components by submerging them in liquid.
Every vendor has an explanation of the differences between the two, with useful infographics, however I like Submer's;
In single-phase immersion cooling, the liquid remains in a liquid state throughout the cooling process. The heat from the electronic components is transferred to the liquid through conduction. The liquid is then cooled by a heat exchanger or other cooling system.
In two-phase immersion cooling, the liquid boils to a vapor as it absorbs heat from the electronic components. The vapor then condenses back to a liquid on a heat exchanger, releasing the heat to the cooling system.
Some other disadvantages of two-phase have been a less competitive market for the fluid (3M's Novec was a default choice), and given the evaporation process, there are some fluid losses which require topping up. Earlier this year, 3M announced they are phasing out production of Novec (and Fluorinert) as they contain PFAs and whilst there are newer solutions, AFAIK, they are still based on fluorinated coolants - disclaimer: I'm not a chemist.
Below are the key differences between single-phase and two-phase immersion cooling.
The best approach to immersion cooling will depend on the specific application. For applications where efficiency is critical, two-phase immersion cooling may be the better option. However, for applications where cost and complexity are important factors, single-phase immersion cooling may be a better choice.
Some things to consider when choosing between single-phase and two-phase immersion cooling:
IT Load being cooled - some components are more sensitive to the effects of boiling and condensation than others.
Environmental conditions - two-phase immersion cooling systems are more sensitive to changes in temperature and humidity than single-phase immersion cooling systems.
Maintenance required - single-phase immersion cooling systems are generally easier to maintain than two-phase immersion cooling systems. De-pressurizing a rack full of IT equipment to replace a PSU or HDD is a heavy operational burdon.
On the topic of cooling, I'd be remiss to not cover another type, Direct to Chip (D2C). It is achieved when fluid is passed through cold plates which are directly attached to chips (CPU/GPU). This design can use either a dielectric or non-dielectric fluid (for example, water glycol).
A example of D2C cold plate with a hot and cold loop
A quick history lesson...
Immersion cooling isn't new; it has roots that date back several decades. The early iterations of this cooling method can be traced to the 1960s when it was employed in specialized applications like military electronics and space exploration. IBM has been involved in immersion cooling since that time, having the first patent granted in 1968!
In the 1980s, IBM developed the Thermal Conduction Module (TCM) cooling concept, which used water to cool high-performance bipolar circuits in large multi-chip modules. The TCM cooling concept was used in IBM's mainframe computers, such as the ES/9000.
In the 2000s, IBM began to explore the use of immersion cooling for data center applications. In 2005, the company released a rear-door heat exchanger for use with its dense blade servers. Interesting given the RDHx developments in recent years…
In 2008, IBM introduced the Power 575 supercomputer, which used a direct cooling system with water-cooled copper plates above the microprocessors. This was the first liquid-cooled system for a mainframe since the ES/9000.
In recent years, IBM has continued to invest in immersion cooling research and development. In 2020, the company announced a partnership with Submer to develop immersion cooling solutions for data centers.
And now?
Whilst HPC workloads are being cooled via immersion methods today, the real litmus test for many, for the past few years, and before pursuing Cloud Services, was to throw converted air-cooled Bitcoin miners into tanks.
High power + high heat = great use case for immersion.
Hundred's of ASICs, littered across multiple PCB's, and sandwiched together with huge heatsinks, a SUPER loud fan, and a 2-3000W (usually) poor quality PSU.
Whatsminer, Bitmain, Canaan and others, all have air-cooled SKU's, and now at least Whatsminer and Canaan's both have immersion-first designed SKU's (and 5000W+ PSUs!), with Bitmain actually offering a full container for hosting.
I started today's article with the idea of talking about different fluids in immersion, and ended on Crypto - I'll guess that will be the topic for a future article.