Advanced Thermal Solutions

The IoT – Internet of Things – includes all devices connected to the Internet, a fast growing world, expanding well past PCs and smartphones. IoT devices are our every day and special purpose items like appliances, sensors, and motors.

Per one authority, an IoT product combines hardware and software, measures real-world signals, connects to the Internet, transfers data to a centralized location, and provides value to a customer. [1] There are many such products, already in the tens of billions of IoT connected devices.

Consumer Uses

Among the most common IoT devices are smart speakers, headphones, appliances, and leak detectors. At the enterprise level are smart lighting and security systems in factories, office buildings and public places.

Automotive Use

Secure, strong IoT connection is essential for today’s auto infotainment systems. It enables everything from music platforms to navigation aids to car maintenance and diagnostics. Connected cars have more and more software-reliant components in their cabins, under their hoods and just about everywhere. With IoT-connectivity these components can be updated with OTA (over the air) software fixes without visiting a garage or dealership.

Industrial Use

The IIoT – Industrial Internet of Things – consists mainly of sensors uploading localized data to monitor and control manufacturing processes. It is active in manufacturing, transportation, and other  areas, from facility energy usage to equipment performance.  IIoT devices collect, analyze, and share data with other devices and with everyone needing to know.

IoT Connections

Devices like these described so far are one end of the IoT technology stack. They are the public-facing “things,” but just one layer of the stack. The data they send or receive travels up and down layers of connection points and software programs. At the stack’s other end are the device master applications and data storage residing in the datacenter-held cloud.  Because IoT things/devices differ so widely by function, there are many different software platforms in use. The largest telecom and data companies are all active IoT developers in this continuously evolving arena.

Thermal Management Issues in the IoT

Connected consumer IoT devices, or things, are typically very low power. No added thermal management is needed. Power levels are more likely to increase with devices serving the Industrial IoT, though passive cooling, e.g. heat sinks, remedy most heat issues.

To effectively manage generated heat, cloud-hosting datacenters are using air-cooling, direct-liquid, and immersion cooling (submerged server) systems. As transistor densities increase on smaller package chips, the centers must enhance their thermal management capabilities, while managing power consumption and operation costs.  

IoT – What Comes Next?

These days the IoT is focused on two more letters: AI. In fact, the Artificial Intelligence of Things (AIoT) takes the I (Internet) for granted and enables smart devices and systems to analyze data, make decisions, and act on that data without any interference of humans.

Most AIoT applications are currently retail product-oriented and focused on the implementation of cognitive computing in consumer appliances. For instance, computer vision systems can leverage facial recognition to recognize customers, and compile demographic and preferences data about them. Also, Tesla’s autopilot systems are using radars, sonars, GPS, and cameras to glean data about driving conditions. Then an AI system makes decisions about the data the internet of things devices are collecting to optimize the car’s piloting. [15]

References

1. DanielElizalde, https://danielelizalde.com/what-is-the-internet-of-things/
2. Kangaroo, https://heykangaroo.com/products/doorbell-camera-chime
3. GoodWorkLabs, https://www.goodworklabs.com/apple-watch-app-is-the-smart-watch-revolution-finally-making-sense/
4. StreamLabs, https://streamlabswater.com/products/streamlabs-water-monitor
5. PlanetDave, https://planetdave.com/2023/11/a-delightful-drive-the-2023-toyota-crown-platinum/
6. Biz4intellia, https://www.biz4intellia.com/iot-sensors/
7. IoTsens, https://www.iotsens.com/en/product/sound-monitor/
8. Weisstechnik, https://weiss-na.com/product/webseason-the-controller-designed-by-and-for-end-users/
9. https://iotbusinessnews.com/2022/07/13/86750-what-is-the-iot-technology-stack/
10. RAKwireless, https://news.rakwireless.com/wisgate-connect-why-did-we-did-it/
11. JetCool, https://www.datacenterknowledge.com/power-and-cooling/liquid-cooling-adoption-data-centers-becoming-zero-sum-game
12. Engineered Systems, https://www.esmagazine.com/articles/100400-air-cooled-chillers-are-back-in-data-centers-and-they-mean-business
13. Microsoft, https://news.microsoft.com/source/features/sustainability/project-natick-underwater-datacenter/
14. Tealcom, https://tealcom.io/post/the-intersection-of-ai-iot-and-connectivity/
15. IndustryWired, https://industrywired.com/how-artificial-intelligence-can-help-manage-flood-of-iot-data/

The use of artificial intelligence (AI) programs is growing very quickly, despite some concerns and precautions. It’s being spurred by powerful new hardware from companies like Nvidia, and by new lower-cost, open source large language model (LLM) software like those from DeepSeek.

Likewise, AI chip sales are soaring, and more powerful and specialized AI chips are being steadily introduced. At this writing, Nvidia, the leading AI chip provider is now the third-most-valuable company in the world, valued at over $2.2 trillion. Nvidia is both developing new AI chips and acquiring smaller AI companies that design processors and develop AI applications. [1]

Other chip companies are also thriving. Micron Technology is reporting record sales, much of them from supplying memory chips to Nvidia. AMD provides chips that rival Nvidia’s flagship AI machine learning chip. And Intel is getting $8.5 billion from a US federal program (CHIPS) to support its goal to build the largest AI chip manufacturing site in the world. [2,3]

Innovative Chips Bring High Heat

AI chip technology, which evolved from the graphics processing units, GPUs, developed for the data needs of video games, may be the most understood part of the AI world. But this evolution is remarkable. The Nvidia A100 AI Chip features 54 billion transistors. By comparison, an AMD Ryzen 7 1700 gaming processor for a contemporary PC has 4.8 million transistors. [4]

By leveraging parallel processing capabilities, AI chips effectively handle large datasets, allowing multiple tasks to be executed simultaneously. These chips interact with special ASICs, FPGAs, TPUs and VPUs to perform machine learning and neural network processing. The AI networks can solve complex algorithms and are teaching computers to process data in a way that is inspired by the human brain.

For example, AI can use inference – combining reasoning and decision-making based on available information – to apply real-world knowledge for facial recognition, gesture identification, natural language interpretation, image searching  and much more. [5]

AI chips demand high power to support increased processing demand. As a result, excessive waste heat can degrade performance or trigger system failure. AI system designers depend on thermal management solutions to manage AI processor temperatures. Cooling resources at both chip-level and facility, e.g. data center scale are needed to keep AI chips functioning at proper temperatures.

Liquid Cooling AI Chips

The heavy lifting in AI processing is done in data centers, which are the focus of most technical developments. Their high concentration of high-power chips presents formidable heat management challenges, especially when the thermal design power of the GPU has increased over the past two decades, rising from 150 watts to more than 700 watts.

Now consider the recently unveiled 1200 watt Nvidia Blackwell B200 tensor core chip—the company’s most powerful single-chip GPU, with 208 billion transistors—which Nvidia says can reduce AI inference operating costs (such as running ChatGPT) and energy. Two of these B200 chips are combined with an Nvidia Grace CPU to complete the newly released, even higher-performing GB200. Its total projected power draw: up to 2,700 watts.

The GB200 chip is a key part of Nvidia’s new GB200 NVL72, a liquid-cooled data center computer system designed specifically for AI training and inference tasks. Amazon Web Services, Dell Technologies, Google, Meta, Microsoft, OpenAI, Oracle, Tesla, and xAI, are expected to adopt the Blackwell platform. [8]

The ever increasing number of transistors attached to data center PCBs translates to higher performance but also more heat than ever before. Liquid cooling systems, like in the Nvidia data center system can significantly reduce energy consumption. This leads to lower operating expenses in the long run. It also produces less noise and, for direct on chip cooling, takes up less space. 

Direct to chip or node cooling involves circulating a coolant directly over heat-generating components, including AI chips. This method significantly increases cooling efficiency by removing heat directly at the source. These systems can use a variety of coolants, including water, dielectric fluids, or refrigerants, depending on the application’s needs and the desired cooling capacity. [8]

Immersion cooling takes liquid cooling a step further by submerging the entire server, or parts of it, in a non-conductive liquid. This technique can be highly efficient as it ensures even and thorough heat absorption from all components. Immersion cooling is particularly beneficial for high-performance computing (HPC) and can dramatically reduce the space and energy required for cooling.

Air Cooling AI Chips

Nvidia’s Jetson chips bring accelerated AI performance to IoT and Edge applications in a power-efficient and compact form factor (smaller than 100mm x 100mm). Less power-consuming (up to 75 watts) than data center AI chips, thermal management is still needed. Jeston components are typically cooled with heat sinks, which can be configured as active (with attached fan) or passive (fanless). 

Conclusion

Artificial intelligence is hot in the marketplace. So are AI chips. As complex as they are, simply surpassing a heat threshold can affect their proper function. Their thermal management is essential.