As AI and machine learning tasks put more strain on cloud computing systems, a new technology called Gigaflow could help ease the pressure.
Developed by researchers at the University of Michigan and their partners, Gigaflow is a smart way of managing memory in large data centers, and it significantly improves how data is processed and directed.
The team introduced Gigaflow at a major computer science conference in the Netherlands, and their results show impressive gains.
Compared to current systems, Gigaflow delivers up to 51% more accurate memory use (known as “cache hit rate”) and 90% fewer errors when trying to find stored data.
Modern cloud systems handle much more than just basic web browsing or email. With more people and companies using AI-powered tools, data centers must now handle much larger workloads.
To keep up, they rely on special hardware and software that lets multiple users and programs run on a single server efficiently.
In these systems, virtual switches play a key role in directing traffic. But as internet speeds climb—now reaching up to 400 gigabits per second and soon 800—older systems that depend on CPUs can’t keep up.
That’s why newer data centers use SmartNICs—programmable network cards that handle data traffic faster.
Gigaflow improves how SmartNICs manage memory. Normally, systems store entire flows of data as they arrive, based on the idea that recently used data will soon be used again. But this method struggles with large AI workloads, where the processing rules are more complex and vary more often.
Instead of storing full data flows, Gigaflow stores smaller, shared rule segments that are used by many different flows.
This approach, called “pipeline-aware locality,” looks at the sequence of rules used to process traffic and keeps the most commonly used rules easy to access. As a result, the system uses fewer memory spaces while handling a much wider range of rules—450 times more than existing solutions.
The key innovation is rethinking how caching works. Traditionally, systems only consider time-based or space-based memory storage. Gigaflow challenges that thinking by creating a new kind of memory management that matches today’s changing technology.
This improvement could have a big impact, especially as data centers deal with increasing demand from large language models (LLMs) and AI inference tasks.
The team now plans to explore how this method can help key-value (KV) caching, a key part of how AI systems store and retrieve information.
Other researchers from Purdue University, Feldera Inc., and Politecnico di Milano also contributed to the study.
