Augmented reality (AR) devices such as smart glasses may soon become much smarter.
Researchers have developed a new technology that can predict where a person will look next, potentially allowing AR systems to prepare information and graphics before the user even turns their eyes.
The research was led by Fiona Ryan, a Ph.D. student in the School of Interactive Computing at the Georgia Institute of Technology.
She recently presented the work at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Today’s AR devices typically react to a user’s gaze after it happens. If you look at an object, the system responds by displaying information or graphics related to that object.
While this works, there is often a slight delay because the device is constantly trying to catch up with the user’s actions.
Ryan’s new approach aims to change that. Instead of simply reacting, the system predicts future eye movements and attention, giving the device a chance to prepare in advance.
“We want AR systems to anticipate what a person will interact with next and where they are likely to look,” Ryan explained. This could make virtual experiences feel smoother and more natural.
A key feature of the research is that it works in three dimensions. Previous gaze-prediction studies mainly focused on predicting where someone would look in a flat, two-dimensional image. However, people move through real-world environments that are three-dimensional and constantly changing.
Ryan’s system tracks a person’s attention as they move through space. Rather than predicting a single glance, it models a sequence of eye movements, known as a scanpath, showing how attention shifts from one object to another over time.
Much of the work was carried out during Ryan’s internship at Meta. She used data from Meta’s Aria Digital Twin dataset, which contains first-person videos of people performing everyday tasks in a realistic apartment setting. The dataset also includes detailed 3D reconstructions of the environment, making it possible to accurately track where users are looking.
In one demonstration, the software follows a user walking toward a table and picking up a cup. After the cup is picked up, the system correctly predicts where the user will look and move next.
This ability is based on how humans naturally focus their attention. People do not absorb every detail of a scene at once. Instead, they look at specific objects that are relevant to their goals. For example, someone picking up a cup will often look at the cup first and then at the place where they intend to put it.
The current system can predict gaze up to three seconds into the future on average and, in some situations, as much as 10 seconds ahead. Even a few seconds of warning could give AR devices enough time to load information, graphics, or interactive content before it is needed.
Beyond augmented reality, the technology could also help train robots. By understanding where people look while performing tasks, researchers may be able to teach robots to pay attention to the same important details and carry out tasks more effectively.
While the technology is still in its early stages, it offers a glimpse into a future where smart glasses and other devices can anticipate our actions, making digital experiences faster, more responsive, and more intuitive.
