Imagine taking a short video of your kitchen with your phone, and then being able to walk through a detailed 3D version of it—opening drawers, moving objects, and exploring the space as if you were really there.
Thanks to new AI research from Cornell University and its partners, that futuristic idea is now becoming possible.
A team led by Professor Wei-Chiu Ma has developed a new system called DRAWER, which can turn an ordinary video of a room into a fully interactive digital model.
This “digital twin” is more than just a visual copy—it allows users to open cabinets, move objects, and explore the space in a realistic way.
The technology could be useful for everything from creating more lifelike video games to training robots to perform tasks in real homes.
What makes this method special is how easy it is to use. You don’t need fancy equipment or special sensors.
Just take a casual video on a smartphone—no need to open drawers or touch objects—and the AI does the rest.
The DRAWER system analyzes the footage and creates an interactive 3D scene.
To build this digital room, the researchers combined several types of AI. One model focuses on creating realistic-looking images, while another ensures accurate shapes and sizes.
A third model figures out which parts of the scene can move, like doors or drawers, and how they should move. A final step fills in the parts of the room that the camera didn’t capture, such as what’s inside a cabinet.
Graduate student Hongchi Xia, who helped lead the project, said the challenge wasn’t just building each AI component, but making them all work together. After putting the pieces into a single system, the team tested it by recreating a kitchen, a bathroom, and an office.
The results can even be used in video game engines. In one demo, users could shoot virtual balls to knock over objects in the kitchen, like the kettle or soap bottle. The team also used the digital room to train a robotic arm virtually.
Once trained, the real robot was able to perform the same task—putting an object away in a drawer—in the physical world.
In the future, you might be able to upload a video of your home and have a robot trained on that digital twin before it even arrives at your door. Although DRAWER currently works only with hard objects, the researchers plan to expand it to include soft materials and larger spaces.
Their long-term dream? To create digital twins of everything in the world.
