Imagine for a moment that you’re sitting on a sofa, controller in hand, guiding a character through a lush digital jungle. You push the thumbstick to the left, tap a button to leap over a fallen log, and squeeze the trigger just in time to grab a ledge. To you, it’s just a game. To a small but growing group of AI researchers, those mundane finger movements—the twitches, taps, and presses that feel so natural—are something far more valuable. They’re a hidden treasure trove of information. A British startup called Worldmodeldata is betting that the simple sequences of actions players make in video games can help teach artificial intelligence how to move through the real world. The idea is both elegant and a little strange: instead of relying solely on text, which has powered the current wave of large language models, why not train AI on the rich, physical interactions that happen in virtual spaces? Every time a player navigates a digital environment, they’re unknowingly generating a kind of demonstration—a lesson in cause and effect, action and consequence—that machines could someday use to drive cars, operate robots, or handle objects with human-like precision. It’s a concept that flips the usual conversation about AI upside down, suggesting that the path to smarter machines might be paved not with more words, but with more play.
The growing excitement around this idea stems from a real limitation in today’s most famous AI systems. Large language models—the engines behind chatbots like ChatGPT—are trained almost exclusively on written text. They’ve become incredibly good at generating sentences, summarizing documents, and holding conversations. But they don’t truly understand the physical world. They’ve never held a cup, walked across a room, or felt the resistance of a heavy door. Ask them to plan a route through a busy street or coordinate a robotic arm to pick up a fragile object, and their lack of embodied experience becomes painfully obvious. That’s why a number of celebrated researchers, including Stanford’s Fei-Fei Li and Meta’s Yann LeCun, have turned their attention to a different kind of AI: world models. These are systems designed to grasp the physics, geometry, and dynamics of real environments. To do that, they need more than text. They need video footage paired with action data—details like how hard to grip something, how fast to rotate it, and how much force to apply. This kind of data is essential, but it’s also incredibly scarce. As Xiatian Zhu, an AI professor at the University of Surrey, points out, the internet is overflowing with words, but it has very little of the cause-and-consequence data that world models crave. Without a massive dataset of physical interactions, progress in robotics and autonomous systems will remain slow. That’s the gap Worldmodeldata wants to fill.
The problem of data scarcity is one of the biggest bottlenecks in AI research today. Some labs have tried to create their own datasets by attaching sensors to humans and robots in controlled environments. A person might demonstrate a pick-and-place task over and over again, while a robot records every movement. But this approach has serious limits. It generates only a trickle of data compared to the ocean of text available for language models. And more importantly, it fails to capture the messy, unpredictable nature of the real world. Life is full of edge cases—the sudden gust of wind, the oddly shaped package, the cluttered tabletop that wasn’t in the training plan. As Nicole Fraenkel, a partner at venture capital firm Khosla Ventures, explains, repetition alone won’t help a machine deal with the disorder it’s likely to encounter outside the lab. The cost of getting it wrong in the real world is high. A mistake in a factory, a drone flight, or an autonomous vehicle can damage expensive equipment or even endanger lives. So researchers need training data that includes rare and unusual scenarios, not just the boring, repetitive ones. That’s where video games come in. Modern games are astonishingly realistic and varied. They feature complex 3D worlds with shifting light, moving obstacles, weather effects, traffic, crowds, and physics that mimic reality. And every second a player spends in these worlds, they’re producing a stream of action data—every button press and thumbstick movement is paired with what happens on screen. For a world model, that’s exactly the kind of material it needs: visual input combined with the commands that caused changes in the environment.
Worldmodeldata’s pitch is simple. Instead of each AI lab trying to negotiate individual deals with hundreds of game studios, the startup acts as a broker. It curates, organizes, and packages game controller data into clean training datasets. The company’s CEO, Rhea Loucas, argues that video games offer an almost endless supply of diverse experiences. There are millions of games, and they’re becoming increasingly similar to the real world in terms of graphics and physics. So why not use that rich, abundant resource to teach AI? The startup says it has already licensed nearly a million hours’ worth of data from studios behind popular games, though Loucas has declined to name them. That’s a staggering amount of gameplay. To put it in perspective, one million hours would take more than a hundred years of continuous playing to record. By packaging this data, Worldmodeldata hopes to save AI labs from the headache of building their own datasets from scratch. It also hopes to create new revenue streams for game developers, who might otherwise never realize the value of the data their players generate. In the future, the company even wants to explore ways to compensate individual players for their contributions, similar to how people might be paid for helping train other kinds of AI systems. It’s a vision that turns play into something more than entertainment—it becomes a kind of labor, a source of knowledge that could help build the intelligent machines of tomorrow.
The emphasis on corner cases is one of the most compelling parts of this story. In AI development, edge cases are the rare events that can trip up even the most advanced systems. A self-driving car might handle a sunny day perfectly, but what about a sudden downpour or a child darting into the road? A robotic arm might be great at picking up identical boxes, but what about an oddly shaped tool or a slippery surface? These are the situations where mistakes are most dangerous, and they’re also the hardest to capture in staged training environments. Video games, by their nature, are designed to be unpredictable and challenging. They feature crumbling platforms, enemy attacks, shifting terrain, and physics puzzles that force players to adapt. This means the data generated by players is full of the very scenarios that world models need to learn from. Fraenkel makes a striking point: the corner cases are the ones that really matter. The cost of error with a car, plane, drone, factory forklift, or autonomous quadruped is very high. If game data can help AI anticipate and handle these rare but critical moments, it could be a game-changer. And while the hypothesis that more game data will lead to better world models hasn’t been fully proven, researchers are generally optimistic. They believe that, like large language models, world models will improve as their training datasets grow. If that holds true, then unlocking the vast reserves of video game data could be one of the most important steps toward truly intelligent, physically aware AI.
Of course, there are still big questions and challenges ahead. Is data from video games really a reliable substitute for real-world experience? A game is, after all, a simulation. It follows its own rules and approximations of physics, which may not perfectly match reality. A robotic arm trained on a game might learn to grip a virtual object in a way that doesn’t translate to the physical world. Some researchers worry about these differences, and it’s a legitimate concern. But the general sentiment is that this kind of data is still hugely valuable as a starting point. It can help AI learn the basics of navigation, coordination, and cause-and-effect before fine-tuning on smaller amounts of real-world data. And as game technology continues to improve, with photorealistic graphics and increasingly sophisticated physics engines, the gap between virtual and real environments will only narrow. Worldmodeldata’s approach—acting as a middleman between the gaming world and the AI world—reflects a broader trend in the industry. As data becomes the most precious resource in AI, unexpected sources of that data are being discovered everywhere. Who would have thought that the hours spent gaming by millions of people around the world could one day help machines learn to move, see, and act? It’s a reminder that intelligence doesn’t just come from books and text. It comes from interaction, exploration, and play. And in that sense, the humble video game controller might be holding the key to a new frontier in artificial intelligence—one where machines don’t just understand words, but understand the world itself.
