In 2010, Microsoft unveiled a new Xbox 360 accessory it said would change the way gamers played. The Kinect was the first motion controlled gaming system that didn’t require a controller, instead allowing players to use their whole body to move the game.
While the Nintendo Wii popularized motion controls with its innovative Wii Remotes in 2006, Microsoft planned to take it a step further with the Kinect, originally dubbed Project Natal. The Kinect uses a complicated system of sensors, lasers and cameras to reflect a player’s gestures and actions on screen.
If you’ve ever played a Kinect, that first interaction seems so futuristic. Using your hand as the controller, swinging an imaginary light saber or trying to copy the dance moves of Justin Bieber are just some of the ways the Kinect allows us to play. But how does it work?
A Complicated Camera Setup
If you look at the Kinect, you’ll see that it has three lens areas. These help the Kinect “see” the room you play in. First, there is a regular RGB camera. It simply acts like a basic webcam, and records the room. The Kinect uses this camera any time it displays your image in the game, or for other functions like video chat. According to Microsoft documentation, it captures images around normal video speeds of 30 frames per second, and projects at a 640×480 resolution.
The more complicated camera relies on infrared light to work. One of the lenses on the Kinect is actually an IR emitter, which bathes the play space in light the camera can pick up. Don’t worry; infrared light waves are a totally safe type of radiation. The camera sees these waves as they bounce off people and objects in the room; the brighter the light, the closer the object is. Objects too close to the camera become a little too bright and hard to distinguish, which is why players have to stand more than four feet away from the Kinect sensor bar.
Before, gestural control systems like the Wii were based on the time of flight — how long something takes to return after bouncing off objects in the room. The camera in the Kinect encodes information in that light as it goes out, then measures that degradation over time with the sensor in the Kinect’s Infrared camera.
The data fed in by the camera is instantly processed by some really smart software. It identifies shapes that could be human by heads and limbs, so your Kinect doesn’t think your armchair or your cat are the ones trying to play Dance Central. This software already understands how a human body can move, so it knows your head can’t turn 360 degrees on your neck, and it captures movement through more than 48 points of articulation.
The Kinect software has been programmed with more than 200 possible poses, so it has an idea of where your body is probably going to go, too. This is especially important in two-player games where one player might move in front of the other, or a piece of low furniture might block the camera’s view of your legs. The Kinect software then pairs with the game software to see what pose you are supposed to be creating to do a certain task.
Some of the frustrations people can experience with Kinect games happen here, often because while they may be performing the right action, the camera doesn’t match it to what the game wants, or the game has a very specific parameter of what is correct.
The Kinect’s other much touted feature is its ability to recognize voice commands. The Kinect has four microphones in its sensor bar to pick up players’ voices, all pointed down to pick up soundwaves more effectively. This large amount of audio equipment allows the Kinect to hear people even when they are several feet away.
Voice commands require calibration so the Kinect knows the general audio levels of your living room. If you ever rearrange your furniture or add carpet, you’ll have to recalibrate because sound bounces around the room differently.
The Kinect still has its flaws, but as the first peripheral to offer motion controls without a controller as well as sophisticated voice controls for gaming, it’s the first step into a new realm for video games. Microsoft has also made the API available to the public, opening larger possibilities up to any developer who wants to explore motion controls.
BONUS: 7 Awesome Kinect Concepts