Apple’s Patents Point to 3-D Gesture Control for Apple TV
A growing number of patents and patent reassignments point to the possibility of 3-D gesture control for a future generation of the Apple TV, the set-top box which users could eventually control with hand and arm gestures detected by the technology developed by PrimeSense, a 3-D sensing firm that Apple acquired in 2013.
Apple Insider reports that a month after it received a patent reassignment from its acquisition of Israel-based 3-D mapping firm PrimeSense, Apple was granted another patent that details a software user interface that could be used with PrimeSense’s motion-sensing hardware. Patent no. 8,933,876, for “Three dimensional user interface session control,” describes a system that in practice would consist of a motion and depth-sensing sensor outlined in a previous patent and software that would create a three-dimensional, non-tactile user interface. The system would be able to recognize a user’s gestures, and “translate” them into computer commands.
The invention applies the tracking, motion, and depth input data collected by the camera and sensor setup to a 3-D user interface. When a user moves his or her hand, the motion is recognized in a 3-D coordinate plane along X, Y, and Z axes. The gesture is then processed by a computer or app as a system command, which would enable the user to use a rising gesture, for example, to unlock the system’s user interface. As Apple Insider notes, the patent explains that the device moves among three distinct operating states, operating in any combination of locked/unlocked, tracked/not tracked, and active/inactive.
While locked, the user interface would ignore all gestures except for an unlock gesture. When unlocked, gestures can interact with the user interface. Tracking refers to the user interface focusing on a body part of an individual in order for the user to interact with the user interface, so that when the interface is in the tracked operating state, the user interface can interpret gestures from that body part (such as a hand).
While the device is in the not tracked state, the interface isn’t focus on an individual or his or her body part. The device is active when the user interface is unlocked and tracking a user, and able to accept gestures from the user. When the user disengages from the interface, the interface is inactive.
Users can alert the system to an upcoming unlock command with a “focus gesture,” which would bring it out of a not tracked and inactive state to a locked but tracking and active state. A focus gesture could be a push or a wave that would activate the user interface. The system would similarly enable users to drop sessions, effectively canceling the user input that would unlock the system and activate motion tracking by dropping their hand out of the hardware’s view.
The patent notes that the gestures a user makes must be within the sensor’s field of view, a requirement that could detract from the user experience if implemented incorrectly. The patent suggests that an onboard light array could illuminate when a user is within the device’s field of view, providing a visual cue to the user when the system is ready for his or her input.
The user interface patent follows the December reassignment of a PrimeSense patent that detailed a 3-D mapping and motion-tracking device. As Tech Cheat Sheet reported in December, Apple recently won a patent for touch-free, motion-sensing gestures within a 3-D iPhone user interface, which would use onboard sensors to recognize and interpret gestures made above a touch sensitive display.
The patent coincided with rumors that Apple was working on the technology for a glasses-free 3-D iPhone display, and also with the first patent reassignment from Apple’s acquisition of PrimeSense. That patent, for a “Lens array projector,” detailed methods of light projection for 3-D mapping. The patent outlined the optical components of PrimeSense’s “light coding” technology, and details a system for creating uniform light with an optical projection system. That enables the system to more accurately determine a scene’s spatial information.
As Patently Apple notes, other recent patents point to the possibility of a 3-D camera for future Apple devices, and detail the 3-D scanning technology that Apple has inherited from PrimeSense. Patently Apple notes that PrimeSense developed a 3-D camera module, and another that could double as a projector, and such technology could show up on Apple devices in the near future.
While it’s unclear exactly how Apple plans to implement PrimeSense’s technology, the patents and recent rumors indicate that Apple could be looking to implement gesture-based control for the Apple TV set-top box, or even in mobile devices like the iPhone. It’s been rumored that the next generation of Apple TV will feature a 3-D mapping system to enable gesture control, much like Microsoft’s Kinect system. Additionally, job listings posted by Apple have hinted at possible plans to include a camera (or cameras) in a future iteration of the Apple TV — a device that has reportedly been stalled due to disagreements over licensing with content providers.
Apple acquired PrimeSense in a deal rumored to be worth between $345 million and $360 million. The company was founded in 2005 and developed silicon and middleware for motion-sensing and 3-D scanning applications, and Microsoft famously licensed its infrared motion-tracking and depth-sensing chip for the Kinect sensor in the Xbox 360.