An extensive — and imaginative — patent application gives us a look inside one secretive startup’s ideas for its augmented reality system, and its vision for how augmented reality technology could change our everyday lives. As The Verge reports, it’s been about three months since Google led a $542 million round of funding for the secretive startup Magic Leap. But little was known about what Magic Leap was really working on, until its recent patent application was spotted. It was reported that the company had been working on augmented reality glasses that can create digital objects that appear to exist in the world around the wearer. Rather than displaying images on the glasses, or projecting them into the environment around the wearer, Magic Leap’s glasses reportedly project an image directly onto their wearer’s retinas, and tracks exactly where he or she is looking.
In addition to the funding, Android and Chrome leader Sundar Pichai corporate development joined Magic Leap’s board, and vice president Don Harrison became a board observer. The funding came from Google itself, not from an investment arm like Google Ventures. As The Wall Street Journal reported at the time, the deal valued Magic Leap at just under $2 billion, the amount that Facebook paid to acquire Oculus Rift, the maker of a virtual reality headset.
In an interview with The Wall Street Journal, Magic Leap founder and chief executive Rony Abovitz revealed relatively little about the startup, but did say that he sees Magic Leap as a new interface that could replace PC monitors and smartphone screens. Abovitz says that Magic Leap’s technology can project accurate images onto the eyes, and make it possible to see virtual 3-D objects as if they were part of the real world — tricking the mind into believing that virtual objects are part of the physical space.
Most virtual reality goggles use stereoscopic imaging, which involves showing two separate images to the eyes to create a sense of depth. The method is known to trigger nausea, especially when users experience a lag between the head’s movement and the rendering of the virtual space. Abovitz said that Magic Leap’s technology doesn’t cause nausea, and says it could find applications in movies and video games, create a new platform for patient and doctor interactions, or give rise to a new communication platform that pushes technologies like video chat to a new level, but relatively little was known about what Magic Leap was really working on.
1. Augmented reality wearables could include both glasses and other peripheral elements.
The total secrecy around Magic Leap’s vision for augmented reality changed with a patent application that The Verge spotted on the website of the U.S. Patent and Trademark Office, for a “Planar waveguide apparatus with diffraction element(s) and system employing same.” While many of the ideas involved in the wide-reaching application may never make it into the final version of Magic Leap’s technology, the application covers an array of possible uses for the proposed augmented reality headset. The most interesting — and the most easily understandable — parts of the patent are the many illustrations that show what Magic Leap is envisioning for its device and how it will be used. The figure above, for example, shows both a head-mounted augmented reality system and a component to be worn at the waist.
2. Gesture control will likely be a very important part of augmented reality systems.
The wearer could interact with the augmented reality system in a number of different ways. The figure above shows an assortment of possible finger commands, which could be used to control the system. The system could detect and map the user’s location, which would be uploaded to the cloud so that the system could constantly adjust based on the user’s movements. That would enable it to tailor all virtual content to the user’s environment.
3. User input could also be given and interpreted through objects called totems.
But where things get really interesting is when the patent application explains that another method detecting user input would rely on objects that the user can manipulate to give input to or interact with the augmented reality system. These objects, referred to as “totems,” could take the form of inanimate objects, like a piece of metal or plastic, a wall, or the surface of a table. Alternatively, other totems could take the form of animate objects, such as the hand of the user. Magic Leap explains that “the totems may not actually have any physical input structures (e.g., keys, triggers, joystick, trackball, rocker switch). Instead, the totem may simply provide a physical surface, and the AR system may render a user interface so as to appear to a user to be on one or more surfaces of the totem.”
4. Wearable totems could also be an option for user input.
Some totems could take the shape of devices that we already use to interact with the computers and devices we use today. the system could render the image of a computer keyboard and trackpad, which would appear “on a surface of a thin rectangular plate of aluminum which serves as a totem.” The plate itself wouldn’t have physical keys, a trackpad, or sensors, but the augmented reality system would be able to detect user manipulation or interaction with the plate as inputs made via the virtual keyboard and trackpad. A totem could take the form of a virtual keyboard, a mouse, a lotus structure, or more imaginatively, rings, a haptic glove, a pen, a paintbrush, or even a keychain or a charm bracelet.
5. Augmented reality systems could use a variety of user interfaces depending on the task at hand.
Magic Leap’s patent application provides an extensive array of illustrations of possible user interfaces through which the user could interact with the augmented reality system. The system would be able to render both floating user interfaces, which aren’t mapped to any surface, and mapped user interfaces, which appear to reside on a physical surface. The system could employ pupil tracking as a means of user navigation of virtual content. It would also recognize real-world activity — for instance, that the user is chopping or cooking in the illustration below — and float the user interface in relation to that activity.
6. Virtual icons could represent applications, functions, menus, and more.
Regarding a drawing that shows a circular menu rendered in the user’s field of view, the document explains, “The user interface rendered by the AR system may include an icon based virtual main menu, in addition to, or in place of the expanding radial menu. The AR system may additionally employ three-dimensional helix content management. The AR system may additionally employ pop up two-dimensional alert or notification virtual structures.” The user interface could show icons in a number of different arrangements, and the user could point at or “touch” one of these icons to select it. (Apparently, in the future, we’ll have augmented reality systems that are safe enough to use while driving.)
7. The user could activate the user interface with a gesture.
In an illustration showing another possible configuration of the augmented reality system’s user interface, the patent application notes that the user could call up the user interface with a gesture. “The user may, for example, perform a swipe gesture. The AR system detects the swipe gesture, and interprets the swipe gesture as an instruction to render the generally annular layout or configuration user interface. The AR system then renders the generally annular layout or configuration virtual user interface into the user’s field of view so as to appear to at least partially surround the user, spaced from the user at a distance that is within arm’s reach of the user.”
8. Augmented reality systems could render the controls you need over your hand.
In response to a gesture such as the user spreading his or her fingers apart, the augmented reality system could render a navigation menu to the user, so that it appears to hover or be attached to the user’s hand. The menu items could appear over the fingers (otters than the thumb), and the document notes that the thumb could be left free to serve as a pointer, which would enable the user to select the desired menu item or icon by touching the thumb to the appropriate portion of the corresponding finger.
As in the illustration below, another gesture, such as the user making a circular gesture in the palm of his hand with a finger of the opposite hand, would signal to the augmented reality system to scroll through the menu, and a number of fields could scroll from one finger to the next. New menu items would enter the field of view from one direction while others exit it from the other. The patent notes, “The direction of scrolling may correspond to a rotational direction of the finger in the palm. For example the fields may scroll in one direction in response to a clockwise rotation gesture and scroll in a second, opposite direction, in response to a counterclockwise rotation gesture. ”
9. Augmented reality could make the office a lot more interesting.
The figure above illustrates a user sitting in a physical office space and experiencing a virtual room or virtual office space — which is where the user would perform his job (virtually). As Magic Leap’s patent application explains, “the virtual office is populated with various virtual tools or applications useful in performing the user’s job. The virtual tools or applications may for example include various virtual objects or other virtual content, for instance two-dimensional drawings or schematics, two-dimensional images or photographs, and a three-dimensional architectural model.” (Some examples of tools include a ruler, caliper, compass, protractor, templates, stencils, etc.)
The virtual tools and applications could also include “interfaces for various software applications, for example interfaces for email, a Web browser, word processor software, presentation software, spreadsheet software, voicemail software, etc. Some of the virtual objects may be stacked or overlaid with respect to one another. The user may select a desired virtual object with a corresponding gesture. Based on the recognized gesture, the AR system may map the gesture, and recognize the command. ”
10. Entertainment could become much more interactive.
With an augmented reality system, your living room could become a virtual entertainment or media room. The figure above shows the use of hand gestures to interact with the user interface, and the augmented reality system could recognize gestures of the user’s arms or hands — or even spoken commands — to give the user access to “an icon based cluster user interface virtual construct.” That cluster would provide a visual representation of the virtual rooms or spaces that the user can access.
11. Decor could become changeable with the user’s mood.
The augmented reality system would enable the user to experience virtual decor, with the patent application explaining that the illustration above shows “the user executing pointing gestures to interact with a user interface virtual construct.”
12. Grocery shopping could get much more interesting with augmented reality.
The patent application includes a surprisingly large number of illustrations of a mother’s experience at the supermarket, as enhanced by the augmented reality system. (As The Verge points out, gender roles seem to be one of the few things for which Magic Leap hasn’t envisioned a more modern future.) She accesses information about foods, is presented with coupons, and uses a virtual grocery list as she navigates around the store.
The augmented reality system would also enable shoppers to watch demonstrations by celebrity chefs, who could be shown executing a recipe for which all of the ingredients would be accessible at the end cap. The patent application notes of the example that based on the user’s location in the store, the AR system would retrieve the information from the “passable world” associated with the chef’s live performance. Stores could show the performance live, enabling shoppers to ask questions in real time, and the system could use cameras pointed at the shoppers to provide the chef with a view of his audience.
13. Games could get much more engaging.
As the daughter in the patent application’s illustrations accompanies her mother through the supermarket, she plays an augmented reality game which encourages her to search for a monster (named Gerald) in the store. When she accomplishes that goal, an illustration shows the animated monster bursting from a shelf of boxes. The document notes that the child could animate the character once it’s found.
14. Virtual yoga classes or physical rehabilitation could become an option.
The augmented reality system could enable users to participate in a virtual yoga class — and perceive virtual friends also participating in the class. In another image, the patent application illustrates the patient riding a stationary bicycle, and the system rendering a virtual view of the simulated cycling route. The system could show the route’s map, altitude, and distance, and also display the user’s performance statistics. The augmented reality system could also render a “virtual avatar” as a motivational tool that would, for example, replicate a previous ride to enable the user to compete with his or her personal best performance.
15. Augmented reality could find many applications in health care.
Several illustrations from Magic Leap’s patent application show the augmented reality system being used in a health care setting, but one of the most interesting shows a team of surgeons using the system during a surgery. Before the surgery, the system could render a patient’s pre-mapped anatomy in 3-D for the team to analyze while planning the procedure. The application explains that “The AR system may render the anatomy using a light field, which allows viewing from any angle or orientation. For example, the surgeon could walk around the heart to see a back side thereof.”
During the planning process or during the procedure itself, the augmented reality system could render the patient’s information on the surface of a wall or a table, and users could request and retrieve specific patient data. During a meeting pre-procedure, the augmented reality system would enable the patient’s spouse to join the meeting virtually while at work. The surgeon would be able to reference the 3-D renderings of the patient’s anatomy during the surgery, and cameras would enable a medical student to observe remotely.
During a post-operative meeting with the patient, the surgeon would be able to explain how the procedure went with a view of the patient’s anatomy of a 3-D model of it. As the patient convalesces in a hospital room, the augmented reality system could create a relaxing environment, such as a beach setting.