How Siri for Mac Could Best the iOS Version
Have you ever wondered what it would be like to have access to Siri not only on your iPhone or iPad, but also on your Mac? You don’t have to wonder anymore: Apple (NASDAQ:AAPL) has filed a 92-page patent application that describes exactly what the company (currently) thinks a desktop version of Siri could be like.
A new patent application from Apple describing an “intelligent digital assistant in a desktop environment” was first spotted by Apple Insider, which reported that while the feature has some functionality that’s similar to what Siri can do on iOS, the digital assistant is also capable of performing more advanced system tasks, complete dictation, and even act as a “third hand” for when users need a little help multitasking.
“Just like human personal assistants, digital assistants or virtual assistants can perform requested tasks and provide requested advice, information, or services,” the patent notes. “Such digital assistants can interpret the user’s input to infer the user’s intent; translate the inferred intent into actionable tasks and parameters; execute operations or deploy services to perform the tasks; and produce outputs that are intelligible to the user. Ideally, the outputs produced by a digital assistant should fulfill the user’s intent expressed during the natural language interaction between the user and the digital assistant. “
When the user gives a voice command, Siri would be able to distinguish whether that person wants an informational answer or the performance of a task as a response. That’s especially important for the functionality of the feature, given that its array of capabilities and possible tasks is much wider, as the patent describes it, on Mac than its current version on iOS devices.
In an example that Apple gives in the patent application, when the user asks, “Where am I?” Siri knows, based on the user’s location, to answer, “You are in Central Park near the west gate.” Alternately, if the user requests, “Please invite my friends to my girlfriend’s birthday party next week,” Siri can acknowledge the request and then send a calendar invite to each of the user’s friends listed in the address book.
Since the assistant can also perform dictation, it’s equipped with the ability to distinguish between dictation input and commands based on differing voice commands, gestures, and other criteria. Apple explains:
“During performance of a requested task, the digital assistant sometimes interacts with the user in a continuous dialogue involving multiple exchanges of information over an extended period of time. There are numerous other ways of interacting with a digital assistant to request information or performance of various tasks. In addition to providing verbal responses and taking programmed actions, the digital assistant also provides responses in other visual or audio forms, e.g., as text, alerts, music, videos, animations, etc. In some embodiments, the digital assistant also receives some inputs and commands based on the past and present interactions between the user and the user interfaces provided on the user device, the underlying operating system, and/or other applications executing on the user device.”
So how would a hypothetical Siri for Mac feature work, and how would you interact with it? We step through the basics of what Apple’s outlined in the patent application, though of course, it remains to be seen if and when the technology will show up in a real product release.
How would you activate and deactivate Siri?
The patent describes several different ways that users can activate Siri. In some embodiments of the technology, users could invoke Siri from a dormant state by hitting the “home” button on the device or a dedicated “assistant” key on a keyboard. On a touchscreen, the user could also call up the digital assistant with a gesture, like tracing a circular motion, with a mouse, on a keyboard, or on a touchscreen, as the patent illustrates. Alternately, Siri could run in the background, listening for a phrase like the “Hey Siri” command that will activate the assistant in iOS 8. (But to save battery, the feature will work only when the iPhone or iPad is plugged in to a power source.)
The Siri for Mac feature would also look to conserve power, and the patent explains how in some embodiments, the open voice channel needed for the digital assistant to “listen” for a command to wake up would be a concern not only because it consumes power, but also because it might be inappropriate for use in some locations due to “noise or privacy reasons.” In those situations, a touch-based activation, like a button or gesture, is important.
In other embodiments detailed in the application, a Siri icon could reside in the dock and notify users that the assistant is ready for use, and could be activated for specific operations when the user drags and drops objects onto the Siri icon. The user can also speak commands and invoke Siri to help with tasks that are already begun.
When users no longer need Siri, they can put the digital assistant into standby with a voice command. “Go away now,” “Standby,” and “Good bye” are the examples given in the patent. The user could also tap on the icon or use a specific gesture to deactivate Siri, sending the assistant back to its home location on the dock, desktop menu bar, or another desktop location.
How would you interact with Siri?
Once the computer has launched or activated Siri, users can make simple voice commands, as on iOS devices, to complete tasks. The patent notes that while the desktop version of the digital assistant will still be able to operate in the same question-and-answer form with which current iOS users are familiar, Siri combined with the additional power of a full computer will be able to complete more advanced operations.
And unlike the mobile version of Siri, a desktop version would be able to use the context in which it was invoked to complete a task. In the illustration above, the user can give Siri a simple voice command — to mark an email as urgent — and the assistant will complete the task.
While the voice activation can in some places eliminate the need to use a physical tool (like a mouse or a keyboard, or a button in the case of the iPhone), at other times the voice input will supplement the input from a mouse or keyboard to enable users to complete more complex tasks. That enables functionality like using drag-and-drop actions to inform Siri what to do, giving users an efficient way to activate Siri to complete a specific task.
The desktop version of the digital assistant uses a “focus selector,” such as a cursor, to help Siri understand the exact context of a command. Users can hover their finger or a cursor over a folder or highlight an icon to indicate a file that they’d like Siri to copy, as an example. And when using a mouse, users can indicate a folder with contents that they’d like sorted by dragging and dropping the folder’s icon on top of the icon for Siri.
Multiple files and folders can be dragged and dropped to Siri for more advanced features, including “sorting, printing, comparing, merging, searching, grouping, faxing, compressing, uncompressing, etc.,” according to the patent. The process of competing these advanced tasks could proceed as a conversation, with repeated exchanges between the user and the digital assistant to identify the exact files to use and tasks to complete.
The document notes that users won’t have to verbally identify the files as the subject of the command. Instead, Siri interprets commands based on the context of the objects that they’ve targeted, the provided speech input, and the sets of operations that the assistant knows are available for each object.
What can you do with Siri?
Beyond the tasks that logically extend from the way that users can invoke and interact with Siri, Apple has also thought of ways that Siri can make common tasks easier. In the most interesting feature detailed in the patent, Apple describes how Siri can be used as a “third hand” to help the user complete multiple tasks at a time.
Based on a request from the user, Siri can either maintain “the ongoing performance of the first task on behalf of the user, while the user performs a second task on the user device using the first input device,” or alternately perform “performing the second task on the user device, while the user maintains the ongoing performance of the first task.”
What that means, as the patent explains in a section called “Using Digital Assistant as a Third Hand,” is that Siri can help relieve the “significant cognitive burden” placed on the user when he or she needs to switch among documents, objects, and applications. For example, if the user is working on a document in a text editor, he or she can get Siri to perform a Web search to find information that the user needs to access to complete the editing task.
Similarly, the user could ask Siri to find an image to insert into the document, and the digital assistant can use contextual clues like the name and text of the document to “disambiguate” the terms used in the voice command. So instead of switching in and out of windows, the user can maintain continuous attention on the document as Siri finds the necessary image.
Other possible functions are described in the patent, and in theory, Siri could be able to perform any tasks that can be completed by the user. While it’s not yet clear if Apple intends to actually roll out a version of the technology anytime soon, there have been rumors in the past that the company planned a desktop version of the digital assistant. The patent application is the first evidence that Apple is working on the technology.