The competition between Apple (NASDAQ:AAPL) and Google (NASDAQ:GOOG) encompasses a huge number of features in Android and iOS, but one that many consumers use everyday is each operating system’s voice assistant. Piper Jaffray analyst Gene Munster compared Apple’s Siri and Google Now. He tested each with 800 queries and tracked their ability to recognize and answer speech requests. Munster tested each assistant in both a controlled environment and an uncontrolled environment, and measured the improvements that both Apple and Google have made in enabling Siri and Google Now to correctly interpret and answer questions.
In a similar test last December, the voice assistants came in tied with a grade of C+. The winner this time? Google Now, which was able to interpret and answer questions slightly more accurately than Siri, and earned a B to Siri’s B-. Google Now was able to answer questions correctly 84 percent of the time, while Siri was able to answer them correctly 82 percent of the time.
One of Munster’s measures of Google Now’s improvement was its progress in the rate of its ability to correctly answer questions that it’s correctly heard. In December 2012, Google Now was only able to correctly answer questions correctly heard 72 percent of the time. In December 2013, that rate had climbed to 81 percent, and it’s since risen to 86 percent. Its overall ability to answer questions rose from 79 percent in December 2013 to 84 percent in July 2014, compared to Siri’s improvement from 79 percent to 82 percent over the same period. Munster’s grades — B for Google Now and B- for Siri — seems largely predicated on those figures.
The report notes that Google Now’s strong points were questions related to Navigation, Local, and Information, three of the five categories of questions that were asked of each assistant during the tests. One of its weak points is another category, OS commands, such as “Play song by the Beatles.” The report points out that that area of weakness contrasts with Apple’s Siri. “Siri enables full control of the music application via Siri, but Google Now does not understand all song change/pause commands.” However, Piper Jaffray notes that it expects Google Now to improve its OS Command performance as it releases voice-compatible devices such as Google Glass, Moto X, and Moto G.
The report also outlines the differences between the sorting functions available to users after they’ve asked a question and been provided with an answer. “Google Now provides the option to filter results after posing the query (around price, location, rating and hours), but Siri does it automatically based on key words. However, Siri does not sort by price or hours (ie: if asked ‘Where is the cheapest place to buy shoes?’ or ‘Which bike shop is opened the latest?’)”
Munster considers Siri’s automatic sorting of results based on distance or rating and advantage. That capability is also predicated on Siri’s detection of keywords. “If the user requests a ‘good pizza place’, Siri will compile a list of local pizzerias and sort them based on Yelp ratings. On the other hand, if simply asked for a ‘pizza place’, or a ‘close pizza place’ Siri will sort results based on distance from user.”
In a separate note, Munster reported that Apple continues to make improvements to Siri even though it fell slightly behind Google Now in the latest round of tests. (And Munster attributes a slight decline in both Siri’s and Google Now’s ability to correctly hear a question in an uncontrolled environment to unspecified survey error.) The biggest, “most noticeable” improvement made to Siri is its new ability to aggregate information from multiple data sources in answering a question. That feature enables Siri to provide more comprehensive answers to questions:
“For example, when asked, ‘Where is Mt St Helen?’, Siri provides both a map view of the area via Apple Maps as well as a Wikipedia blurb about the mountain. Siri further provides the option to tap ‘bing’ or ‘wolfram alpha’ to get more information.”
The change also sees Siri closing in on a traditional advantage of Google Now, which uses Google Search, Google Maps, and Google Play to answer questions. Siri also reduced its reliance on Google search, answering only 3 percent of questions using Google. During the tests, Siri only directed to Google when specifically asked to “Google” something, and otherwise used Microsoft’s (NASDAQ:MSFT) Bing as its default search engine. Additionally, Munster notes that the percentage of questions that Siri can answer without outside sources has increased.
“Apple Maps provided 33% of answers, up from 21% in Dec-13, a 59% increase. Siri reduced reliance on Yelp to answer navigation questions from 16% to 1%, directing the user to Apple Maps instead. Also notably, Siri was able to answer 4% of queries herself, up from <1% in Dec-13.” That’s a trend that Munster sees continuing as Apple works to get users to input queries to Siri instead of bypassing the assistant to go directly to Google or another search engine.
Munster explains that the methodology of the tests to compare Siri and Google Now was structured as follows: each voice assistant is asked a total of 800 questions, 400 asked indoors and 400 asked outdoors. In the uncontrolled environment, the questions were asked above background noise kept between 60 and 80 decibels. The questions are evenly divided among five categories — Local, Commerce, Navigation, Information, and OS Commands. The answers are judged successful if they provide the “appropriate information,” or alternately, “anything containing the information without having to open additional links.” A key difference between Munster’s findings with Siri and Google Now were the differences in the assistants’ ability to answer questions correctly most of the time. “Siri was able to correctly identify 96% out of 400 indoor and 400 outdoor queries vs Google Now at 93%. Google Now, however, was able to correctly answer 86% of queries that were correctly heard vs Siri at 84%.”