History will rate the digital personal assistant as an important landmark – not because there isn’t a lot of improvement to be made to the services offered by Siri, Cortana and Google Now, but because the apps are a glimpse of the entirely new mobile user experience of the future – one driven by voice, gesture and even virtual reality, not touch and type; and with fully context aware information delivered by massive artificial intelligence engines in the cloud.
This approach will eventually undermine the traditional search process on which Google has built its business, as well as revolutionizing the way we interact with the web. The traditional browser will morph and disappear – Microsoft is pointing to some of those developments with Project Spartan, especially with the full integration of Cortana. And just as Microsoft is grasping the opportunity of a shift away from the Google-dominated web model, hoping to push Windows 10 into the gap with features like AI and holographics, so Google itself is moving to evolve and defend its own platform.
It acknowledged, as soon as Siri was launched, that such products would be a rising threat to its search box and all the services that rely on it. It has responded with huge investments in the AI and deep learning systems that will power Google Now and future applications, as well as making acquisitions and R&D moves in many technologies which could underpin future web experiences – robots, virtual reality and so on.
And its Physical Web experimental project aims to define how we search and interact with the internet of things, repeating the trick Google pulled off so convincingly in first generation web search. A primary aim is to interact with connected objects without the need to install applications for each purpose. “Our core premise is that you should be able to walk up to any ‘smart’ physical object (eg a vending machine, a poster, a toy, a bus stop, a rental car) and interact with it without first downloading an app,” says the document introducing Physical Web to the open source community. “The user experience of smart objects should be much like links in a web browser: ie just tap and use.”
The role of the digital assistant in this trajectory is significant because these are real apps, which are experiencing significant uptake and so preparing consumers and developers for the platforms of the future. No wonder there is such a battle between these products, which are highly strategic to their respective creators’ bids to dominate the next generation mobile and web user interface.
However, there is a challenger to the big three, emerging from the open source world and so with the potential to be disruptive to the cosy smartphone-based trio. Researchers from the University of Michigan have presented an open source voice-analyzing app called Intelligent Personal Assistant (IPA).
The IPA might be open source on the software side of things, but the hardware requirements needed in the centralized cloud installations mean that ‘open source’ in this case will not be synonymous with cheap. An IPA system listens to a speaker’s command or question, before piping the data up to a cloud server for analysis. The results of that query are then piped back down to the device, so that action can be taken. The latency between the cloud and the device is still the source of most complaints – and the trade-off between a timely, mostly-correct response versus a thorough but time-intensive answer is a fine balancing act in the UX.
This is why the first generation assistants are not themselves particularly intelligent, merely acting as conduits to massive cloud installations that reference phonetic markers, crunch numbers, and produce the best guess at the desired answer. There are physical limits to trying to push this functionality into the handsets themselves, as they would need a lot of storage to hold a database and a lot of processing power to adequately search it. While cloud is slow, it is the only realistic way to offer this sort of user experience on a smartphone.
The researchers also investigated the steps that could be taken on the server-side implementation to speed up this process, and went as far as recommending silicon approaches. “We show that GPU and FPGA-accelerated servers can improve the query latency on average by 10 to 16 times. Leveraging the latency reduction, the GPU and FPGA approach can reduce the total cost of ownership by between 1.4 and 2.6 times,” they said in their documents.
In addition, the researchers say that the GPU and FPGA approach is entirely necessary because the data centers of today were not designed to handle these sorts of processes. The team said that the work of the IPA system is about 100 times as intensive as serving up web pages, and that the current data center architecture cannot scale to accommodate this workload given its current level of power consumption and cost – what the team calls the “looming scalability gap”. It says that, because of that gap, “there has been significant interest in both academia and industry to leverage hardware acceleration in data centers using various platforms such as GPUs, many-core co-processors, and FPGAs, to achieve high performance and energy efficiency.”
But this horsepower, and the expense that comes with it, is necessary to achieve a suitable user experience for these IPAs. With voice and image recognition capabilities needing to be merged with contextual awareness (so that a user can ask the IPA what it is their phone is seeing through a camera, or knowing that after identifying a landmark based on an image’s URL, the user may want additional information on the topic), sizeable investments or licensing needs to be made to ensure that the required server capacity is available to the users. Bandwidth isn’t the problem here but latency is.
An open source approach from Sirius could open up a door to those who lack the R&D departments necessary to homebrew an IPA, but deep pockets will be required to rent space in a public cloud or to build a specialized hardware private cloud installation. But this is a heavy-duty research project with powerful backing and so is likely to turn up in major platforms even if it does not survive as a standalone product.
The team’s paper thanks US defense R&D arm DARPA, a frequent and fairly unsurprising presence in the white paper scene; and ARM, which has a strong interest in anything that stimulates use of smartphones and of GPUs. But it also thanks Google for its partial sponsorship, without specifying which division of the technology colossus was responsible for the funding and support. That’s the joy of open source – if someone comes along and does a better job of it than you, there’s nothing stopping you adopting their approach. It wouldn’t surprise us at all if parts of Sirius crop up in Google Now, or even Siri or Cortana.
For now, while Apple’s and Google’s implementations dominate the marketplace, Cortana may be positioned to surge to similar levels of deployment when it comes to Windows 10 PCs and to Xboxes – with reports that Cortana will soon find her way into iOS and Android too, supported by an AI project codenamed Einstein.
According to Reuters reports, Microsoft is planning to port Cortana to iOS and Android, even though some of its appeal lies in its tight integration with Windows features such as contact books. However, in W10 it is likely to be a cloud-based service, and could more easily target multiple client OSs. Indeed, back in November, Julie Larson-Green, Microsoft’s chief experience officer, told reporters that she did expect that Cortana would be made available on other platforms.
Reuters says that Microsoft is working on next generation of Cortana which harnesses research from the Einstein effort. Eric Horvitz, managing director of Microsoft Research, told the news agency that the project was focused on technology “which can read and understand email” and will “play a central role in the next roll-out of Cortana” in the autumn.
Cortana – which many reviewers rate more highly than Siri for its usability and the richness of its AI-driven results – was developed for Windows Phone and is one of many elements of the mobile OS which have been transferred to W10. The digital assistant could be expected to be closely guarded by Microsoft as a reason for Windows users to stay faithful, and to attract new customers to its platform. It has even greater strategic importance as a possible tool to weaken the hold of the web experience dominated by Google.
Perhaps the latter objective – to undermine Google – is even more important to Microsoft than differentiating W10. Cortana will be embedded deeply into the new W10 browser, Spartan, which aims to erase the Internet Explorer name, with all its bad connotations, and take on Chrome and Firefox. The digital assistant will “show up right at the moment when you need to get things done”, according to the vendor (hopefully less intrusively than the notorious Microsoft paperclip). However, any new browser, no matter how slick and modern, will have a tough time competing with Chrome, which has evolved to be genuinely effective across all platforms. While Spartan is intended to lure users to Windows 10, it will surely need to evolve to be as multiplatform as Google’s offering, taking Cortana with it.
While being multiplatform necessitates Windows living in a more friendly fashion alongside Android and iOS devices, Microsoft will also hope it will help it steal Android users – or at least some of their usage. Currently, large numbers of users have multiple OSs, most commonly an Android handset and Windows PC. Microsoft cannot hope to take massive handset share away from the dominant OSs, but it must hang onto its position in PCs and the devices which will replace them, such as tablets and hybrids.