Smart replies & predictive assistance for Microsoft 365
How the simple goal of giving your thumbs a rest transformed messaging products throughout Microsoft
Writing messages is tedious.
Not once have I typed a message on my phone and thought, "Hey, this is way easier than just saying it!"
Ever since smart phones introduced the virtual keyboard, we've been trying to figure out ways to make composing messages less painful, especially if you have big fat thumbs like I do.
What if we could use AI & machine learning to provide accurate, contextual assistance before you typed a message?
My role
As part of the Cortana team in 2016, I was given the opportunity to lead design efforts for such an experience within Microsoft's relaunch of Skype.
I was the sole UX designer for Cortana, partnering with the Skype team in Palo Alto & the Microsoft data science team. My partners and stakeholders were a multitude of designers, PMs, engineers, and researchers split between Seattle and Silicon Valley.
What opportunities exist for us?
There are 4 primary use cases we believe natural language assistance is well suited for:
Information gathering
"What's the weather today?"
Task
completion
"Order a pizza from Dominos"
Transforming content
"Share this with Jen"
Quick
capture
"Remind me to do my laundry later"
We're still in uncharted territory
At this point in time, there wasn't much in the way of a competitive landscape, aside from clumsy chatbots. The closest comparison we had to draw upon was Google's newly launched Allo messaging app with Google Assistant.
We also wished to avoid drawing any comparisons to
our own unemployed office assistant.
How does assistance work in Allo? In a conversation, the user would invoke Google via @mention to act as a participant, performing non-personalized search queries.
How can we differentiate through product philosophy?
We felt there was a big opportunity to align our experience with Microsoft's principles for AI, specifically by finding ways to celebrate our users as the hero.
Therefore, we wanted to ensure Cortana's presence never steals the spotlight. Suggestions from Cortana should be personalized for you, providing control over what you choose to share and always in the service of enhancing your conversation.
Defining our hero scenarios
Providing assistance within a chat platform is a lot different than being on a speaker or phone. You have to tailor scenarios for contexts that make sense for the given medium while preserving the metaphor of a natural language assistant.
Partnering with research, we identified several core use cases that addressed the most pressing unmet user needs.
Phrase
suggestions
To-Dos & Reminders
Restaurant suggestions
Movie
showtimes
So how does it work?
In a nutshell, we determine suggestions by training a neural net based off the most common responses made in millions of conversations.
We then leverage implicit & explicit feedback to hone the accuracy of our model's responses over time, as well as personalize responses for each user based off of their own usage.
What if you get it wrong?
Confidently understanding user intent is the key to surfacing suggested actions successfully. The catch is, we're not always super confident.
Designing a system that accounts for being wrong isn't just important, it's essential. Every time you get it wrong, you erode trust. Even worse, if you don't give people the opportunity to tell you you're wrong, you're not growing.
"Perhaps the most important aspect of designing for AI is crafting language that helps people understand how conclusions are being made on their behalf."
Challenging our assumptions
One of the most important crossroads we faced in this process was determining the ideal interaction model for how we presented in-context suggestions. We specifically had 3 key questions we wanted to answer in testing:
-
Is the design unobtrusive, yet noticeable?
-
Does the user understand why they were getting suggestions?
-
Does the user find the solution more convenient than the alternative?
With that in mind, we took options to the lab for a 3 day usability study with participants engaging in simulated conversations over the course of a 60 minute session.
4 design options for presenting suggestions to the user
Finding our bearings
A key outcome from our initial study was that option #4 was the clear-cut winner. Users felt having in-context suggestions within the negative space between the keyboard and chat was unobtrusive as well as familiar, given the proximity to other inputs.
Additionally, participants found our four suggestion categories to be "extremely valuable" and indicated they were likely to use them if available in-product.
Hey Cortana, chill.
One design principle we adhere to is that any solution Cortana offers must be faster and simpler than the alternative. Of course, making something simple isn't necessarily a simple task.
One problem we encountered early in internal builds was over-firing. Users would see suggested actions so frequently, they began to ignore them. When we looked closer at the data, we determined a need for stricter rules governing how and when Cortana offers help.
In the end, we learned that broad intent inference simply wasn't enough. Requiring a time condition from our extraction lead to a lower firing rate but dramatically improved our user's perception of accuracy.
Get in and get out
Representing the product truth of being faster and simpler also means being able to complete a task quickly and exiting the stage. Eventually we refined our flows to a point where a user could create a reminder in 2 steps without leaving their conversation.
In the case of movie and restaurant suggestions, we designed a system that allowed the user to curate content before adding to chat as an attachment.
The finished product
Phrase suggestions & movie showtimes
Perhaps the biggest change made to our movie scenario prior to launch was narrowing scope to a single movie suggestion, rather than a carousel of what was in theaters. This change aligned with our goal to only fire when we were highly confident in what the user wanted.
Reminders & To Dos
This flow remains largely unchanged; it relies on common controls found within device operating systems and the user has the choice to manage their reminders in a private 1 on 1 conversation with Cortana.
1 on 1 assistance with Cortana
Unfortunately, what ended up taking the most amount of engineering work resulted in least amount of value for users.
While a tremendous proof of concept in how a full-fledged agent could operate within the confines of a conversational canvas, the reality is that nobody wants to go out of their way to interact with an assistant exclusively via typing.
The aftermath
The funny thing with some projects is that you never know where it will take you, even after you think it's over. The lessons learned from our Skype integration ended up proliferating to numerous other projects, both within Cortana and Microsoft as a whole.
The adoption rate of smart suggestions was considered a big success. So much so that the system we patented is now finding its way into Outlook, Linkedin, and Teams.
Outlook
Teams
As this project shipped, we were eager to leverage what we learned and apply it to future Microsoft 365 experiences.
If you'd like to learn more, feel free to take a look at my case studies on Conversational UI or Suggested Tasks.
