January 30, 2017

How can A.I. help parents raise great kids?

Introducing Avo, the family robot

In a Voltron move, our business unit (and studio) had joined forces with Microsoft Research to form the Microsoft Artificial Intelligence + Research Group. The company was making great strides in voice synthesis and recognition, image recognition and labeling, conversational interfaces, and a whole lot more. With this momentum, our senior leadership wanted to explore new ways to leverage these advances. We typically focussed on productivity and “on-the-go” scenarios in our line of work. But for this design sprint, we were tasked with how A.I. products could empower people’s lives in their homes.

COO Kevin Turner leaves Microsoft in July to become CEO at Citadel Securities, Qi Lu leaves Microsoft to recover from a bike accident, Satya Nadella merges the largest tech research team in the world with Bing, Cortana, etc. to create an A.I. Voltron…Microsoft Artificial Intelligence and Research Group is born, Basically, a Voltron scenario.

Collage by MrBM

Sprint format

For the kickoff meeting, our design and research leadership spent some time defining resources and expectations—schedule, timing, deliverables, etc. They also shared high-level research findings to better understand where we stood today in order to design experiences for tomorrow.

We were then divided up into four teams of five, each team having four designers and one researcher. So, twenty of us on the hook for delivering the future. The sprint would span three weeks with incremental check-in and feedback sessions on Fridays. And since the normal product work kept trucking along, each person was also given a suggested percentage of their time to dedicate to the 3-week sprint (some more than others).

My first take

My original list (no bad ideas, right?)

Identify malware; security/privacy assistant
Add sound and/or color to photos and videos. (Silent film enhancer?)
Auto-translate films into my language
Summarize complex and long texts (Legal docs, prevent the news filter bubble, news, research, plagiarism detection)
Auto-generate film trailers or YouTube clips based on existing patterns of “how to make a good trailer”
Preserve dying languages (reading texts and audio interacting)
Going deeper with existing image labeling tech; not “this is a cat” but “this is a cat on a couch in the evening in Seattle around 1987”
Estimate value of car, 401k, house, or item along timeline
Creativity: what happens when you randomly throw two datasets together and allow ML to find patterns?
Better notification platform (devices need to be aware of each other and user behavior patterns)
Better contextual awareness in mapping scenarios (“turn left after the fire station”) Also discussed: other layer awareness – “entering the Tolt River watershed”, “west through Wallingford”, etc.
Creativity: throw tons of images,video, and text—have A.I. compose a story, film, etc.
A.I. Critic: “This film is good because of X, Y, and Z”

The initial brainstorm was almost like a palette cleanser, shaking the A.I. tree to expose all the little things stuck in my brain. At the time, I was watching a lot of [that handsome, know-it-all guy] Simon Sinek videos and his “Golden Circle” graphic kept popping into my head. It’s actually a terrific proof within any design process—figure out (and write down) the why before you jump to the how you use it or what it is. Pretty basic stuff, but you’d be surprised how many folks skip straight to surfacing a newfound widget front and center…just because there’s a new widget.

The other thing I kept visualizing was a lesson learned from working on Cortana for a a few years. Well, less about an A.I. Assistant per se, more about trying to convince someone to login and share their behavior with us. ML/AI scenarios feed off of data, much of it is a user’s personal data. It’d be critical to “show” the user how the product delivers value by leveraging their personal data.

So, a lot of the “palette cleansers” got cut pretty quickly.

Step 2: Core value proposition: Augmenting parents

After aggregating everyone’s lists, an interesting pattern began to emerge. Lifecycle-specific use cases kept coming in. Whether it was helping Grandma remember to take her medicine or help Timmy with his stutter. Then there was the question of having an array of robots with very specific functions or a do-it-all, much like Rosie from the Jetsons. Would the robot belong to the parents, the kids, or the whole family? Would the robot enable lazy parents to not raise their kids properly? Would the kids hack it for nefarious reasons? The back-and-forth discussions were truly fascinating!

Helping parents help their families

We decided not to be the family pet or babysitter, nor the one-trick pony stuck in the kitchen. Instead, we began to think of a household A.I. entity as a bridge between all the knowledge of the internet and busy, multi-tasking parents. The robot would also have an intimate context of everyone’s schedules and the age-related context of children’s developmental sensitivities. The robot would also need to have a ubiquitous presence, much like a home’s electrical system, except better—accessible through several of the family’s rooms and devices.

Crafting the story

When it came down to how we’d express this story, there were a few touchpoints we agreed to ignore in order to maintain focus. We chose not to show any mocked-up software or hardware in order to focus on voice as the primary interface, we shied away from the perceived gender of the agent, and finally, we wanted to avoid any of the children interacting with the robots—this was primarily a tool for parents.

Presenting results

After putting together a few scenes into a narrative, recording audio commentary, and compiling it into a gigantic, auto-playing PowerPoint presentation, we sent it off to our senior leadership. We were also able to present in-person for a selected audience of design, PM, and development partners.

I’ve attempted to capture that presentation below, in a more browser-friendly manner. Each slide has an audio commentary with the talking points. It’s a little janky, hope you can make do.

FEATURE
PRESENTATION

What if there was a thorough, unbiased, & inclusive source of knowledge that helped parents manage their kids’ crucial developmental milestones? A super-smart guardian angel that’s got your back.

Maya just turned seven. Super-energetic! Very social at school & in the neighborhood. Loved piano on Monday, Ponyo on Tuesday, on Wednesday she’ll fall in love with soccer. Smart girl—sometimes struggles with maintaining focus.

Avo (4:47)

Highlights

Bing Intelligence and Knowledge Services as the fabric for unifying a multitude of life experiences

Conversation

We are looking to make improvements in our overall conversational experience. This includes natural language recognition, such as when Alina says, “Mya is interested in soccer now. I don’t know anything about it. Is it dangerous?” But we’re also showing natural language responses as well, such as when Avo says, “Sure, but registration doesn’t open until next week. I’ll sign her up then.” Another aspect we are evolving is multi-turn beyond the one or two turns we have today. The whole soccer scenario is one multi-turn conversation, getting answers to questions about injuries and soccer, competitiveness, time commitment, as well as setting up registration, ordering, and confirming shoe size. All of this is on one topic, Maya and soccer, that has lots and lots of facets. A final component that’s necessary for true Conversational UI is memory. Avo remembers details about people and applies that knowledge appropriately. Avo remembers Maya’s shoe size and asks if it is still correct. And Avo also remembers what book, Mike and Maya are reading and where they are in it.

Contextual notifications

Avo is checking Mike’s email in the background. When it detects something important, in this case an email from my as teacher saying that Maya was upset at school today, it bumped it up in priority and decides to notify Mike about it. Mike gets the notification and decides to read the email for himself on his watch.

Knowledge graph

Avo heavily utilizes the big knowledge graph throughout the scenario, whether it is addressing Alina’s concerns about soccer, finding local soccer teams, identifying deals on cleats and shin guards, or suggesting turn-taking as a way to help Maya improve her reading. This is a big scenario through and through.

Complex task completion

The entire soccer scenario is a super task. Make a decision about whether to sign my up for soccer, finding an appropriate league, registering, buying soccer gear, getting the practice and game schedule on the family calendar. It’s the full end to end task, not just bits and pieces.

Time-shifting

We also introduce a time shifting feature. Soccer registration is not open yet, so Avo is able to virtually wait in line and complete the registration when it becomes available. This is a useful evolution of our current solution, which would be to set a reminder for Alina to complete the registration in a week.

Emotional intelligence

Mike is concerned about Maya’s problems with reading. Avo recognizes this and responds appropriately, letting Mike know that this is a common problem, and suggesting turn-taking is an effective way of addressing it.

Internet of things

Avo integrates with IoT. Mike asked Avo to set bedtime half an hour earlier. So when that time rolled around, Avo dimmed the lights and started playing soothing bedtime music.

Human as hero

All of this is in service of letting Mike and Alina focus their energies on what’s important, spending time with Maya. By efficiently helping decide whether to sign up for soccer through getting registered, Avo lets Alina get things in motion quickly so she can get to playing with Maya. Avo also gives Mike tips about reading and sets him up to have quality reading time with Maya. In both cases, there’s less time spent figuring things out and more time actually doing.