We joined RTE’s Big Life Fix team to help Róisín, who has Motor Neuron Disease. She will eventually lose the use of her voice. We created a solution that will allow her to continue communicating with her loved ones.
eHealth
Work with the Big Life Fix team to help to give MND sufferer Róisín Foley the ability to communicate in her own voice and help her to pass on her experience and wisdom to her children.
A smart soundboard app that allows Róisín to quickly and easily use her most used phrases that automatically updates based on who she is speaking with. To help Róisín to record messages for her children to access when she can no longer communicate with them we created an easy to use audio/video recorder.
The communication app allowed Róisín to communicate with family and friends even when attached to her breathing machine while the legacy app enabled her to comfortably record video and audio to leave behind for her loved-ones.
Róisín
Róisín Foley was diagnosed with Motor Neuron Disease (MND) in September 2017, just one month after her 30th birthday. This rare illness is destroying her motor skills, making everyday tasks increasingly difficult. MND will eventually render her unable to speak or communicate with her three daughters and family.
RTÉ TV series, Big Life Fix, got in touch with us to see how we could help Róisín improve her quality of life.
We had two main challenges:
Róisín’s health was deteriorating every day so we had to move fast. She was spending half of her time on her breathing machine so we needed to figure out how we mightintegrate our solution into her routine.
We wouldn’t have the time to build the "perfect" solution, so we asked, "what can we do in a very short space of time to bring value to her daily life?"
We worked in a very agile way - designers working very closely with developers, building a solution with the aim to get it to her as soon as possible. We could then iterate and adapt it to her needs and limitations as much as possible.
We used a combination of recordings and AI-generated sentences for Róisín’s digital voice. The aim of this was to be as close as possible to her own voice. Using an MVP approach, we built a solution she could start using on her tablet straight away.
We had access to a library of recorded phrases for when Róisín wasn’t able to use her own voice. We set out to figure out what could be a good way for her to access these.
After our initial brief, we did some research on soundboards, computer vision, eye gazing, beacons and AI, amongst others. We saw there were multiple solutions out there to help the voiceless, however most of them would use someone else’s voice. We wanted for Róisín to be able to speak in her own voice.
Róisín and her brother came to meet us at Marino HQ where we got to chat with her and hear about the main challenges she faces living with MND. She told us about her daily life, family, interests and her relation to technology.
Róisín is a very sociable woman, who is not just the mother to her children but she is also the mammy to all her brothers and sisters. She told us there are people coming and going every day in their house and they have a lot of conversations around the kitchen table. She also told us about interests which she enjoyed until she was diagnosed such as meditation, music, going to gigs, hiking and going to the gym.
She was such an inspiration - we really wanted to do all we could to try to make her life even slightly easier.
We used the "How Might We?" (HMW) method of note taking to uncover what we had to consider when designing for her. We had some concerns about mobility, accessibility and overall how to design the flow of a conversation to make it as natural as possible.
We had plenty of unknowns but one thing we knew for sure at this stage is that we wanted to preserve Róisín’s voice and unique personality.
Since our solution was going to allow Róisín to have conversations, we asked ourselves "how do we design a conversation?" Our starting point was that conversations usually have a clear structure - they have a start, middle and end.
We started sketching out ideas around structuring conversations. What might she need to start with? Probably a greeting or a conversation opener such as ‘How is it going?’ or ‘How was school today?’. And, from there, how would the conversation develop? Predicting in which way conversations could go from this initial stage is extremely difficult - there are endless ways a conversation could follow.
One initial assumption was that after the first conversational exchange Róisín might want go deeper and that words such as Why, What, Who, How and Where could be really handy for her to develop the conversation further.
There were multiple ideas brainstormed at this stage, even though we knew we’d have to settle with a simple solution for our MVP (Minimum Viable Product). An idea to speed up Róisín answers was the app reacting to external sound - such as offering predictive answers when a child asked if they could do something (‘Can I watch TV?’, for example).
We also identified that conversations are also highly dependent on context - who you are talking to, where you are, the time of the day, how do you feel or even what is the weather like (especially in Ireland!), so we started thinking about how we might identify and use context within the design.
At the end of our hackathon day, we had built the first working version of our soundboard. We could scroll through all the sentences we had thus far, but it was unclear how they might be categorised so that Róisín could easily find them?
Alongside this, we had a very first draft of what the interface might look like. This considered the first possible conversation step, who Róisín was talking to and gave her shortcuts to frequently used words.
Róisín recorded hundreds of sentences for us to use. It was amazing to have this material but how could we organise it for Róisín to easily access it? We all have different mental models so categorising these sentences was a particularly challenging task.
"We all have different mental models so categorising these sentences was a particularly challenging task."
We had a head start and found a way for Róisín to participate in this process, making sure sentence categories matched her own mental models.
On top of this, we used LyreBird.ai, an artificial intelligence tool, to analyse Róisín’s voice and allow her to create new sentences by mimicking her own voice. This technology was of course still far from perfect but at least gave Róisín the ability to generate new sentences on the spot.
Róisín was losing her hand strength so actions like holding a mobile phone were increasingly difficult for her. We decided on developing our main soundboard app on a tablet she could easily use on a stand.
“At the beginning I didn’t tell them a whole lot about it. My 12-year-old understands. They know that my hands don’t work properly. My eldest girl does all the hair and stuff for me. She does the ponytails going to school.”
We had accessibility in mind first and foremost. We were very mindful of these while designing the apps:
We also did some research into eye gazing for future-proofing. We tested the Hawkeye app, that would allow Róisín to control the interface using her eye gaze only. The implementation of this was scheduled for a later stage.
Once we had our MVP prototype ready we gave it to Róisín to play around with it and give us her feedback. We wanted to learn about things like:
Roisin’s movement had deteriorated since we last saw her so she had some difficulties hovering over buttons or accessing categories in their current placement. It was great to be able to observe her reaction and the difficulties she encountered, so we could take this and work on improving the app.
Overall Róisín was very happy with the soundboard and told us she saw how she could start using it straight away.
“I see how I could start using it straight away”
We wanted to make it easy for Róisín to access sentences quickly.
While categorising her sentences, we identified that some were specific to either children or adults also some were specific to certain people. We thought we could use beacons to identify when a person entered the room and allow Róisín to quickly access sentences she would be able to use in this context. A good example of these were sentences related to school or home tasks she could ask her children about.
Wearable beacons such as a bracelet or pendant for the children and closest family members could be used for this functionality.
Her house was mapped out to test this approach.
After the test, we made some changes to the layout and positioning of actions to ensure it was easier for Róisín to tap on the screen.
We also changed the interface colours to a cleaner, muted palette based on Róisín's own colour preferences.
In addition to the main soundboard app, we built a Legacy app for Róisín to be able to record audio and videos and choose who she would like to receive these.
This app was intended to keep her voice alive, pass on her experience and capture her wisdom for her children and family.
The communication app helped Róisín to communicate with family and friends even when attached to her breathing machine while the legacy app gave Róisín a way to easily and comfortably record audio and video to leave behind for her loved-ones.
Working with Róisín was truly an inspiration for all involved. We aimed to positively impact her life and hopefully we have done that. What we didn't expect was the impact Róisín would have on us and how we work. Getting to know Róisín and her situation really heightened the importance for us , as an industry, to focus on accessibility and designing and developing for people of all needs. We plan to make this a key part of everything we do and continue to work on serving those with all levels of ability.
We’re ready to start the conversation however best suits you - on the phone at
+353 (0)1 833 7392 or by email
.gif)
