IBM Blue Acres
Role: UX researcher
Teammate(s): Jan Hassel, Katy Lewis, Caroline Bivens, Yiwen Yan, Richelle Newby, Dusan Zivkovic
Methods & tools: Qualitative interviews, usability testing, affinity diagramming, Mural, Sketch, InVision
the problem
Product development teams are constantly gathering user feedback to drive their designs and roadmaps. However, without the ability to track user engagements, large enterprise organizations - with various folks and teams talking to clients - can overload their customers with feedback requests and harm their relationships.
Lack of consistent tracking also makes it challenging to understand the impact of user feedback across the products at an organizational level.
“A client once told me: ‘I just got almost the same e-mail [from another researcher]. Figure your s**t out and I’ll talk to one of you’.”
the opportunity
How might we increase visibility into user feedback engagements across product development teams?
the approach
There were 2 research phases for this project: 1) the discovery phase, and 2) the evaluative phase.
Qualitative Interviews
We first wanted to learn about the current experience with recruiting users, conducting sessions, and tracking feedback engagements. I led the team in facilitating 6 interviews with potential users - 3 design researchers and 3 product managers.
Key findings
Current engagement tracking, metrics, and tools used vary widely between roles and product teams across the business unit.
The process of finding and onboarding new user participants is complicated and time-consuming, which results in product team members feeling stuck with the same pool of users.
There is a need and a desire for more transparency and communication within teams, within products, and within the business unit into who is communicating to particular users and what is said.
Who are our users?
What is the current user feedback engagement journey?
We mapped out the journey of a typical user feedback engagement as described across our interviews, and identified major pain points in the journey.
Having a better understanding of our users and their motivations, as well as the current end-to-end journey for user feedback engagements, pointed us to the need for a central repository for folks across the business unit to be able to track not only user feedback engagements, but also the participants, internal team members, and projects. Having this data in one place can then enable teams to paint a picture of the entire ecosystem of user engagements.
Usability Testing
Once the team had concrete design artifacts to communicate our ideas, I begin conducting evaluative research using the RITE usability method. Due to time and scheduling constraints, we had to be agile in order to get the user feedback needed, so two out of the six sessions were unmoderated.
We tested two tasks across both rounds:
Add a new engagement
Add a new project
For both rounds, I focused on measuring: 1) errors made while conducting the task, 2) concerns on the user workflow, and 3) areas of confusing content and terminology.
Key outcomes
Overall, our user testing study generated a total of 14 recommended action items to improve the design for our users, and moving forward I led the team to prioritize action items.
the solution
the impact
Within 5 weeks, our team designed and built an interactive coded-prototype of a user engagement tracker for product teams. We projected that our tool will save a single business unit $4.8 million annually, with the potential to scale across the entire organization.
Our prototype generated a lot of buzz within the user research community at IBM. Within a year, the design was handed off to an engineering team that turned it into a fully-fledged tool that currently has 300+ internal IBM users.
“This will revolutionize the way we work with sponsor users and how we engage them with our offering teams.”