user experience monitoring system.
2017 | Qualtrics
In 2017 I led the creation of Qualtrics' comprehensive, automated user experience monitoring system across its various SaaS products. The goals for the project:
- track customer experience across several dimensions,
- observe how customer experience changes over time and as a result of product enhancements
- compare products across any factor,
- do all of the above with automatic data collection and presented on a dashboard, and
- notify key stakeholders of important trends and feedback so they can act on the data
My goals for the team:
- prioritize research team around generative, foundational research, not usability testing & establish a team culture around data-driven decision making
I led a team of 4 researchers in the creation of the system. My specific contributions included proposing the overall approach (based on a single intercept survey a team member created), working with stakeholders, building the system dashboards, and building the system automations (automated messages, tickets, etc.)
The process for creating the system was similar to most research projects. I interviewed stakeholders to understand their research questions, I worked with research colleagues to determine the best way to answer those questions, I co-authored a survey instrument to collect that data, I piloted the survey and got feedback on the data, I then led a full deployment of the system across all the major products.
how it works
The system is represented in the diagram at right. Essentially every product in the Qualtrics offering has an in-product survey injected into it. Those surveys then drive internally-facing dashboards, messages & tickets, and populate a research panel.
Let me describe what the customer experiences of the system. The balance between collecting enough data to make the program valuable and not bothering customers was a chief concern. We found that balance in a lightweight invitation that appears occasionally in the bottom of the Qualtrics user interface.
The customer experience
Here's what users/respondents see when they interact with the program.
The value of this program is allowing internal stakeholders to understand customer feedback can monitor and improve the customer experience. Along those lines there are a number of communication channels and actions I set up to make the data and insights actionable.
System dashboards (internal)
Dashboards showing the "health" of the products hangs in the office and updates continually. (This example features fictional data.)
System messages (internal)
Email summaries of response themes, trends, and other data are automatically sent to internal product leadership.
Our organization uses Slack heavily, so I created Slack channels that automatically send in customer feedback so stakeholders can see discuss feedback as it comes in.
We have dashboard pages for every product that stakeholders can explore and query. (Again, the data is fictional.)
The system is used on a continual basis by Qualtrics stakeholders at every level. Customer issues are frequently addressed, and just as importantly, everyone at the company has the ability to remain connected to how customers.
This system was so impactful I was encouraged to speak about it. Below are a couple of examples of external information-sharing I've done on the system.
Check out the case study I wrote for Boxes & Arrows (click to read):
I also presented a case study of the system at the X4 Summit: