Retention Analytics - Case Study

Background

Small Improvements is a SaaS platform for performance management and employee feedback, focusing mostly on qualitative feedback.

A team was assembled to assess the value of exploring “People Analytics” in the context of Small Improvements, after a developer’s technical exploration into the subject (pictured next) generated some excitement with the Leadership team.

As it was selected to be released as part of an “up-sell package”, the end goal was not only to be functional and useful but also “sellable”.

Our task was to validate that this was a good investment for our product, and discover how could we leverage that to bring the most value to HR professionals.

img01.png

Team

I worked alongside a PM, a PMM/Copywriter, and two developers. Due to the strategic relevance of this project, the VP of Product was also involved and assisted at key points of the process.

My Challenges

Due to lack of bandwidth, the PM took more of a “project manager” role, and there was no UX researcher in the team — Although familiar with UX processes, I was not used to being the main driver — which made me accountable for the design but also for driving a big part of the initial research, and being the central person between departments.

Unfamiliarity with People Analytics or general best practices in Data Visualization. Which required a significant amount of desk research throughout the whole project.

 

Research

We ran a series of qualitative interviews with customers in our target segment. I drafted the scripts and conducted interviews, assisted by the VP of Product and a Customer Success colleague.

Key learnings:

  • HR professionals’ decisions are always guided by “Retention”. To quote a customer, they need to answer the questions “who’s gonna leave?” and “do I care?”. Those 2 questions essentially became the basis for our main user stories.

  • The level of “data-literacy” among HR professionals in our target segment varies widely from ‘None’ to ‘Data Analyst’.

  • Most interviewees depend on time-consuming manual processes of data wrangling in less-than-friendly spreadsheets to learn from their data.

Since we did not want to try competing with customisable, feature-rich analytics tools, our aim was to create a solution that would be easier than manual spreadsheets, simple to set up, and that would focus just on what matters most.

 
img02-interview script.png

Ideation/High-Fidelity Mockups

With a clear message from the initial research, we opted for jumping from sketches to high-fidelity mocks as we were convinced that, given the nature of data-visualization and that we wanted to assess the usefulness and ability to read the data being shown, tests with low-fidelity testing would not generate sufficient learnings.

 
retention sketches1.png
retention sketches2.png
 
 
img04.png

First Usability Testing - Clickable Dummies

We ran a series of unmoderated user tests via UserTesting.com where the testers (HR professionals with no familiarity with Small Improvements) navigated a clickable InVision prototype, explained their reading of the tool, and were asked to perform simple tasks.

The prototype was generally well understood. There was confusion with some terminology, as well as some difficulty interpreting the charts. A specific color palette for data viz was developed for the second round.

 
img04b.png
img04c.png

Second Usability Testing - Live Data Prototypes

Given the developers’ exploration, it was inexpensive to implement a first version of the mocks and let Small Improvements users see and interact with live data in a coded prototype. The main goals were:

  • To assess the perceived value (e.g. “ Would this be better than what you use currently?”)

  • Find blind spots (“What did you miss that would be a blocker for you?”)

  • And assess willingness to pay

This round of testing revealed some features that users really valued but were still missing, but also confirmed they were willing to invest in such a feature.

We gathered enough confidence that the core proposal had the validity we needed and there was a valid business case, so we decided to move towards launch.


Working Towards Launch – ESP (Earliest Sellable Product)

The most needed features uncovered during testing were prioritised and a release roadmap was made focusing on reaching the earliest version of the product that can be sold and provide the core value proposition without compromising on quality.

 
img05.png
img05b.png
Retention-home.png

Knowledge transfer

Being the central person between the involved departments, I was the person with the biggest collective sum of knowledge about this feature. 

Given how different this feature was from any of our previous features, I wrote extensive documentation on purpose and functionality of the tool to assist both the Product Marketing Manager in creating a go-to-market plan and messaging to market the feature, but also for the Sales and Success colleagues to get them ready to confidently explain the value and functionality of the feature.

 
img06.png

Conclusion

This was the first time that I had such a central role in a product feature that went from the discovery phase to development, to launch. I gained first-hand experience in a number of different validation and UX research methods, from planning and setting up user interviews and tests, to moderating, and synthesizing learnings.

Albeit very challenging for me personally, this was a successful project but also a an incredible learning experience that will surely influence how I face similarly complex projects in the future.

 

“During my time as Paulo’s manager, I’ve watched him dive into his professional growth with enthusiasm. There is no question he’s a fantastic visual designer, his detailed thought process shows through in his high-quality output, but what sets him apart is his passion for learning. He’s jumped into challenging and new work with a learn-by-doing style that allowed us to work with speed. He handles complex topic in-person user research with high-value clients and can distill them into testable visual concepts. He can iterate over time, never losing focus on the value proposition to the user. And lastly, though perhaps most importantly, he can work with a diverse set of stakeholders, from educating customer teams with hi-fidelity prototypes for specialized products to understanding how to give constructive feedback to help get the best out of others, I’ve been impressed by his communication skills.”

— Arielle Kilroy, VP of Product @ Small Improvements