Background

Buzzer moments are triggered when a player or team that a user follows gets ready to do something exciting. When Buzzer users witness their magic moment (that crazy overtime buzzer beater that they caught watching in app), they’re more likely to keep coming back to the platform in an attempt to chase the excitement they experienced. 

Our team hypothesized that users aren't receiving buzzer moment notifications because we place a heavy burden on users to follow their favorite players and teams. Users have the option to either search for who they want to follow or scroll through sorted tabs of players and teams. This results in fewer follows which means fewer users watching Buzzer moments.

If we can simplify the following process by recommending smarter, relevant, and personalized interests(players, teams and leagues), then more users will receive and watch Buzzer moments.

Approach

The idea to improve the following process was drawn from inspiration via Spotify’s recommendation-led music discovery experience. After learning that users follow on average 4.4 interests, we wanted to increase the number of interests followed to get users one step closer to their magic moment. 

This project consisted of a 3 person team including a product manager and data scientist. The next step in our process was generating lo-fi wireframes to illustrate flow and interaction, collaborate with product to define requirements and partner with data science to build and refine the recommendation algorithm. It was important that before handing off our proposed solution to engineering we gathered input from users to see things from their perspective. 

At that stage of the product development process, usability testing was the best way to test and understand if dynamic recommendations would encourage users to follow more interests and increase app value for users through personalization. As the designer and researcher for this feature, I worked closely with the product manager to solidify what the research questions should be. Based on these questions, I created a research plan, moderator guide for testing and a testing template to teach our product team how to conduct usability testing sessions.

We kicked off recruiting, walked through the testing plan, and conducted an internal pilot study before we were ready for testing. 

Methodology 

We conducted 10 moderated usability tests, 60 minutes each. While they can be time consuming, moderated usability tests made the most sense for us because they allowed us to validate early assumptions and ask the participants follow-up questions backing up the “why”. They also helped me get to know the user, build trust and establish strong relationships with participants so we could quickly bring them back for additional testing. 

Participants were asked to join via video call and we discussed  pre-test user interview questions before jumping in. They were then given access to a prototype, encouraged to think aloud during task scenarios and to share any feedback throughout the session.

​​The scenarios were designed to be tested by Buzzer users that:

• Received a Buzzer moment notification

• Visited the interest page to make additional follows

• Watched at least 1 Buzzer moment

• Were recruited from Buzzer community discord

Creating scenarios that appeal to this cohort of users ensured that the feedback was as relevant as possible, as the people being tested were more likely to follow interests and watch more moments.

Scenarios 

Tasks and scenarios involved following a participant's favorite interest on Buzzer through a variety of approaches utilizing both tab navigation and search to find specific interests. Follow up questions for the team to ask:

• How did you find the experience of completing that task?

• Does the information on this screen make sense to you?

• What would you expect to see here?

• What information on this screen do you find most valuable?

• What information is most important in helping you determine who to follow?

Insights

Our team was able to gather actionable feedback and implement changes quickly to the designs. 6/8 participants stated that dynamic recommendations would encourage them to continue following more sports, teams and players on Buzzer.

“Following” expectations 

After testing, we learned so much about what’s most important to fans when they are deciding who they want to follow. For example, some participants wanted to know how many notifications they could expect if they followed a player. Since Buzzer relies on notifications to inform users about moments they don't want to miss, some participants worried they would receive too many notifications. 

Participants want Buzzer to do the heavy lifting for them

We assumed that people would want an easier way to follow their favorite interests and receive notifications. But by usability testing, we discovered that people were just as interested in following suggested `packs” and being auto-opted into popular follows. which required much less work on the user to receive moments. Auto-opting fans into interests when they first sign up for Buzzer helps users know “what’s hot in sports right now” and decreases the amount of work for users to still have an enjoyable user experience. 

Iterating quickly 

Based on the insights we got from final results and feedback, we ended up removing recommendations in search from our MVP because users expressed the placement of this feature as distracting while trying to achieve their goal.

Reflection

Moderated tests require investment, both in terms of resources like a tool or a lab to organize the tests, but also an investment of time.  Moderated usability testing sessions take time to plan, organize, and run, as each individual session needs to be facilitated by a researcher or someone with experience in the field.

With these constraints, your pool of possible participants may also shrink. Finding participants to come to your lab or join a user interview call can be a hassle, so usually, you can only collect qualitative user feedback.

Overall, we found that in these sessions the dynamic recommendations experience performed well from a usability perspective. Users were able to discover more interests quickly and felt curious to play around with the algorithm. Following more interests to see what would be recommended next. 

Participants understood how to navigate to the interest page and when to expect recommendations. The only frustrating action for users was following interest within the search mechanism. It felt “unnatural” and “distracting” to follow more interest after intending to only search for one particular interest. 

There is a need to further explore how to ensure users are less reliant on notifications and increase access to Buzzer moments for users who are uninterested in receiving notifications. The ability to increase engagement and interaction inside the app would greatly add more value to the Buzzer user experience.