Background

Months before Buzzer signs with NBA, NHL and PGA Tour in quest to reinvent the live sports landscape with new streaming service. The product team and myself were fixated on testing, iterating and validating our eventual Buzzer moment algorithm which triggers personalized sports moments for fans to watch based on their preferences. Utilizing one of the best methods for capturing real-world context and behaviors, I led one of our initial “discovery phase” research studies: Crafting NBA Buzzer moments notifications via Dairy Study.

Goals

1. Test our early-stage moment algorithm to gather information on its current success and identify any issues to address in future iterations.

2. Understand what motivates NBA fans to watch regular season games and what challenges they face in doing so.

3. Evaluate notification content strategy to continue crafting the right brand voice for the product that resonates with next-gen sports fans

Process

At this point in our product development process, the right opportunity emerged when my team needed to understand what and how moment notifications motivate fans to watch live sports. Normally we would cohort a segment of our user base and plan to target them with personalized recommendations or A/B testing. But this study was conducted before our first product launch. We needed a way to simulate our eventual user experience to uncover how participants will interact with our notifications.

I proposed to my team that we conduct a diary study over 2 weeks with simulated notifications via Discord app. So we ran our NBA Moments api through Discord where we were able to trigger notifications based on anyone's fan preferences. This would be the first time a diary study is conducted at Buzzer.

Below is a step-by-step breakdown of my diary study process from formulating the research plan to sharing key insights after the study cross functionally.

Phase 1: Plan research study 

The first week was all about establishing our goals, process, simulated notification structure and analysis methods. Considering we didn't have a product yet, we got a bit hacky and ran our NBA Moments api through Discord so we would be able to trigger notifications based on a participant's player and team favorites.

• Team structure: Me as UX Researcher, Product Manager, Data Scientist, Product intern

• 6 week research study 

Phase 2: Recruit diverse participants & virtual group kickoff

For this study, I recruited 6 participants with varying basic demographics, sports fandom levels and streaming habits. A screener survey was shared in Buzzer’s sports discord community and I selected from a group of responses that expressed an interest in participating in our 2 week Diary Study. No one recruited participated in a diary study before.

For onboarding, I created a mini brief and walked all participants through it on a virtual call. My goals for the call were to:

• Set expectations by going over timeline, schedule and effort level required

• Provide instructions for how to log responses in respective spreadsheets

• Answer any early questions

• Foster sense of community by allowing participants to meet others in the study and add everyone to our group discord chat so we can continue to communicate

Phase 3: Simulate NBA moment notifications and collect diary entries

We asked each participant to submit a google form sharing their favorite players and teams they enjoyed watching. We used this itemized list to create a profile in our system in order to trigger notifications when simulated moments occur. Throughout the 14 day span each user received an estimated 20-30 moment notifications. 

For every moment triggered participants were tasked with logging responses by answering questions in their separate feedback form. Every morning I reviewed participant logs from the previous night to follow up on issues, questions and began identifying common themes.

One pleasant surprise was the increased amount of organic feedback from participants starting on day 3 or 4. As the study progressed we began to see more diary entries stating that they weren't interested in watching more moments but included feedback on what could make each moment notification sound more exciting.

Phase 4: Follow-up individual user interviews

Once the diary study period ended, I conducted follow-up user interviews with each of the participants to discuss potential ideas shared in their entries.

Phase 5: Analysis & Insights 

For analysis, since the participant’s responses were recorded through google forms I relied heavily on google sheets and Miro. 

Key Insights:

Exciting games such as OT & close game moments were more interesting to participants than big player performances

We found that games mattered more for those who have a greater affinity for their favorite teams than players. This helped us tweak our player moment algorithm by defining more meaningful moments and sub-moments. For example, while triple-doubles and double doubles are popular stat categories for collecting NBA data, these moments did not motivate participants to watch more hoops. However, close games, overtime games, 50+ point scoring games had a higher rating from the participants that were interested in and would actually watch.

Storylines and relevant game context play a major role in building anticipation to watch daily regular season games 

"What about notifications during the day (1-3 max) that build up the anticipation for what looks like it could be a great game or a great duel between superstars. It might be easier to capture that ether for the fans earlier in the day when they are at work and more willing to be distracted than in the moment when they may or may not be doing something else."

This was early validation for launching Buzzer Beats, our 5 minute daily live sports email newsletter which later went on to surpass 60K+ subscribers.

Participants wanted a fun, hip and relatable brand voice throughout our notifications 

While close games and viewing stat lines from your favorite player were deemed as “helpful” for participants the sports notifications landscape is crowded. With giants such as ESPN and Bleacher Report if we want to differentiate ourselves we need a brand voice that is memorable and catchy.

Actions 

• The project team began iterating on the Buzzer Moment algorithm to emphasize close games, overtimes, and player moments. The following season we successfully triggered 700+ Buzzer moments to sports fans across the US. 

• The insights spawned many new ideas for the product roadmap and future research such as “Buzzer Beats”, a daily newsletter helping fans stay up to date with the biggest events in sports each day. Eventually generating 30k+ subscribers and contributing to year over year growth in Buzzer moments watched.

• I followed up this research study by creating fan personas to better understand product market fit and target various types of fans who would find value in Buzzer.

Reflections

There's so much value in running contextual research studies, this foundational project paid dividends later down the road as we worked to explore what our actual users via analytics might be doing in their everyday lives while using Buzzer. 

The lack of a functional app to send moment notifications and track logging analytics increased the effort level for my team and the participants. Speaking on behalf of my team, we’re grateful to work closely with participants who worked around loopholes with us as we conducted this study as a <10 person early stage start up. 

One of the biggest takeaways that we would learn later and is a solid rule of thumb for diary studies is that what people say and do will oftentimes differ. We were unable to charge to actually watch moments so while many participants said “I would pay to watch”, even though we were only charging micro-transactions…. What people say and do are two different things.

Many thanks to my teammates who helped make this project happen.