PAYDAY 3 was the much-anticipated sequel to one of the most popular co-op shooters ever. Since its release, PAYDAY players have revelled in the thrill of a perfectly planned and executed heist. That’s what makes PAYDAY a high-octane, co-op FPS experience without equal.
However, although it has generated over $50 million in sales , PAYDAY 3 still received mixed reviews after a series of updates.
To gain more clarity and improve decision-making, Starbreeze Entertainment set out to understand WHY & invested time in evaluating the early onboarding experience of new players.
"Our core audience understands the nuances of Payday 3; however, we need to ensure newcomers to the franchise are better equipped to enjoy the game"
Andreas Hall-Penninger, Lead Producer - Starbreeze Entertainment
Below is how we approached it and what we learnt from the activity.
Challenges & Hypotheses
As with any live game of this scale, we tracked behaviour in a variety of ways, and we know that once someone plays the game for more than 4 hours, they're in it for the long game; however, Payday 3 is a complex title with numerous factors influencing how someone plays.
Challenges we faced when assessing our live player data,
- Feedback primarily came from the current community, which was great for evolving the game/DLC updates but didn't directly address new player dwell time.
- We could see drop-off points/areas, but we couldn't tell you why people left accurately, leading to assumptions.
To address this lack of clarity, we decided to get inside the heads of new players & watch them play for the first time.
Our Hypotheses before playtesting,
- Those who engage with tutorials enjoy the game more.
- If new players don't feel like a competent criminal (Master the four core Heist mechanics successfully) within 2 hours, they don't experience the game's differentiation and leave.
- Early enjoyment and replayability are affected by the type of heist players choose and its difficulty.
How we approached the Playtest Setup
When reflecting on the three hypotheses, we set up in the following way:
- [1] Players would need to play the game for at least 2 hours to assess;
- How playstyles evolved over time
- If applicable, at what point did new players begin losing interest in the game
- How long did it take players to master the 4 core skills in the game.
- [2] Players would be split into two groups;
- 50% would be required to play all the tutorials to assess whether having a better understanding of the game within the first hour impacted long-term motivation.
- 50% were free to play the game in whatever way they chose, re-creating players' current journey when downloading the game for the first time.
- 50% would be required to play all the tutorials to assess whether having a better understanding of the game within the first hour impacted long-term motivation.
- [3] Players would be required to play heists in a particular order to;
- Determine if difficulty and Heist type played a role in long-term retention or not.
For the audience, we felt people should have;
- Never played Payday before, as we wanted to stress test the first-time experience.
- Experience with other FPS titles so we could determine if our differentiators were noticeable.
- Been located within the US or EU regions to align with our current player base.
When it came to the survey, we created two,
- A benchmark survey, supplied every hour to assess motivation over time
- An exit survey to assess future engagement.
Both were focused on providing insight that may not been voiced during the gameplay feedback.
Top 5 things we learned from watching people play the game
[1] The tutorials didn't breed success
Although they enjoyed the unique experience, the tutorials punished players for making small mistakes by forcing them to redo everything, even if they had spent 15 minutes meticulously achieving the objectives until then.
For Example
The following video showcases a player carrying a body, but you can't crouch when you're carrying a body. Keep in mind, there is no indication that you can't do it. So she tried to crouch, and by trying to crouch, she pressed all the buttons, causing her to throw a grenade and fail.
"You can clearly see where players were going wrong by watching the gameplay. This made it painful to watch but also extremely insightful, and an eye-opener in how we need to improve our messaging to help new players.
Jonathan Piller, Data & Insights Director - Starbreeze Entertainment
[2] Players struggled to master Detection & Crowd Control skills
These are critical mechanics for new players to master & fulfil our intended Heist experience, and we weren't providing players with the necessary feedback/information to achieve success.
[3] Players found Heists overwhelming & chaotic
A knock-on effect of people not being able to master Detection & Crowd Control under pressure meant players often ended up in a shootout earlier than they wanted and found following objectives more challenging due to the chaotic nature of a Heist not going smoothly.
"Interestingly, when people went 'Loud', people FELT like they were failing when, in fact, they weren't. Mastery in all our core game pillars will help players; however, we must do more to change that sentiment when the Heist changes its state."
Andreas Hall-Penninger, Lead Producer - Starbreeze Entertainment
[4] Whether people interacted with the tutorials or not, player satisfaction went up every hour
This confirmed one of our hypotheses: when people start to grasp the game, they enjoy it more; however, it also made us re-think the importance of tutorial onboarding.
In today's market, we only have a few hours to show our USPs.
[5] More visibility on how people behaved/felt made our team discussions more meaningful.
This was eye-opening for the team; however, we look forward to shaping Payday 3 based on a more in-depth perspective and helping future fans become 'Master Criminals'!
Payday 3 Beginner's Guide 🚀
Key Takeaway
Analytics will only take you so far; however, to make meaningful game design decisions, you need the full picture by watching people play your game.