What is SBMM and Does it Work? Blizzard Sure Thinks So

Patrick_Yu · August 20

What is SBMM (Skill Based Match Making) and Does it Work Blizzard Games Sure Thinks So.jpg

If you've been playing online PVP multiplayer games in the past decade, you've likely been subjected to a skill-based matchmaking (SBMM) system, knowingly or unknowingly. Whether it's in League of Legends, Overwatch, or even casual games like Fall Guys, most games now incorporate SBMM. Gone are the days when you could hop into a casual lobby in Call of Duty and stomp noob after noob; with SBMM, you are now paired with players of similar skill levels.

But wait, isn't that what competitive or ranked modes are for? To some extent, yes. However, game developers have increasingly applied the ranked/competitive mode algorithm to all game modes to ensure balanced gameplay across the board.

Blizzard Entertainment, known for its successful gaming franchises, has conducted extensive research and testing on SBMM, particularly within the Call of Duty series. This article will summarize Blizzard’s 25 page white paper and explore what SBMM is, how Blizzard conducted their testing, and whether SBMM enhances gameplay for users, based on empirical data and detailed analysis.

What is SBMM (Skill-Based Matchmaking)?

Skill-Based Matchmaking (SBMM) is a system in multiplayer video games designed to match players of similar skill levels in the same game. The primary objective of SBMM is to create balanced and competitive matches, ensuring that all participants have a roughly equal chance of success. This approach enhances the overall gaming experience by preventing highly skilled players from consistently dominating less skilled ones, which can lead to frustration and a less enjoyable experience for the latter.

Call of Duty Warzone matches players based on similar skill level.jpg

SBMM works by analyzing various performance metrics of players, such as their win-loss ratio, kill-death ratio, in-game achievements, and other relevant statistics. These metrics are used to estimate a player's skill level, which is then used by the matchmaking system to group players into matches where their skill levels are comparable. This method helps maintain a level of competitiveness and fairness, ensuring that players are appropriately challenged and can improve over time without being overwhelmed by significantly more skilled opponents.

How is skill defined?

In the context of SBMM, "skill" is defined as a player's expected performance based on their past gameplay. This is measured through various in-game statistics such as kills per minute (KPM) and score per minute (SPM). Skill values are continuously updated to reflect a player's current ability, allowing for dynamic adjustments in matchmaking.

The document outlines several key properties for the skill value used in SBMM:

Constrained Range: Skill values are kept within a specific range to simplify analysis and matchmaking.
Predictive Ability: The skill metric should reliably predict future performance based on past data.
Summable: The average skill of multiple players should predict their combined ability, aiding in balanced team formation.
Adaptability: Skill values should quickly adjust to reflect changes in a player's performance.
Resilience: The overall skill distribution should remain accurate even with large changes in the player population.

How is skill calculated?

Skill is calculated based on a player's relative performance on specific metrics. After each match, these metrics are computed for each player and compared to the other players in the match, irrespective of team. The skill value is then updated based on these comparisons, with the adjustment being inversely proportional to the likelihood of a player achieving their observed outcome against the other players in the lobby. This calculation focuses solely on match performance and excludes factors like player progression or total time spent playing the game.

The document evaluates several performance metrics used to define skill:

Match Total Kills: Indicates how well a player performed in terms of kills but has poor cardinality since many players can achieve the same number of kills.
Kill/Death Ratio: Reflects both primary and secondary objectives of the game mode but can be manipulated through self-kills.
Kills/Deaths by Enemy: Ensures players cannot artificially drop their skill by self-killing but does not account for the time spent in the match.

How did Blizzard conduct their testing?

Blizzard's approach to testing Skill-Based Matchmaking involved a series of methodical experiments designed to assess its impact on player engagement and enjoyment. The testing process included both controlled experiments and comparative analysis across different game titles within the Call of Duty franchise.

In the Deprioritize Skill Test conducted in early 2024, Blizzard employed an A/B testing framework to compare the effects of different matchmaking configurations. The player population in North America was divided into two groups: a treatment group (50% of players) where the skill constraints were loosened, and a control group (the remaining 50%) that continued with the standard configuration. This setup allowed Blizzard to observe and measure behavioral differences between the two groups.

Blizzard conducted the experiments in a controlled environment to ensure reliable data collection. Players were divided into control and treatment groups to measure the impact of changes in SBMM parameters accurately. Several Key Performance Indicators (KPIs) were tracked to gauge the effectiveness of SBMM, including player retention (rate of players returning to the game within 14 days), quit rate (frequency of players quitting matches before completion), Kill Per Minute (KPM), Score Per Minute (SPM), and blowout rate (occurrence of matches where one team wins by a large margin).

By using this structured approach, Blizzard was able to gather empirical data on the effects of SBMM, providing insights into how changes in matchmaking algorithms impact player behavior and overall gaming experience.

Findings from the Deprioritize Skill Test

Blizzard's Deprioritize Skill Test yielded significant insights into the impact of loosening skill-based matchmaking (SBMM) constraints. Here’s a detailed look at their findings:

Player Retention

The data revealed that loosening skill constraints led to a decrease in player retention for 90% of the player base. Only the top 10% of players, who benefitted from less constrained matchmaking, showed increased return rates. This indicates that while high-skill players enjoyed a less restricted environment, the overall player base experienced reduced satisfaction.

Overall Result: Player retention decreased significantly for the majority (90%) of players.
Top 10% of Players: Showed a slight increase in return rates.

Quit Rate

There was a notable increase in quit rates across 80% of the players when skill constraints were loosened. This suggests a strong correlation between poorly balanced matches and higher quit rates, which negatively impacts player enjoyment.

Overall Result: Quit rates increased significantly for 80% of players.
Top 10% of Players: Experienced a decrease in quit rates.

Kill and Score Metrics

The bottom 20-30% of players saw a significant drop in their Kill Per Minute (KPM) and Score Per Minute (SPM), while the top 10% experienced higher metrics. This disparity suggests that loosening skill constraints disproportionately benefited high-skill players at the expense of lower-skill players, leading to a less enjoyable experience for the majority.

Bottom 20-30% of Players: Significant decrease in KPM and SPM.
Middle 60% of Players: No significant change in KPM and SPM.
Top 10% of Players: Significant increase in KPM and SPM.

Blowout Rates

The rate of blowouts, defined as matches where one team wins with a score delta greater than 30, increased across all players. Blowouts are known to have a negative correlation with player enjoyment, indicating that the experience was less fun for most players when skill constraints were loosened.

Overall Result: Increased blowout rates across all skill levels.
Player Experience: Blowouts negatively correlated with self-reported "fun."

Long-Term Impact

Blizzard’s analysis also indicated that the negative effects observed during the test would compound over time. If SBMM constraints were permanently loosened, the overall player population would likely erode, starting with lower-skill players leaving first, eventually leading to a less enjoyable environment even for high-skill players as the average skill level rises and the player base shrinks.

Long-Term Prediction: Continued loosening of SBMM constraints would erode the player base, negatively impacting all players eventually.

By carefully analyzing these metrics, Blizzard concluded that maintaining a balanced approach to SBMM is crucial for overall player satisfaction and retention. The test highlighted that while some high-skill players might prefer less restrictive matchmaking, the majority of the player base benefits from tighter SBMM constraints that ensure more balanced and enjoyable matches.

Does SBMM make gameplay more enjoyable for users?

Blizzard's findings suggest that SBMM is crucial for creating a balanced and enjoyable gaming experience. The data indicates that well-calibrated SBMM systems lead to higher player retention, lower quit rates, and overall better player satisfaction. However, the benefits of SBMM are nuanced and depend on the balance between different factors.

Benefits of SBMM

Balanced Matches: SBMM ensures matches are balanced, reducing the likelihood of one-sided games. Balanced matches are more competitive and enjoyable for players of all skill levels.
Player Retention: Properly tuned SBMM results in higher player retention rates, as players are more likely to return to the game if they have a positive and fair experience.
Engagement: Balanced matchmaking encourages longer engagement, reducing quit rates and increasing overall playtime.

Challenges of SBMM

High-Skill Player Experience: High-skill players may find SBMM less enjoyable as they consistently face equally skilled opponents, which can be more demanding and less casual.
Party Disparity: Balancing matches involving parties with significant skill disparities (e.g., a mix of high and low-skill players) is challenging. SBMM must strike a balance to ensure fair gameplay without disproportionately disadvantaging any group.
Dynamic Adjustment: Player skill can fluctuate due to various factors, such as experimenting with new loadouts or changes in performance. SBMM algorithms must adapt quickly to these changes to maintain balance.

Conclusion

Blizzard's research and testing on SBMM in Call of Duty highlight its importance in creating a balanced and enjoyable gaming experience. The findings indicate that SBMM significantly enhances player retention and engagement by ensuring fair and competitive matches. However, SBMM also presents challenges, particularly for high-skill players and parties with diverse skill levels.

The data-driven approach and rigorous testing by Blizzard provide valuable insights into the effectiveness of SBMM. By continually refining SBMM algorithms, Blizzard aims to balance the diverse needs of its player base, ensuring that everyone has a fair shot at success and enjoyment in the virtual battlefield. As online multiplayer games continue to evolve, SBMM will remain a crucial component of the matchmaking process, contributing to the long-term health and enjoyment of the gaming community.