Looking back, we saw many changes go into Season Three to minimize role deltas in matches, and we wanted to take a look at the results we saw following the change. Let’s jump into the details with Senior Software Engineer Morgan Maddren and Lead Meta Designer Scott Mercer.
Season Three Role Delta Results
Role delta is something we’ve talked about previously on matchmaking blogs as well as the Season Three mid-cycle DevChat. “Role delta is the difference in skill between the best-paired players within each role across teams,” explains Morgan. “We implemented changes at the top of Season Three to minimize this difference.”
These changes significantly changed how our matchmaker works overall, and it took the team some time to smooth out the rough edges. But in the weeks following this update, we saw the difference in skill shrink between players across teams. “We learned a lot about the more complex dynamics of our matchmaker after this change,” Morgan elaborates. “Season Three was the first time our matchmaker started to pay close attention to the internal structure of matches, and of groups. It’s really exciting for us, because now that we’ve seen the new matchmaking tech perform in the real world, we have a bunch of new ideas for future improvements.”
The Y-axis represents the difference in MMR between the two Tanks expressed in Rank divisions. The delta between the two tanks in Competitive got closer between Season Two and Season Four.
This adjustment not only affected role delta, but it caused a chain reaction in other areas of matchmaking as well. “There were a lot of surprises for us with the new systems we built for Season Three, Morgan says “For example, the changes sped up queue times for the competitive mode without sacrificing match quality."
Damage Competitive Role Queue Times going down between Season Three and Season Four.
Morgan continues, “We also saw some issues regarding high-MMR match quality and longer queue times for parties. This was a rockier change than we would have liked, and we’re taking steps to ensure that future changes will be rolled out more slowly to prevent major disruptions like these. Matchmaking is incredibly dynamic. We simulate our changes extensively before pushing them live, but we still can’t always anticipate some of the knock-on effects of the larger changes we make.
Narrowing the MMR range
“Another goal for our matchmaking adjustments is to form matches with a narrower total range of MMRs,” says Morgan. This is a difficult challenge to take on, as the way we narrow the MMR range without worsening the queue times is to make our matchmaking smarter in every way. We need to adjust how we place large groups together in matches, meaning that we may hold some parties in queue a bit longer to ensure matches that have similar groups together.”
There’s a bunch of little details to get right too. For example, ensuring that we improve how role combinations get prioritized to make matches for large groups.
Finally, we need to be able to accurately predict the future state of the queue and use that prediction to know whether it’s okay to make a match, or whether we’d really prefer to hold some players in queue longer to wait for a better match.
Fairer matches for new and returning players
“For competitive ratings, our major goal is faster calibration and reducing the total uncertainty in the system,” says Morgan. Uncertainty here refers to how confident the matchmaking system is regarding your MMR.
This is one reason we sometimes see new players jump around or skip skill tiers. All new players start at a Bronze MMR, but not every new player is going to belong in Bronze. As the system learns more about your skill and shapes your MMR, those players might jump to different levels quickly.
“One challenge is understanding individual players over time. To do this, we’re asking ourselves how we can quickly and accurately gauge a new player, or a player returning after months or years of inactivity,” Morgan explains. “We also consider how much better a player can realistically be from role to role, and how quickly a player can improve over time. Finally, we think about how much a player’s individual performance varies naturally from day to day or in specific circumstances like solo queuing vs being in a group.”
Another challenge is learning about the dynamics of the population of players as a whole. How much does the average skill level of the population change depending on the time of day or over the course of a day? How do we handle large jumps in aggregate uncertainty when a bunch of inactive players return for a new season? These are some of the types of problems we’re trying to understand.
To this end, we’re making significant changes to MMR decay in the Season Four midcycle patch, and we’ll continue looking at other ways we can improve ratings in future patches.
Continued research and community collaboration
Our data has also revealed an additional area for improvement. Specifically, stomps that occur in matches that should be balanced.
One of our team goals is to discern why lopsided matches occur and what the matchmaking system can do to mitigate them. “These types of games could indicate areas where a player gets mis-calibrated, and our ratings need to be smarter,” says Morgan.
“What makes looking at stomps tricky is that they also happen with two well balanced teams. A really great ultimate, a key pick, or a few lucky critical shots can make a huge impact in a game of Overwatch,” says Scott.
Part of what we’re currently researching are indicators or measures that show how close or stomp-able actual matches are in-game. “Those measures are really important for long-term improvements because it gives us much more precise signals to look at when assessing new changes,” Morgan explains.
As we researched these types of matches and what causes them, we saw there were backfills occurring that were a really bad fit for the match, leading to a predictably one-sided match. Season 4 introduced changes that prioritize quality backfills over quick backfills. We’ve already seen improvements in backfilled matches just by holding the match a little longer to look for a player who fits better with the lobby MMR.
These changes are supported by the data and statistics we collect internally, but they’re driven by our players. We hear your concerns regarding matchmaking, and we plan to keep shedding light on our development process.
Thank you, as always, for sharing your feedback with us!