Published on April 5th, 2013 | by Toby Ross3
Xaor’s Cave: Ranking Systems
In FIFA 12, EA made a number of changes to the online system which had been with us throughout the generation. Seasons mode made its first appearance on the main FIFA games, having debuted on the 2010 World Cup game, and the new Flow matchmaking system aimed to smoothen the experience of finding matches. Following this, in FIFA 13, Seasons was expanded both to Ultimate Team and the Pro Clubs Championship.
I have to say off the bat that I’m a fan of both of these changes. Seasons adds an extra level of appeal to the one-off-match after one-off-match gameplay that FIFA’s online modes represent. The importance of each match is heightened hugely, the final game in a season often being a genuine nail biter.
This was exemplified by a recent Pro Clubs experience I had. The manual club I play with were in the 3rd Division, last match, 3 points needed. Into the 80th minute, things weren’t looking too positive. The score was 0-0, and we hadn’t had a great number of chances. All of a sudden though, and we had a massive break and found ourselves in the fortuitous position of having 5 of our players breaking down on goal with no defender in sight.
I took up position to the right of the ballcarrier who faced the keeper, who also had an option on his left. This was it – the simplest chance in the world and we’d be in Division 2 for the first time. When the time came the ballcarrier slid the pass to the left. I secretly breathed a sigh of relief, as it wouldn’t all come down to me. My relief was short lived as moments later my teammate blasted the ball wide of the far post.
For the next couple of minutes, all I could hear down the mics was pained, wailing laughter. No division 2 for us, and though we wanted it desperately, I’m fortunately part of a club which found it more funny than annoying. Without Seasons, this would have been an amusing moment which would have long been forgotten, but thanks to Seasons this will now be a moment which will live in the memory as one of those truly great gaming moments.
The meaning Seasons adds to each game simply has to be adored. It’s a great example of how online in games can be improved by adding a story to them, making them something more than a one off match. EA certainly shouldn’t stop here. Deepening the Seasons experience should be high on the online agenda. In this article though, I’m going to focus on something altogether more complex: Does Seasons make for a good ranking system? Does it matter if it doesn’t?
The best place to start, if anything, is where FIFA was before Seasons. Prior to Seasons’ inclusion, FIFA had a far more traditional ranking system at the forefront of its online which rewarded players for doing well by giving them a higher rank and skill level. All such systems aim to use a variety of criteria to hone in on a player’s true rank (Microsoft’s own system for doing this is known as ‘Trueskill’), and FIFA used to use a system much like this. I don’t know the formulae which ran FIFA’s system, but, most of these systems are pretty similar to eachother. One of the best known is called the Elo ranking system, which was originally used for chess.
In essence, the system works by looking at the two players ranking, guessing the result, and then comparing the actual result with the guess. If the actual result is better than the estimated result, then the players’ rank increases, otherwise it decreases. The more the actual result deviates from the estimate, the more the rank changes. All players start at the same rank (typically 1600), and typically there is also an additional factor which means that newer players’ ranks change faster than older players’ ranks.
For the most part, this is how FIFA’s system worked previously. But how well does it actually work when it comes to ranking players? Well, I could just throw assertions out there, but I can do better than that. I coded up a ranking simulation to show how an Elo type system would work with FIFA. The simulation starts with 10,000 players. Each player has three main attributes: how good they are, how often they play, and how good the team they play with is. These attributes aren’t linearly distributed, just like in real life. There are far more ‘average’ players than there are who are extremely good or bad, similarly, there are far more people who play a fair amount than who play a little bit or a huge amount. Finally, most people play with the best teams, with very few playing with the worst.
These pictures illustrate the state of the simulation. Each dot is a player. The size of the dot represents how much the player plays. The colour of the dot represents the level of team they play with (blue = worst, red = best). The position of each dot tells you how good they are and how they are ranked. From left to right, the players go from bad to good. From bottom to top, the ranks go from worst to best. The first image illustrates the simulation being run for some time, with millions of matches played overall, and over a hundred for the average player.
Immediately, the image shows up a lot of interesting trends. The most important, is that as you go from left to right, the dots rise. This implies that better players have better ranks, which is obviously as you’d hope. The second noticeable trend is shown up by the colours. At any particular skill level (left to right), the red dots tend to be above the blue ones, and this represents that people who play with good teams are ranked higher than those with bad teams. You’d expect this, as it is a clear advantage. This is a fairly decent representation of the ranking system FIFA used at the beginning of the generation.
At some point, EA sought to deal with the imbalance caused by good teams and bad. They did this by adding in a bias towards those using bad teams. I decided to simulate this too, shown in the following image:
This clearly works pretty well. The blue dots and the red dots are now mixed together, and this illustrates that players who play with weak teams are no longer being disadvantaged so much in the ranking stakes. It’s still not perfect though, because it is at its pretty arbitrary. How do you set the biases? How much of a disadvantage is using a bad team against a good team? It’s very hard to put a value on. Arguably, EA never got this quite right, and it was always advantageous to use the best teams.
EA did come up with a better solution with the Flow matchmaking system. Since FIFA 12, FIFA has done a good job of creating matches with fair teams. If you play with a 3 star team, you’ll very rarely have to play a 4 star team, for example. If this had been mixed in with the old ranking system, it actually would have worked remarkably well, as shown by the next picture.
Where biasing didn’t really work because it was difficult to tune correctly, the stricter matchmaking works well without such concerns.
So, what about Seasons? I’ve also tooled up the simulation to visualise the divisions players are in. In the following pictures, the height of each dot represents their Seasons progress. The higher they are, the higher their division, and the higher within each division, the closer they are to being promoted. Like in FIFA, each division has a higher relegation and promotion threshold.
In this first simulation, the matchmaking algorithm matches players who are in the same division and prefers to match players with similar teams.
It works remarkably well when it comes to ranking people correctly into divisions. There is a very clear trend as we go from left to right towards the higher divisions. It is somewhat blurred, but that is fairly natural because people move up and down between divisions quite a lot.
But this isn’t really that great a representation of how FIFA works right now. As many Seasons veterans will have noticed, you aren’t actually necessarily matched up to people in the same division as you. A lot of other factors go into matching, and the system doesn’t strictly enforce that your competitors are in the same division as you. According to some, the matchmaking system used in Seasons still looks at players’ skill levels (which can still be seen from some menus). This strikes me as a fairly flawed concept.
Though there is certainly a correlation between someone’s skill level and someone’s division, there is also a high potential for divergence. Two players who have just reached Division 1 for the first time may not in fact have similar rankings. If the skill level is still used for matchmaking, this would mean that one player would find Division 1 significantly harder than the other.
To illustrate this, I ran the simulation one last time, but this time the matchmaking matches players based upon skill levels rather than divisions.
The differences between the two Seasons style images is very clear. Though in the latter there is some correlation between skill and division, it is far weaker. It suggests that if skill levels were used for matchmaking in a divisions system, it would indeed see a divergence between the skill level players had and the division they were in.
Immediately, this phenomenon feels recognisable to the experience in Seasons. It illustrates why Seasons, exciting as it is, often feels like a facade. Why going up a division doesn’t necessarily feel like you’ve gone up a level and why going down a division doesn’t necessarily lead to things getting much easier. This simulation goes a long way to showing that a Divisions based ranking system is reliant upon a Division based matchmaking system to provide accuracy. Using other ranking criteria simply won’t work.
It’s not that easy though. The multiple criteria used in FIFA’s matchmaking are there for good reason. Though FIFA is an enormously popular game, it isn’t necessarily popular enough to mean there is always a perfect match. When we search for a match, we are looking for a variety of things, including a good connection and a fair team match up.
We’re also split across a variety of different modes – UT, Clubs, and normal Seasons. Each has additional constraints. In Seasons you can try to match by control setting. In Clubs, there are a variety of filters to prevent coming up against human goalkeepers and the like. With all of these things to deal with, it’s easy to see how getting the right matches to support the ranking system can be difficult.
Balancing the quality of matches and the accuracy of the ranking system isn’t easy, and for the majority not at the very top, accurate rankings/fair divisions probably aren’t that high of a priority. Seasons is fundamentally a facade – but it’s one that provides a lot of entertainment value. Unfortunately, while a Seasons like system -can- rank quite effectively, it requires strict matchmaking to do so.
One of the major advantages of an Elo style system, is that it isn’t so reliant on strict matchmaking. It can cope with matches being played between players at different ends of the spectrum, whereas in Seasons, matches between Division 1 players and Division 10 do entirely screw things up. Running an Elo style system behind the scenes, like FIFA does now, actually makes a great deal of sense when it comes to providing a ranking system alongside Seasons, but for the sake of the Seasons system, it shouldn’t drive the matchmaking. More over, matching as strictly to division as possible should be a high priority.