Xaors Cave 339340

Published on April 5th, 2013 | by Toby Ross

3

Xaor’s Cave: Ranking Systems

In FIFA 12, EA made a number of changes to the online system which had been with us throughout the generation. Seasons mode made its first appearance on the main FIFA games, having debuted on the 2010 World Cup game, and the new Flow matchmaking system aimed to smoothen the experience of finding matches. Following this, in FIFA 13, Seasons was expanded both to Ultimate Team and the Pro Clubs Championship.

I have to say off the bat that I’m a fan of both of these changes. Seasons adds an extra level of appeal to the one-off-match after one-off-match gameplay that FIFA’s online modes represent. The importance of each match is heightened hugely, the final game in a season often being a genuine nail biter.

This was exemplified by a recent Pro Clubs experience I had. The manual club I play with were in the 3rd Division, last match, 3 points needed. Into the 80th minute, things weren’t looking too positive. The score was 0-0, and we hadn’t had a great number of chances. All of a sudden though, and we had a massive break and found ourselves in the fortuitous position of having 5 of our players breaking down on goal with no defender in sight.

I took up position to the right of the ballcarrier who faced the keeper, who also had an option on his left. This was it – the simplest chance in the world and we’d be in Division 2 for the first time. When the time came the ballcarrier slid the pass to the left. I secretly breathed a sigh of relief, as it wouldn’t all come down to me. My relief was short lived as moments later my teammate blasted the ball wide of the far post.

For the next couple of minutes, all I could hear down the mics was pained, wailing laughter. No division 2 for us, and though we wanted it desperately, I’m fortunately part of a club which found it more funny than annoying. Without Seasons, this would have been an amusing moment which would have long been forgotten, but thanks to Seasons this will now be a moment which will live in the memory as one of those truly great gaming moments.

The meaning Seasons adds to each game simply has to be adored. It’s a great example of how online in games can be improved by adding a story to them, making them something more than a one off match. EA certainly shouldn’t stop here. Deepening the Seasons experience should be high on the online agenda. In this article though, I’m going to focus on something altogether more complex: Does Seasons make for a good ranking system? Does it matter if it doesn’t?

The best place to start, if anything, is where FIFA was before Seasons. Prior to Seasons’ inclusion, FIFA had a far more traditional ranking system at the forefront of its online which rewarded players for doing well by giving them a higher rank and skill level. All such systems aim to use a variety of criteria to hone in on a player’s true rank (Microsoft’s own system for doing this is known as ‘Trueskill’), and FIFA used to use a system much like this. I don’t know the formulae which ran FIFA’s system, but, most of these systems are pretty similar to eachother. One of the best known is called the Elo ranking system, which was originally used for chess.

In essence, the system works by looking at the two players ranking, guessing the result, and then comparing the actual result with the guess. If the actual result is better than the estimated result, then the players’ rank increases, otherwise it decreases. The more the actual result deviates from the estimate, the more the rank changes. All players start at the same rank (typically 1600), and typically there is also an additional factor which means that newer players’ ranks change faster than older players’ ranks.

For the most part, this is how FIFA’s system worked previously. But how well does it actually work when it comes to ranking players? Well, I could just throw assertions out there, but I can do better than that. I coded up a ranking simulation to show how an Elo type system would work with FIFA. The simulation starts with 10,000 players. Each player has three main attributes: how good they are, how often they play, and how good the team they play with is. These attributes aren’t linearly distributed, just like in real life. There are far more ‘average’ players than there are who are extremely good or bad, similarly, there are far more people who play a fair amount than who play a little bit or a huge amount. Finally, most people play with the best teams, with very few playing with the worst.

These pictures illustrate the state of the simulation. Each dot is a player. The size of the dot represents how much the player plays. The colour of the dot represents the level of team they play with (blue = worst, red = best). The position of each dot tells you how good they are and how they are ranked. From left to right, the players go from bad to good. From bottom to top, the ranks go from worst to best. The first image illustrates the simulation being run for some time, with millions of matches played overall, and over a hundred for the average player.

RankingSimulation-2013-04-04-09-07-00-681

Immediately, the image shows up a lot of interesting trends. The most important, is that as you go from left to right, the dots rise. This implies that better players have better ranks, which is obviously as you’d hope. The second noticeable trend is shown up by the colours. At any particular skill level (left to right), the red dots tend to be above the blue ones, and this represents that people who play with good teams are ranked higher than those with bad teams. You’d expect this, as it is a clear advantage. This is a fairly decent representation of the ranking system FIFA used at the beginning of the generation.

At some point, EA sought to deal with the imbalance caused by good teams and bad. They did this by adding in a bias towards those using bad teams. I decided to simulate this too, shown in the following image:

RankingSimulation 2013-04-04 09-16-05-440

This clearly works pretty well. The blue dots and the red dots are now mixed together, and this illustrates that players who play with weak teams are no longer being disadvantaged so much in the ranking stakes. It’s still not perfect though, because it is at its pretty arbitrary. How do you set the biases? How much of a disadvantage is using a bad team against a good team? It’s very hard to put a value on. Arguably, EA never got this quite right, and it was always advantageous to use the best teams.

EA did come up with a better solution with the Flow matchmaking system. Since FIFA 12, FIFA has done a good job of creating matches with fair teams. If you play with a 3 star team, you’ll very rarely have to play a 4 star team, for example. If this had been mixed in with the old ranking system, it actually would have worked remarkably well, as shown by the next picture.

RankingSimulation 2013-04-04 09-20-02-681

Where biasing didn’t really work because it was difficult to tune correctly, the stricter matchmaking works well without such concerns.

So, what about Seasons? I’ve also tooled up the simulation to visualise the divisions players are in. In the following pictures, the height of each dot represents their Seasons progress. The higher they are, the higher their division, and the higher within each division, the closer they are to being promoted. Like in FIFA, each division has a higher relegation and promotion threshold.

RankingSimulation-2013-04-04-09-32-56-273

In this first simulation, the matchmaking algorithm matches players who are in the same division and prefers to match players with similar teams.

It works remarkably well when it comes to ranking people correctly into divisions. There is a very clear trend as we go from left to right towards the higher divisions. It is somewhat blurred, but that is fairly natural because people move up and down between divisions quite a lot.

But this isn’t really that great a representation of how FIFA works right now. As many Seasons veterans will have noticed, you aren’t actually necessarily matched up to people in the same division as you. A lot of other factors go into matching, and the system doesn’t strictly enforce that your competitors are in the same division as you. According to some, the matchmaking system used in Seasons still looks at players’ skill levels (which can still be seen from some menus). This strikes me as a fairly flawed concept.

Though there is certainly a correlation between someone’s skill level and someone’s division, there is also a high potential for divergence. Two players who have just reached Division 1 for the first time may not in fact have similar rankings. If the skill level is still used for matchmaking, this would mean that one player would find Division 1 significantly harder than the other.

To illustrate this, I ran the simulation one last time, but this time the matchmaking matches players based upon skill levels rather than divisions.

RankingSimulation 2013-04-04 09-51-47-306

The differences between the two Seasons style images is very clear. Though in the latter there is some correlation between skill and division, it is far weaker. It suggests that if skill levels were used for matchmaking in a divisions system, it would indeed see a divergence between the skill level players had and the division they were in.

Immediately, this phenomenon feels recognisable to the experience in Seasons. It illustrates why Seasons, exciting as it is, often feels like a facade. Why going up a division doesn’t necessarily feel like you’ve gone up a level and why going down a division doesn’t necessarily lead to things getting much easier. This simulation goes a long way to showing that a Divisions based ranking system is reliant upon a Division based matchmaking system to provide accuracy. Using other ranking criteria simply won’t work.

It’s not that easy though. The multiple criteria used in FIFA’s matchmaking are there for good reason. Though FIFA is an enormously popular game, it isn’t necessarily popular enough to mean there is always a perfect match. When we search for a match, we are looking for a variety of things, including a good connection and a fair team match up.

We’re also split across a variety of different modes – UT, Clubs, and normal Seasons. Each has additional constraints. In Seasons you can try to match by control setting. In Clubs, there are a variety of filters to prevent coming up against human goalkeepers and the like. With all of these things to deal with, it’s easy to see how getting the right matches to support the ranking system can be difficult.

Balancing the quality of matches and the accuracy of the ranking system isn’t easy, and for the majority not at the very top, accurate rankings/fair divisions probably aren’t that high of a priority. Seasons is fundamentally a facade – but it’s one that provides a lot of entertainment value. Unfortunately, while a Seasons like system -can- rank quite effectively, it requires strict matchmaking to do so.

One of the major advantages of an Elo style system, is that it isn’t so reliant on strict matchmaking. It can cope with matches being played between players at different ends of the spectrum, whereas in Seasons, matches between Division 1 players and Division 10 do entirely screw things up. Running an Elo style system behind the scenes, like FIFA does now, actually makes a great deal of sense when it comes to providing a ranking system alongside Seasons, but for the sake of the Seasons system, it shouldn’t drive the matchmaking. More over, matching as strictly to division as possible should be a high priority.

Tags: , ,


About the Author

Writer for FIFA Insider, and currently a Computer Science undergraduate at the University of Warwick. Lifelong passionate gamer with an interest in a wide variety of games from the well known to the niche, with aspirations to work in the industry in future. Into the more traditionally hardcore games, and forever grumbling about the casualisation of something or other.



  • Glen

    Very nice article.

    I disagree though about the use of skill level.

    In Seasons for example, I am in division 4 but my leaderboard ranking is 2800. I have a skill level of 29/30 (remember skill levels were now carried over from FIFA 12) so when I first started playing I was also matched to similar skill level people in each of the division and continue to be.

    What your graphs don’t take in to account is that someone with a new GT/PSN ID could blitz their first 100 games and get in to Division 1 with a very low skill level. Certainly a lot lower than mine. I have never played anyone in any division who is not +/- more than 5 skill levels of myself.

    So clearly the point here is that its about localised “balance” rather than Divisions being the focus for the game. Whether this is right or wrong is probably a separate discussion.

    You also have to remember that the ELO algorithm is there for Ranking not for Match Making. It is the method for calculating players ability verses the leader board rather than deciding whether players should be matched. It just provides the “scoring” of a given match up rather than actually matching them. This is why skill levels are important to make the games balanced. Whilst that doesn’t mean two people at the same skill level will play in complimentary ways or ways you would like it does mean that you both have enough “skill” at the game to make it competitive.

    • Toby Ross

      Most of what you say doesn’t really disagree with what I was saying. To clarify a few things:

      I am using the Elo system as a rough equivalent to FIFA’s skill level system. There are obviously differences but they shouldn’t really affect the results of the simulations I ran.

      I don’t really know what you mean by it ‘being there for ranking not matchmaking’. It is a ranking algorithm and ranks are used in most games to matchmake. The Elo algorithm doesn’t matchmake, the Elo algorithm provides ranks which are used for matchmaking.

      FIFA doesn’t use Elo, but it does use a ranking system, and it does use those ranks for matchmaking. In all but the fourth graph, the Elo ranks are used for matchmaking too (amongst other criteria).

      In any case, as for the skill level stuff… I don’t see how I didn’t take that into account. As I note in the text, the reason why matchmaking to a skill level in a divisional system is problematic is that there is a potential for divergence between someone’s division and someone’s skill level.

      Your example, of someone really good starting late, is an example of this. As you say, they’ll reach Division 1 with a much lower Skill Level than someone who has been there for a long time.

      In fact, the fifth simulation does take this into account. New players are added while the simulation runs, and they are given the base Elo rank. In the fifth simulation, they will find it far easier to reach Division 1 because their rank will be low.

      This is precisely the type of thing which leads to the fifth graph having such a low correlation between Skill and Division. In the fourth graph however, because the matchmaking is based on Division, the correlation is much higher.

      So, I definitely do take that into account. In any case, my point is that using Skill Levels as a matchmaking criterion screws up the accuracy of the Division based ranking. For that reason, I suggest that the Division # is used for matchmaking rather than the Skill Level.

      The skill levels should still be calculated and stored (if nothing else, they can be used to provide realistic rankings for those top few percent who actually care). This would vastly improve the sense that you were actually going up and down in a real division system.

  • Pingback: FIFA 13 News |

Back to Top ↑