Since I wrote my first article on scripting, handicapping and momentum (SHM going forward) for Ultimateteam.co.uk in May 2014, I have investigated a nearly endless list of claims related to these topics. This is what I learned.
The starting point of my journey
When I decided to put my head in the hornet’s nest and write my first post on SHM, I knew three things: The claims about FIFA being manipulated contradicted everything I knew about game design and business, and the existing evidence was questionable. But I also knew that I would be perfectly fine with accepting SHM as fact if actual evidence emerged.
Even though a lot of people insisted that it is impossible to provide evidence for or against SHM, I gave it some thought and came up with some methods which would allow me to test some of the claims. That naturally triggered my curiosity. What would I discover?
Speaking of claims, one of my first observations was that SHM isn’t a uniform set of consistent and well defined claims. There is a consensus in large parts of the community around the overarching idea that FIFA matches are manipulated. But people do not agree with each other – and sometimes not even with themselves – about how and why the alleged manipulation happens.
As an example, some people believe that the game favours the lesser player while others argue that it favours the lesser team. Some believe both, which is a bit of an impasse as the lesser player may have the better team, meaning that the game would be helping both players.
Although the mere fact that people disagree has no bearing on the validity of any particular claim, it does have one interesting property: When two claims contradict each other, minimum one of them must be wrong.
People often argue that SHM is blatantly obvious, and that you have to be either blind, stupid or both not to see it. I have to disagree with that. What is absolutely obvious is that some of the things that would happen if SHM exists, are very real. What isn’t obvious is that those things happen because EA manipulates our matches.
The mere fact that you lose despite having the better team or despite being the (perhaps) better player does not prove that it happened because EA plays tricks on you. There are indeed other viable explanations. Let me state a couple of things which are absolutely irrefutable:
At some point, you will lose despite having the better team, because the opponent is a better player.
At some point, you will lose despite being a better player, because the other guy is lucky.
Hence, the mere fact that you lose despite being a better player or despite having a better team isn’t abnormal in itself.
What about this video then?
I have seen and analyzed numerous pieces of alleged evidence suggesting that SHM exists, including a few video clips which I’m sure, people will bring up when reading this article. The reality is that none of the stuff people call evidence in favor of SHM has anything to do with evidence. If you filed a law suit against EA on this basis, you would lose without question.
The recurring problem is that people fail to ask themselves whether the evidence they are looking at could have another explanation. Every single piece of alleged evidence that I have looked at so far fails on that parameter. There is always another possible explanation. Most of the time it is completely obvious. Sometimes, it takes a bit more effort to understand what lies beneath.
In order for something to become evidence of SHM, we need to take the argumentation to the next level, i.e. to a point where no other explanation than SHM is possible. When (if) that happens, SHM is blatantly obvious – but not a second earlier.
A lot of people have attempted to take the argumentation to that aforementioned other level, and I’m one of them. So, where did that brings us? I will look into that in the following sections, where I discuss the evidence in regards to what I consider the core claims in the SHM debate, namely (a) that EA helps will attempt to make matches even and (b) that EA helps out players with bad teams. I’m perfectly aware that there are other claims. As I already pointed out, SHM isn’t a consolidated, well-defined claim. But for the sake of this post, I will narrow the scope to those two core claims.
Does EA make matches even?
Earlier this year, a redditor named Arlington69 wrote a series of posts, which received some attention from the community. His posts, which he summarizes here, allegedly present evidence suggesting that the matches are being levelled. Arlington69 collected a load of data from his own matches and analysed it in various ways.
Among his findings are that in matches where you go 2 up early, your goal ratio is likely to drop later in the match. At a glance, this may sound as a convincing piece of evidence, but there is a problem: If you made the same analysis on real football, you would see the exact same phenomenon. And the reason is a statistical phenomenon called regression to the mean. If you narrow the scope to matches, where a team performed extremely poorly or extremely well in the 1st half, then the general picture will be that the 2nd half performances regress towards a more normal level.
Arlington69’s very real experiences of seeing an early lead disappear are part of the reason why people believe in things like momentum. But his approach is not a valid way of determining whether momentum exists or not, because it assumes that performance will be constant unless someone intervenes, which of course is utter nonsense. Having all your shots or scoring all your goals in the same half is a perfectly normal thing in football, and not surprisingly also in a football simulation.
So, what valid methods for testing match levelling are there? Together with my colleagues at FUTfacts.com, I deviced a couple of experiments which would yield a result if momentum exists.
First, we asked people to provide information about (a) their skill level and (b) to tell how often they felt that their matches were subject to manipulation. The logic behind this was that if matches are made even, it ought to affect successful players more often. But we found that there is absolutely no connection between a person’s skill level and his inclination to feel that his matches are manipulated. If matches are manipulated, it doesn’t serve the purpose of helping lesser players out.
Second, we looked at match results because if matches are being made even, this will reflect as a surplus of even results and hence ultimately draws. In a massive sample of match data, we found FUT’s then draw percentage to be just 18 %, which is considerably lower than in real football. This wasn’t quite a surprise though, because the goal ratio in FIFA is considerably higher than in most real life football leagues. But even when we compared FUT’s draw ratio to that of football leagues with a similar goal ratio, we found the draw ratio of FUT to be roughly the same. If matches are manipulated, it doesn’t result in more even matches.
Third, we wanted to know how often the loser of a match evidently is the better player. To achieve that, we calculated the percentage of matches where the losing party dominated on all performance metrics (passing, possession and shooting) and had a better historic track record. This was the case in just 4 % of the 1100 matches in our sample. If matches are manipulated, it doesn’t result in many evidently undeserved wins.
Where this leaves us
I hear a lot of people claim that momentum is real, and I understand perfectly well what experiences have lead to this belief. But when I look at the total state of the evidence at hand, I see nothing which suggests that this belief is correct and a lot of facts which strongly suggests that it isn’t.
I simply can’t come up with an explanation, which makes it a possibility that EA intervene in matches to make them even, while at the same time encompassing the fact that even matches are rare, that better players don’t experience more “momentum” than bad players and that undeserved wins are rare. When I look at the data above from a neutral point of view, momentum isn’t really a viable explanation.
Most of your losses happen because the opponent is a better player, or perhaps because the skill gap just wasn’t big enough to compensate for the opposite distribution of luck.
What about handicapping then?
Does EA help out the player with the lesser team then? Before we look at the evidence, let’s consider the possible motive for a second. Most people are aware that EA sells packs. What I would do if I was in the pack selling business would be to make it absolutely worthwhile to own those rare items, which you need to buy lots of packs to get your hands on. It appears counter intuitive that EA would invest in a concept which makes it advantageous to use a cheap squad that anyone can collect without spending a single dime.
I have seen people argue that EA are trying to make matches even, but first of all, this is against EA’s interest, and second, you don’t make matches even by making it worthwhile to own cheap squads, because the better player may have the lesser squad.
So, at a glance, the entire notion of handicapping doesn’t make a whole lot of sense. But despite that, a lot of people still believe that handicapping – not the skill level of the opponent – is the reason why they sometimes lose against an inferior squad.
The aforementioned Arlington69 also did some analysis in this regard. Among other things, he analysed the performance of icons and found that the prime version sometimes didn’t create as many goals as the base version. This, allegedly, suggests that handicapping could exist.
Better cards perform better
A major problem with this conclusion is however that Arlington69 completely ignores that most of his data actually suggests that prime icons perform better than the cheaper versions. When you only look for data to support the conclusion, you easily fail to see the wood from trees.
But why did Arlington69 find some examples of prime icons that were outperformed by the base versions? The answer has to do with statistics again. Icons are extremely rare, and when you focus on extremely rare items like, you won’t get samples which are sufficiently big to support a conclusion regarding the relationship between stats and performance.
I have made similar experiments based on the same datasources multiple times, but as I was aware of the problems of looking at rare cards, I tested cards released in much larger volumes. My experiments confirm that higher rated version of a player scores more goal that a lower rated version – all other things equal. There are of couse exceptions due to the fact that human skill is a far more decisive factor than stats when it comes to the performance of a card. Someone needs to press the shoot button for Suarez to score a goal…
Smoke without a fire
In spite of the evidence, some people will argue that there wouldn’t be smoke without a fire: SHM wouldn’t have become a majority belief if there wasn’t anything to it. I have some objections to that line of reasoning.
First and foremost, SHM isn’t one belief. It’s an umbrella of beliefs, which are just as different as the alleged observations of the Loch Ness monster. I believe that very few people outside the Inverness Tourist office will argue that the long list of observations of animals of different size and colour increases the chance of anything more exotic than a seal living in that loch. The same logic applies to the SHM debate. To stay with the aforementioned analogy, there really isn’t any smoke.
Second, there is nothing unusual about people believing things which aren’t real. The psychological mechanisms which lead people to believe that they lost because of foul play and not because the opponent simply was a better player or just lucky, are found in many other aspects of life. Why not here?
Third, the experiences that people report about are real – but that doesn’t mean the explanation(s) the come up with are equally real.
Fourth, the smoke-without-fire analogy rests on the assumption that FUT players in general have a realistic take on football. But one of the things I have learned through debates and through the surveys I have conducted is that a lot of the people have a quite unrealistic picture of their own skill level, a poor understanding of football as a game and a low degree of understanding of what a skilled opponent can do to your game plan. I know this will be a hard statement to swallow, but I have the facts to back it up.
You are worse than you think
Among the observations I have made is that people overassess their FIFA skills. A tricky, little survey published in /r/FIFA asked a very simple question: How good are you compared to other people responding to this poll?
70 % thought they were above average. What that tells us is that when in doubt, people in general tend to be optimistic about their own skill level. So, in essence, we are dealing with a lot of people who believe they are top 10 %, when in reality 80 % of them are average.
Another observation is that people in general fail to acknowledge certain inevitable properties of football – and therefore also FIFA. I’m aware that a lot of people will object when I compare those two things, but the fact is that FIFA is a football simulation. Therefore, it absolutely makes sense to assume that many of the properties that apply to real football apply to FIFA as well. And one of those properties is coincidence.
The random game
A German professor called Martin Lames has found that 44 % of all goals in real football matches are random. It is from my experience fair to assume that a similar percentage applies to FIFA. But according to our survey from 2017, only 3 in 10 SHM believers acknowledges that FIFA is a highly random game like real football. When you don’t acknowledge the importance of luck, you will dismiss it as an explanation when you lose – even if coincidence very often is a decisive factor.
The survery delivered another revelation about SHM believers. Complaints about SHM are very often about your players being unable to sprint, shoot and pass – a feeling which I believe that most FIFA players can relate to. According to our survey, 8 in 10 SHM believers reject the notion that the opponent can influence your ability to pass, shoot and sprint. At a glance, this may sound logical, but surprisingly, the same proportion – 8 in 10 SHM believers – believed that they were able to influence the opponent’s ability to pass, shoot, head and sprint.
What this tells us is that one of the mechanisms behind the SHM beliefis that people are likely to attribute their own success to skill while attributing their opponent’s success to foul play. The reality is, unpleasant as it may seem, that the opponent has the exact same means at his disposol as you do. This means that he can make your passes fail by marking the right players, make your shots fail by forcing you to shoot in the wrong situations and make it feel as if your players are slower by giving them less space.
A final word of advice
What I have learned from all the debates and the analyses I have conducted over the last four years is that absolutely nothing indicates that scripting, handicapping and momentum exists in any shape. I know that this conclusion will disappoint and even provoke a lot of people, who have come to rely on the narrative that their lack of success with the game is a product of foul play.
The reality is that if you do practice and do all the right things in order to improve, you most likely will win more matches.