Since 2013 RootMetrics, a subsidiary of IHS Markit, has been subjecting UK mobile networks to rigorous speed and reliability tests, with the aim of helping consumers make better buying decisions.
We buttonholed RootMetrics' General Manager of Europe Scott Stonham for tips on how you can use their test results to ensure you’re getting the best possible mobile phone service for your money.
Along the way, we talked about how the age of your phone affects your connection speed, the RootMetrics testing process and plans to measure speeds on the UK’s creaky rail network.
uSwitch: Why should consumers trust RootMetrics’ tests?
Scott Stonham: In the UK, there’s no other company that has the volume of data that we have that replicates mobile life performance in everyday life. In the last 12 months we’ve collected over 1.5 million samples and we drive the equivalent of Land's End to John O’ Groats nearly 56 times a year.
And we do a lot more than just speed. I know speed is important to consumers, but we also believe that reliable networks are just as important.
uSwitch: How do you test reliability?
Stonham: What we’re looking for in that case is whether there’s a network there first of all. Can you do the thing you’re trying to do in the timeframe you’re trying to do it? Can you download your emails? Can you connect to a website? Can you make and keep a call?
uSwitch: Do you test video streaming too?
Stonham: A lot of the data that we have allows to see what it would be like for video streaming and we actually have a report coming out in the middle of the year that looks specifically at video streaming.
uSwitch: In layman’s terms, what’s the testing process?
Stonham: At its simplest we take consumer devices straight from the high street stores in the same way as any consumer would and we put those handsets through the same kind of activities as a consumer.
So we make calls, we upload things, we download things, we send and receive emails and do things that mimic web and app usage. We then take that in kit form up and down the entire UK 56 times.
uSwitch: How old are the oldest phones that you test?
Stonham: Each year we test the entire UK twice. And before we do that, we actually go out and find the handset that’s going to work best on the network.
So each year we re-evaluate which handset we’re going to use, but it tends to be the leading handset in the market for that operator.
So right now that’s the Samsung Galaxy S7, but there’s a lot of people out there who don’t have the latest handsets.
With that in mind, last year we conducted a study that actually took out a range of Samsung handsets, namely the S4, the S5, the S6 and the S7, and did some head-to-head comparisons on the performance of those devices on today’s modern networks.
Compared to the S7, the S4 suffered with many more call failures. By ‘call failure’ we mean either one drops out or you press the call button and nothing happens, so you can’t even make the call let alone have it drop out on you once you’ve started the call.
On the data-side we noticed that although the typical speed you’d be able to get on the S4 and S7 was quite close, the range of speeds that you might be able to achieve with each device was quite different.
The top-end speed we measured on the S7 was one-and-a-half times faster than the S4 on the same network.
uSwitch: So you use Samsung phones over iPhones. Did you also test iPhones’ performance?
Stonham: No, for us it’s quite operationally difficult to use the iPhone with the scientific kit that we have. So we have to rely on the Android device.
In order to create an accurate and scientific comparison on all of the networks we have to have a phone from each of the networks doing exactly the same thing, at the same time and same place.
What that means is that we have to have an additional device outside of those four to tell them when to start, when to stop and what to do in-between.
So that device controls them. That is not possible with the iOS system. We can’t use another device to tell an iPhone when to start and stop. That is our challenge.
uSwitch: How can consumers make the best use of RootMetrics’ data to make better buying decisions?
Stonham: Our data allows you to see what the true-world performance of each of the mobile operators in the places that matter to you. Whether that’s your home, or whether it’s at work or standing by the football pitch while your son plays football in the morning.
We have that data for all four of the major UK networks. So if you’re looking to switch your mobile contract any time soon, we can help you make sure you can use the service you’re buying in the places where you care about it.
You can go the website, you can type in a postcode, you can zoom around our coverage map and check out which operator performs best for you in those locations.
There’s also the RootMetrics app, which is being used in 194 countries worldwide and is used extensively across the UK. So you can download that app and use that app to check out what the coverage is like on your network by running some tests there and then.
Then you can switch to other networks to see what kind of performance you’d expect to see at the place you’re currently standing.
uSwitch: For consumers who aren’t happy with their network, what do you advise?
Stonham: There are different things you can do. You can obviously talk to your operator and understand what their plans are. Most of their customer service teams will tell you what their plans are for your area.
Then if you’re still not happy with that and you’re at the point of making a switch, I’d highly recommend using our information to help you find an operator that’s going to give you a better experience in the places that matter.
uSwitch: You’re planning to release test results for UK railway lines soon. What can you tell us about it so far?
Stonham: I’ve been working on that for quite a while. It’s not as easy to do it from a very scientific and accurate point of view as you would imagine. There are unexpected challenges along the way, but we’re there and we’ll be starting to roll out those reports in the coming months.
uSwitch: Are there plans for Rootmetics to start assessing networks’ customer service standards?
Stonham: Our testing has been designed specifically to come up with a scientific comparison of mobile network performance and we’re moving into mobile network coverage.
But our business is not to understand customer service and there are no immediate plans to start assessing it.
uSwitch: In recent RootMetrics’ tests rival providers, most notably Three, have closed the gap on EE, which has traditionally dominated when it comes to speed and reliability. Can you foresee a time when the competition catches up?
Stonham: I think it depends on what you define as ‘catch-up’, because we have seen Three challenge EE on a number of our award categories over the last 18 and 24 months, particularly around reliability.
We’re also seeing many more of our call awards being won by Vodafone, although if you are looking just at speed and data then EE is still in a leadership position.
But with continued network technology deployments and rollouts that gap is closing and in the last six months it closed at the most rapid pace we’ve seen.
uSwitch: Compared to overseas networks, do you think UK consumers should be happy with their lot? Do you think UK networks could do better?
Stonham: As much as I’d like to give you a specific head-to-head ranking of the UK versus any of the other countries that we’ve been to, including Japan and South Korea, it’s very difficult to do that.
That’s because treating it from a scientific point of view there are so many variables that differ between those countries, such as the infrastructure, building structure and urban planning, all of which make it very difficult.
There have been continuous improvements in all UK networks since we started measuring in 2013 and we’re seeing that pace of improvement increasing.
I think generally a lot of the things I read in the press and that consumers are reading in the press about the state of our mobile networks tends to be rather pessimistic.
From what I’ve seen, we tend to do rather better than what people think. So the perception is lagging behind reality to some degree.
For most of things that people are trying to do UK mobile networks are fine. But there are grounds for people to be dissatisfied in parts of the country, because mobile service is not ubiquitous and the quality is not the same across the entire nation.
That’s why what we’re trying to do is so important, because it does vary from street to street and it does vary from county to county and from town to town.
So the more information you have about exactly what that performance is, the more equipped you are to understand what you should expect and what you should expect to get better.