Broadband Speed Tester Benchmarks
A comparison of Internet connection speed testers
by Jerry Jongerius - March 4, 2005 - v16
Writing an Internet bandwidth speed tester on the surface appears to be very easy. Just download a test data file, time it, and calculate the resulting bandwidth, right? No, even major companies get it wrong, as we will see below. The result is many speed testers that report an incorrect bandwidth for your connection.
The need for speed test accuracy
Are you getting the bandwidth that you paid for? If you paid for a 8 Mbps service, but are only testing at 6 Mbps, is the problem with your broadband connection, or with your speed tester?
The need for broadband bandwidth consistency - Why QOS Matters
Your broadband connection is only as reliable as its weakest link. So if you paid for a 3 Mbps connection, but the performance of the connection jumps around from 5 Mbps to 1 Mbps (over time averaging 3 Mbps), you will effectively be limited to 1 Mbps for planning purposes. Because if you deploy an application that needs more than 1 Mbps, that application will fail, because at times it really will not have more than 1 Mbps to use. In other words, it is far easier to deploy a 'race car' (application) on a 'track' (broadband) that is 'flat' (consistent) rather than 'rolling hills' (excellent, then poor, then excellent).
Measuring broadband consistency/QOS
In order for your speed tester to provide an accurate quality of service (QOS) measurement for your broadband connection, it first needs to be able to very accurately measure your bandwidth. The concept of QOS is illustrated below.
Both provider A (Fig.2) and provider B (Fig.3) deliver an average of 3 Mbps performance. Provider A provides an excellent consistent service, obtaining a QOS of 100% (3/3). However, provider B does not provide consistent results, resulting in a QOS of 20% (1/5) (details). Most high-quality broadband connections today easily deliver a QOS over 90%. Test your broadband QOS now using MySpeed.
The bottom line
In order to correctly evaluate your Internet broadband connection bandwidth and consistency, you need a speed tester that is very accurate and precise.
In Fig.4 you can see the results of the author's evaluation of various widely-used Internet broadband speed testers:
All tests were run from the author's PC7 via a cable modem broadband connection8 with a known download speed around 4.23 Mbps and an upload speed around 370 kbps.
As discussed, a credible speed tester must provide both consistent and accurate results. From the table above you can see that only MySpeed provided consistent and accurate test results.
Why are some speed testers not as accurate and consistent as compared to other testers? Unfortunately, many companies write their own bandwidth speed test without understanding how easy it is to create a speed tester which contains hidden incorrect assumptions, resulting in a speed test which produces inaccurate test results. Writing an accurate speed tester at a minimum requires:
Let's examine some of the speed testers and analyze the test results.
TIME = (time in seconds for image to download)There are a couple of issues with this code. The first is an error in the KBYTES (in 1000 byte units) of the JPEG image. It is 518.095, not the 500 seen in the code. The second is the 1.02 multiplier, which could be an attempt by the test author to account for TCP/IP header overhead, but as that multiplier is not accurate the true intent of this multiplier is not known.
Timing how long it takes to download an image is a classic, well known and published speed test method. However, this technique is flawed in that it incorrectly assumes that:
The reason why is left as an exercise for the reader (requires a moderate understanding of how TCP/IP works).This initial under-utilization can clearly be seen in the bandwidth utilization over time graph in Fig.6, of the actual image download during this speed test. This initial short under-utilization is one of the reasons that this speed test is not accurate. This speed test assumes 100% bandwidth utilization at all times, when in reality that is not the case. The result (as can be seen in the table above) is a large error (because of the relatively small image file downloaded) by this speed tester in underestimating the true bandwidth speed (a speed of 3.2 Mbps reported for a 4.2 Mbps link).
Understanding Precision: Your boss gives you a digital thermometer which displays the temperature accurate to the degree. He asks you to calculate average temperature outside your office at high noon in May. Every day in May you take a measurement. At the end of the month you calculate the average and present your results to your boss. Do you report the average with any decimal digits, like 85.47°, or do you just report the temperature rounded to the degree, like 85°.
This speed test is making very similar incorrect assumptions that timing a moderately sized web page will provide you with a measure of your bandwidth. For all of the same reasons that Speed Tester Two is not accurate, neither is this speed test.
Moreover, this speed test reports a Mbps value that has been calculated incorrectly. The text on this speed tester's web page appears very clear on the issue of bits vs bytes:
As you can see in Fig.7 (ignoring mathematical precision issues for now), downloading 614405 bytes in 1.609 seconds is 372.91 kilobytes per second, but that is not 2.913 megabits per second (it is that many megabytes per second). It is instead 3.055 megabits per second (614405*8/1.609/1000000).
The bottom line is that Speed Tester Three/Four is not accurate, and the Mbps value reported to the end user is calculated incorrectly.
When the author ran some tests with this speed tester a couple of days later, the results seemed much better, but they were still were not accurate. After reviewing the HTML source code, the test download files were found.
Once again, when using an authenticated download speed program the download results were 4.23 Mbps. This confirms that this speed tester also incorrectly assumes 100% download utilization during the entire download.
When the author tested the download speed from this tester's server using a speed test verification program, results were mostly in the upper 3 Mbps range, sometimes above 4 Mbps, and once around 1 Mbps, confirming that this speed test is assuming 100% bandwidth utilization during the entire download, resulting in a bandwidth measurment that is a lot more incorrect than it should be. Also, since 4.23 Mbps results were not obtained every time (except once), this confirms that either the web server or network of Speed Tester Six is over utilized.
In Fig.9, this speed tester places "Cable" below that of T1, which is around 1.5 Mbps. Given that almost all cable companies provide cable modem download speeds above 3.0 Mbps, it appears that this speed tester was written some time ago and not updated to deal with today's true broadband connections.
The key benchmark for any bandwidth speed tester is: Can it deliver consistent and accurate results for a highly accurate broadband connection of known speed? If not, the speed tester cannot be trusted.
The results of this benchmark analysis determined that only the MySpeed speed tester provided accurate and consistent Internet bandwidth speed test measurements.