|
Broadband Speed Tester Benchmarks A comparison of Internet connection speed testers by Jerry Jongerius - March 4, 2005 - v16
Writing an Internet bandwidth speed tester on the surface appears to be very easy. Just download a test data file, time it, and calculate the resulting bandwidth, right? No, even major companies get it wrong, as we will see below. The result is many speed testers that report an incorrect bandwidth for your connection. The need for speed test accuracy Are you getting the bandwidth that you paid for? If you paid for a 8 Mbps service, but are only testing at 6 Mbps, is the problem with your broadband connection, or with your speed tester? The need for broadband bandwidth consistency - Why QOS Matters Your broadband connection is only as reliable as its weakest link. So if you paid for a 3 Mbps connection, but the performance of the connection jumps around from 5 Mbps to 1 Mbps (over time averaging 3 Mbps), you will effectively be limited to 1 Mbps for planning purposes. Because if you deploy an application that needs more than 1 Mbps, that application will fail, because at times it really will not have more than 1 Mbps to use. In other words, it is far easier to deploy a 'race car' (application) on a 'track' (broadband) that is 'flat' (consistent) rather than 'rolling hills' (excellent, then poor, then excellent). Measuring broadband consistency/QOS In order for your speed tester to provide an accurate quality of service (QOS) measurement for your broadband connection, it first needs to be able to very accurately measure your bandwidth. The concept of QOS is illustrated below. Both provider A (Fig.2) and provider B (Fig.3) deliver an average of 3 Mbps performance. Provider A provides an excellent consistent service, obtaining a QOS of 100% (3/3). However, provider B does not provide consistent results, resulting in a QOS of 20% (1/5) (details). Most high-quality broadband connections today easily deliver a QOS over 90%. Test your broadband QOS now using MySpeed. The bottom line In order to correctly evaluate your Internet broadband connection bandwidth and consistency, you need a speed tester that is very accurate and precise.
In Fig.4 you can see the results of the author's evaluation of various widely-used Internet broadband speed testers:
All tests were run from the author's PC7 via a cable modem broadband connection8 with a known download speed around 4.23 Mbps and an upload speed around 370 kbps. As discussed, a credible speed tester must provide both consistent and accurate results. From the table above you can see that only MySpeed provided consistent and accurate test results. Why are some speed testers not as accurate and consistent as compared to other testers? Unfortunately, many companies write their own bandwidth speed test without understanding how easy it is to create a speed tester which contains hidden incorrect assumptions, resulting in a speed test which produces inaccurate test results. Writing an accurate speed tester at a minimum requires: Let's examine some of the speed testers and analyze the test results.
TIME = (time in seconds for image to download)There are a couple of issues with this code. The first is an error in the KBYTES (in 1000 byte units) of the JPEG image. It is 518.095, not the 500 seen in the code. The second is the 1.02 multiplier, which could be an attempt by the test author to account for TCP/IP header overhead, but as that multiplier is not accurate the true intent of this multiplier is not known. Timing how long it takes to download an image is a classic, well known and published speed test method. However, this technique is flawed in that it incorrectly assumes that:
The reason why is left as an exercise for the reader (requires a moderate understanding of how TCP/IP works).This initial under-utilization can clearly be seen in the bandwidth utilization over time graph in Fig.6, of the actual image download during this speed test. This initial short under-utilization is one of the reasons that this speed test is not accurate. This speed test assumes 100% bandwidth utilization at all times, when in reality that is not the case. The result (as can be seen in the table above) is a large error (because of the relatively small image file downloaded) by this speed tester in underestimating the true bandwidth speed (a speed of 3.2 Mbps reported for a 4.2 Mbps link). Interestingly, when the author used a download performance verification program to download exactly the same JPEG image as is used by this speed tester, the calculated download speed was 4,234,000 bps! So, if you avoid the pitfalls and use an accurate speed test program, accurate results can be achieved. Everything (the client computer, the server computer, the network) was the same except the program used to calculate the bandwidth speed. Clearly, the bandwidth calculation in this speed tester needs to be changed.Finally, how accurate is JavaScript at timing events? The JavaScript Date() object provides millisecond units, but not necessarily millisecond accuracy. Under Windows, the accuracy is only to (at best) 0.01 seconds. This means that depending upon the final download time, there can only exist two or to three digits of precision. So when the test displays a download speed like "3306.3 Kbps" (which implies a precision to 5 digits), given that they only have two (or three) digits of precision in their time measurements, it is clear that this speed test author did not apply the concept of mathematical precision (the end result cannot be more precise than the minimum precision of the intermediate values). Understanding Precision: Your boss gives you a digital thermometer which displays the temperature accurate to the degree. He asks you to calculate average temperature outside your office at high noon in May. Every day in May you take a measurement. At the end of the month you calculate the average and present your results to your boss. Do you report the average with any decimal digits, like 85.47°, or do you just report the temperature rounded to the degree, like 85°.
This speed test is making very similar incorrect assumptions that timing a moderately sized web page will provide you with a measure of your bandwidth. For all of the same reasons that Speed Tester Two is not accurate, neither is this speed test. Moreover, this speed test reports a Mbps value that has been calculated incorrectly. The text on this speed tester's web page appears very clear on the issue of bits vs bytes: As you can see in Fig.7 (ignoring mathematical precision issues for now), downloading 614405 bytes in 1.609 seconds is 372.91 kilobytes per second, but that is not 2.913 megabits per second (it is that many megabytes per second). It is instead 3.055 megabits per second (614405*8/1.609/1000000). The bottom line is that Speed Tester Three/Four is not accurate, and the Mbps value reported to the end user is calculated incorrectly.
When the author ran some tests with this speed tester a couple of days later, the results seemed much better, but they were still were not accurate. After reviewing the HTML source code, the test download files were found. Once again, when using an authenticated download speed program the download results were 4.23 Mbps. This confirms that this speed tester also incorrectly assumes 100% download utilization during the entire download.
When the author tested the download speed from this tester's server using a speed test verification program, results were mostly in the upper 3 Mbps range, sometimes above 4 Mbps, and once around 1 Mbps, confirming that this speed test is assuming 100% bandwidth utilization during the entire download, resulting in a bandwidth measurment that is a lot more incorrect than it should be. Also, since 4.23 Mbps results were not obtained every time (except once), this confirms that either the web server or network of Speed Tester Six is over utilized. In Fig.9, this speed tester places "Cable" below that of T1, which is around 1.5 Mbps. Given that almost all cable companies provide cable modem download speeds above 3.0 Mbps, it appears that this speed tester was written some time ago and not updated to deal with today's true broadband connections.
The key benchmark for any bandwidth speed tester is: Can it deliver consistent and accurate results for a highly accurate broadband connection of known speed? If not, the speed tester cannot be trusted. The results of this benchmark analysis determined that only the MySpeed speed tester provided accurate and consistent Internet bandwidth speed test measurements.
|
  
|
