You are here: Visualware Home » Network Monitoring » MySpeed » Support » Benchmarks

Broadband Speed Tester Benchmarks
A comparison of Internet connection speed testers

by Jerry Jongerius - March 4, 2005 - v16


All Bandwidth Testers are not Created Equal


Fig.1 - myspeed.visualware.com
Is your Internet speed tester accurate?
Writing an Internet bandwidth speed tester on the surface appears to be very easy. Just download a test data file, time it, and calculate the resulting bandwidth, right? No, even major companies get it wrong, as we will see below. The result is many speed testers that report an incorrect bandwidth for your connection.

The need for speed test accuracy
Are you getting the bandwidth that you paid for? If you paid for a 8 Mbps service, but are only testing at 6 Mbps, is the problem with your broadband connection, or with your speed tester?

The need for broadband bandwidth consistency - Why QOS Matters
Your broadband connection is only as reliable as its weakest link. So if you paid for a 3 Mbps connection, but the performance of the connection jumps around from 5 Mbps to 1 Mbps (over time averaging 3 Mbps), you will effectively be limited to 1 Mbps for planning purposes. Because if you deploy an application that needs more than 1 Mbps, that application will fail, because at times it really will not have more than 1 Mbps to use. In other words, it is far easier to deploy a 'race car' (application) on a 'track' (broadband) that is 'flat' (consistent) rather than 'rolling hills' (excellent, then poor, then excellent).

Measuring broadband consistency/QOS
In order for your speed tester to provide an accurate quality of service (QOS) measurement for your broadband connection, it first needs to be able to very accurately measure your bandwidth. The concept of QOS is illustrated below.

Fig.2 - Provider A, 100% QOS
 
Fig.3 - Provider B, 20% QOS
Both provider A (Fig.2) and provider B (Fig.3) deliver an average of 3 Mbps performance. Provider A provides an excellent consistent service, obtaining a QOS of 100% (3/3). However, provider B does not provide consistent results, resulting in a QOS of 20% (1/5) (details). Most high-quality broadband connections today easily deliver a QOS over 90%. Test your broadband QOS now using MySpeed.

The bottom line
In order to correctly evaluate your Internet broadband connection bandwidth and consistency, you need a speed tester that is very accurate and precise.


Benchmarking Various Speed Testers

In Fig.4 you can see the results of the author's evaluation of various widely-used Internet broadband speed testers:

Speed Tester Run #1 Run #2 Run #3 Run #4 Run #5
MySpeed1 4.23/370 4.23/370 4.23/370 4.23/370 4.23/370
Two2 3.43/--- 3.22/--- 3.14/--- 3.18/--- 3.22/---
Three3 2.85/--- 2.88/--- 2.85/--- 3.01/--- 2.80/---
Four4 2.85/--- 3.29/--- 4.22/--- 4.16/--- 4.48/---
Five5 3.35/368 3.09/370 3.32/369 0.67/363 0.62/363
Six6 0.73/--- 0.76/--- 1.30/--- 0.33/--- 0.79/---
Fig.4 - 'Mbps / Kbps' for various Internet speed testers

All tests were run from the author's PC7 via a cable modem broadband connection8 with a known download speed around 4.23 Mbps and an upload speed around 370 kbps.

As discussed, a credible speed tester must provide both consistent and accurate results. From the table above you can see that only MySpeed provided consistent and accurate test results.

Why are some speed testers not as accurate and consistent as compared to other testers? Unfortunately, many companies write their own bandwidth speed test without understanding how easy it is to create a speed tester which contains hidden incorrect assumptions, resulting in a speed test which produces inaccurate test results. Writing an accurate speed tester at a minimum requires:
  • a detailed working knowledge of certain aspects of TCP/IP
  • a knowledge of timer units vs accuracy in the programming language used
  • an understanding of basic mathematical concepts, like precision
  • a server with an appropriate connectivity for multiple simultaneous client speed tests
  • an ability to validate one's own speed test results
Let's examine some of the speed testers and analyze the test results.


Evaluating Speed Tester Two


Fig.5 - Speed Tester Two results
After reviewing the HTML source code for this speed tester, it appears to be timing (cable modem connection type) how long it takes for a 518,095 byte JPEG image to download within the context of a web page, and calculating the bandwidth like:
TIME = (time in seconds for image to download)
KBYTES = 500;
kbps = Math.round(KBYTES/TIME*8*10*1.02) / 10;
There are a couple of issues with this code. The first is an error in the KBYTES (in 1000 byte units) of the JPEG image. It is 518.095, not the 500 seen in the code. The second is the 1.02 multiplier, which could be an attempt by the test author to account for TCP/IP header overhead, but as that multiplier is not accurate the true intent of this multiplier is not known.

Timing how long it takes to download an image is a classic, well known and published speed test method. However, this technique is flawed in that it incorrectly assumes that:
  • downloading a moderately sized image utilizes 100% of the download bandwidth immediately
  • JavaScript language is a good choice for timing events

Fig.6 - Speed Tester Two bandwidth utilization
In reality, downloading a moderately sized image only utilizes 100% of the download bandwidth after an initial startup time.
The reason why is left as an exercise for the reader (requires a moderate understanding of how TCP/IP works).
This initial under-utilization can clearly be seen in the bandwidth utilization over time graph in Fig.6, of the actual image download during this speed test. This initial short under-utilization is one of the reasons that this speed test is not accurate. This speed test assumes 100% bandwidth utilization at all times, when in reality that is not the case. The result (as can be seen in the table above) is a large error (because of the relatively small image file downloaded) by this speed tester in underestimating the true bandwidth speed (a speed of 3.2 Mbps reported for a 4.2 Mbps link).
Interestingly, when the author used a download performance verification program to download exactly the same JPEG image as is used by this speed tester, the calculated download speed was 4,234,000 bps! So, if you avoid the pitfalls and use an accurate speed test program, accurate results can be achieved. Everything (the client computer, the server computer, the network) was the same except the program used to calculate the bandwidth speed. Clearly, the bandwidth calculation in this speed tester needs to be changed.
Finally, how accurate is JavaScript at timing events? The JavaScript Date() object provides millisecond units, but not necessarily millisecond accuracy. Under Windows, the accuracy is only to (at best) 0.01 seconds. This means that depending upon the final download time, there can only exist two or to three digits of precision. So when the test displays a download speed like "3306.3 Kbps" (which implies a precision to 5 digits), given that they only have two (or three) digits of precision in their time measurements, it is clear that this speed test author did not apply the concept of mathematical precision (the end result cannot be more precise than the minimum precision of the intermediate values).
Understanding Precision: Your boss gives you a digital thermometer which displays the temperature accurate to the degree. He asks you to calculate average temperature outside your office at high noon in May. Every day in May you take a measurement. At the end of the month you calculate the average and present your results to your boss. Do you report the average with any decimal digits, like 85.47°, or do you just report the temperature rounded to the degree, like 85°.

The answer is 85° because you can never have more digits of precision in a calculated answer than the minimum digits of precision in any value used to calculate the answer. So if all temperatures have two digits of precision, you can only present an answer with two digits of precision.

Another way of understanding the answer is to look at 85.47°, which is pretty much saying that the average could not have been 85.45° or 85.49°. But given the error possible in each daily temperature measurement, you really don't know that. So reporting 85.47° as the average temperature is implying a precision that does not exist.


Evaluating Speed Tester Three/Four


Fig.7 - Speed Tester Three/Four results
After reviewing the HTML source to this speed tester, it appears to be using a very similar technique to Speed Tester Two, in that it embeds a lot of text within a web page, then times how long it takes for the web page to download, as in:
start = new Date().getTime();
(614405 bytes in the HTML)
end = new Date().getTime();
elapsed = (end-start)/1000;
KBps = Math.round(614405*100/(elapsed*1024)) / 100;
This speed test is making very similar incorrect assumptions that timing a moderately sized web page will provide you with a measure of your bandwidth. For all of the same reasons that Speed Tester Two is not accurate, neither is this speed test.

Moreover, this speed test reports a Mbps value that has been calculated incorrectly. The text on this speed tester's web page appears very clear on the issue of bits vs bytes:
  • Kbps - kilobits per second, or thousands of bits per second
  • KBps - kilobytes per second
  • Mbps - megabits per second, or millions of bits per second
  • MBps - megabytes per second
As you can see in Fig.7 (ignoring mathematical precision issues for now), downloading 614405 bytes in 1.609 seconds is 372.91 kilobytes per second, but that is not 2.913 megabits per second (it is that many megabytes per second). It is instead 3.055 megabits per second (614405*8/1.609/1000000).

The bottom line is that Speed Tester Three/Four is not accurate, and the Mbps value reported to the end user is calculated incorrectly.


Evaluating Speed Tester Five


Fig.8 - Speed Tester Five results
A Java applet is used, which comes the closest to being the correct choice for a speed test, however the results are still far from accurate. A likely cause is that their servers (or network) are over-utilized, making it very difficult to provide an accurate speed test.

When the author ran some tests with this speed tester a couple of days later, the results seemed much better, but they were still were not accurate. After reviewing the HTML source code, the test download files were found.

Once again, when using an authenticated download speed program the download results were 4.23 Mbps. This confirms that this speed tester also incorrectly assumes 100% download utilization during the entire download.


Evaluating Speed Tester Six


Fig.9 - Speed Tester
six results
This speed tester appears to have been written using Macromedia Flash. Given how quickly the test completed (under one second), this tester appears to be using a very small download file size. Also, given the wildly varied results, the web server or network to the web server appears to be highly utilized.

When the author tested the download speed from this tester's server using a speed test verification program, results were mostly in the upper 3 Mbps range, sometimes above 4 Mbps, and once around 1 Mbps, confirming that this speed test is assuming 100% bandwidth utilization during the entire download, resulting in a bandwidth measurment that is a lot more incorrect than it should be. Also, since 4.23 Mbps results were not obtained every time (except once), this confirms that either the web server or network of Speed Tester Six is over utilized.

In Fig.9, this speed tester places "Cable" below that of T1, which is around 1.5 Mbps. Given that almost all cable companies provide cable modem download speeds above 3.0 Mbps, it appears that this speed tester was written some time ago and not updated to deal with today's true broadband connections.


How Accurate is Your Speed Tester?

The key benchmark for any bandwidth speed tester is: Can it deliver consistent and accurate results for a highly accurate broadband connection of known speed? If not, the speed tester cannot be trusted.

The results of this benchmark analysis determined that only the MySpeed speed tester provided accurate and consistent Internet bandwidth speed test measurements.


References
  1. MySpeed: 4,233,464/370,888, 4,233,336/370,912, 4,232,408/370,648, 4,232,984/370,600, and 4,232,008/370,440. These results are fully auditable and can be viewed in the online enhanced reporting database. See tests #190078, #190080, #190081, #190083, and #190084. [Feb 28, 2005]
  2. CNET bandwidth meter: 3434.3, 3225.3, 3148.1, 3182.5, and 3225.3 kbps. [Feb 28, 2005]
  3. McAfee Speedometer (restart via web browser restart): 2.858, 2.885, 2.856, 3.001, and 2.804 Mbps. [Feb 28, 2005]
  4. McAfee Speedometer (restart via the 'retest' button): 2.856, 3.296, 4.223, 4.167, and 4.481 Mbps. [Feb 28, 2005]
  5. speedtest.dslreports.com 3,359,457/368,459 3,093,440/370,851 3,329,373/369,480 bps from server 1 and 679,936/363,967 620,765/363,967 bps from server 2. [Feb 28, 2005]
  6. Intel Broadband Speed Test: 732.2 Kbps, 767.9 Kbps, 1.3 Mbps, 339.4 Kbps, 700.6 Kbps. [March 3, 2005]
  7. Client computer was a Windows XP Professional, 3.20 Ghz Pentium 4, 1.0 GB RAM system. The ethernet card (Realtek RTL8169/8110 Gigabit) was connected at 100 Mbps to a LinkSys 10/100 Workgroup Switch, connected to a LinkSys WRT54G router, connected to a LinkSys BEFCMU V3 cable modem, connected to Comcast; see 8 below.
  8. Tests were performed on a broadband cable modem with a known download speed right around 4.23 Mbps and a known upload speed right around 370 kbps. In the test environment, the cable network amazingly provides virtually 100% consistent bandwidth all the time, which is very unusual for a broadband cable system, but which is great for running benchmarks. This speed was validated by downloading and uploading large files. 51,726,024 bytes were downloaded in 97,786 milliseconds (4,231,800 bps) and 8,626,342 bytes were uploaded in 186.36 seconds (370,310 bps).

  Copyright © 1997-2006 Visualware Inc. · All Rights Reserved