Most of us have suffered with consumer Wi-Fi gear whose performance is crippled by chip sets and manufacturing practices that focus on more on cutting BOM costs than in boosting range or data rates. That’s why I was very pleased to see that Metalink had finally stepped up to the plate and done some comparative performance testing between its chips and those of several competitors (see their August 8, 2007
release). My enthusiasm was considerably dampened however when I took a closer look at the report published by the Tolly Group (available on their
web site) and noticed several problems with their test methodology that, in my opinion, raised at least as many questions as the tests answered. This is especially troubling because I think Metalink has one of the best Wi-Fi solutions available on the market (see this week’s networkZONE product
review ) and that they probably would have demonstrated equally-impressive performance advantages under more rigorously-designed, more carefully-controlled tests.
One of my main concerns with the tests is that they were conducted using a so-called typical residential home (if a 6000 square foot home can be called typical) as a test environment instead of a lab. While such an environment has lots of real-world channel impairments that would be difficult to duplicate in a lab (e.g., multipath and attenuation effects from furniture, appliances and construction features) it would not be my first choice if I wanted reproducible, verifiable data. For one thing, Tolly treats the house in which the units are tested like a black box so we have only a rough idea of what sorts of channel impairments a Wi-Fi radio had to overcome in order to deliver its payload. My own experiences also show that small variations in how equipment is positioned (not to mention the positions of doors, windows, and people) can significantly affect an RF link -- especially when using a complex modulation like OFDM. I’d expect that there were at least some precautions taken to guard against these second-order problems but they were not discussed to my satisfaction in the report I read.
This is not to say that the Tolly tests are useless; it’s just that the results are not definitive and may well raise as many questions as they answer. Nevertheless, tests using an uncontrolled ambient environment have been used on several other occasions and produced some very helpful, if inconclusive information. One good example are the investigations conducted by the
Farpoint Group which produced some highly informative results using a similar set of tests to evaluate interoperability between different pre-802.11n products back in the spring of 2006. Back in the early days of 802.11 (before it was known as Wi-Fi), I also used ambient environments to perform informal tests on products. I usually did this by walking a test unit around a building while running streaming audio or video and seeing how far I could get before the link died. Much to the chagrin of the folks I was interviewing, I’d often perform an
elevator test where I’d take a laptop running a Wi-Fi card into an elevator and see how many floors I could get from the base station before the link failed. There was also the dreaded
toilet test where I’d take the units into restrooms where the tiles and plumbing in the walls presented more of a challenge than plain old sheetrock. Like the Tolly tests, they provided some very useful rough indicators about a product’s performance, even if they were neither repeatable nor quantitative in nature.
If I was running tests on Wi-Fi chip sets, or equipment, I’d want to use a so-called typical home environment in a slightly different way. Instead of using it as my primary test venue, I’d make extensive measurements in selected locations to establish a set of typical channel characteristics and then reproduce them in a controlled lab environment, perhaps using
Azimuth Systems'
series of RF environment simulators and analysis tools and IQflex manufacturing test set up. I’d also run a second set of in-home tests that would be quite similar to what Tolly did to evaluate Metalink’s chip set. I think that comparing the similarities and differences between the quantitative lab results and the qualitative residential tests would be as useful and informative as the actual performance data.
Tolly methodology also bothers me because the chip sets were tested in a piece of consumer gear instead of a more precisely-manufactured reference design or evaluation kit. While running the chips in an off-the-shelf box certainly does give you a first-order sense of how they actually perform in real-world conditions, the data produced in a test like this is clouded by several factors. First, one cannot separate how much of the deficiencies a product demonstrates are a function of the chip set itself and how much of it is a function of a poorly-designed unit. Granted, most Wi-Fi chip makers provide detailed reference designs for their products to ensure best performance, but I’ve seen several cases where those designs were modified to accommodate a manufacturer’s other priorities with little or no thought to how it affected the chip set. Likewise, my experiences as a consumer have shown that normal variations due to slack manufacturing tolerances can make two identical cordless phones, routers, or other wireless devices deliver radically different ranges, data rates, and reliability.
When you add in the fact that some ODMs will cheerfully substitute less expensive components for the carefully-specified ones in the reference design in order to shave a few cents off the BOM, its tough to say how much the Tolly test results tell you about the chip sets and what they reveal about the box manufacturers. About the only way I can imagine one could get a handle on this without using manufacturer-supplied evaluation boards would be to test several units of each model and test a much wider sampling of representative models using each chip set.
Ignoring the Tolly Group's past for a moment (several years ago Tolly unsuccessfully sued a colleague of mine over some well-documented accusations of conflict of interest he made in an Editorial), I’ll assume that the test results they produced were probably well-intentioned. Nevertheless, those results lack the levels of discipline and control to produce the kinds of results I’d like to see. Such tests are at any rate a step in the right direction. While there has been some progress on assuring interoperability of 802.11n hardware, very little has been done about assuring that products will perform well enough to deliver useful amounts of error-free data under anything but ideal lab conditions.
Although I wish they’d done a better job let’s hope that the flawed, but valuable, work done by Metalink and Tolly will put enough pressure on the Wi-Fi industry to get them to focus on real-world performance for the upcoming wave of 802.11n equipment. If anyone learns from the test results they published, or the questions the tests raise, perhaps we won’t have to suffer like we did with 802.11g and be forced to endure several generations of inadequate wireless video products before something useful finally makes it to market.
Comments? Questions? Lawsuits? Write me at lhg at en-genius.net, or weigh in with your own opinions on our blog through the link below.
