The latest shitstorm in PC hardware reviewing

Well, it’s not a matter of forgetting. In the past individual user anecdotal experience didn’t mean much because the typical professional magazine reviewer – the writer for a PC Magazine, BYTE, Compute!, etc. – was likely to be more knowledgeable than a specific user.

With the Web, we’re finding three countervailing trends at work:

  1. Anyone can make a Web site, so “professional reviewers” are actually just “guys that post reviews”.

  2. Individual users are often a lot more educated, using Google-fu, than they were 20 years ago.

  3. Individual users can collectively form a consensus view on a subject (Amazon ratings, epinion ratings, metacritic, gamerankings, etc.). “Wisdom of the Masses” and all that.

So, to retort more succinctly, 20 years ago the informed opinion of one guy held a lot more weight than the uninformed opinion of a couple of guys. Today, the uninformed opinion of one Web site holds a lot less weight than the collective opinion of thousands of people.

I like to test quiet stuff as well, though I can never tell how much readers care about how noisy their PCs are.

Depends on the user. Die hards don’t care so much, but given that so much performance these days doesn’t really manifest into anything concrete, but noise is getting louder, I think a lot of people are interested in the sweet spot between performance and noise. This trend probably is similar to the laptop adoption trend, but lagging by a few years. A lot of people didn’t think laptops would ever be taken seriously except by the business world, but now they’re used by everyone (I don’t have a desktop development machine anymore).

I care! I’d definitely like to see more emphasis on noise and heat (which usually produces noise via increased fan speed). My next CPU will have SpeedStep technology. It’s ridiculous that my P4 is waiting for me at a howling 3.2 GHz most of the time.

I care, too! The louder the better. I like my PC to sound like an aircraft carrier launching all of its planes through a curtain of pots and pans.

Hey, if you’re talking about the WRT54G, I have had two problems with it.

Every 6-8 months, I have to reboot it as it seems to get flakey.

The only other problems have been with wireless cards in the pcs… My compaq laptop hated it. My XBox wireless bridge loves it. My iBook loves it. My father in laws powerbook loved it.

So other than the stability, well, no problems.

Where im from we call them “Dickriders” of course, thats sexist language, but thats because it originated as a term for men who use the poularrity of friends to look good in front of women. It then expanded from people who fanatically defend certain Music artists (ie JayZ has the worst dickriders) to anybody who identifies too much with anything. Basically, the inverse of a “player hater”

It’s worth noting that in a lot of instances, the reader ratings on Gamespot are higher than those of the original reviewers. Grade inflation, or deflation, is just as likely to happen with consensus. And what makes you think those amazon and epinion ratings aren’t skewed by companies asking their fanboys to vote, en masse?

Today, the uninformed opinion of one Web site holds a lot less weight than the collective opinion of thousands of people.

This depends on the website, doesn’t it? If Loyd Case or Jason Cross tells me some piece of hardware is good or bad, I’m going to give more weight to their opinion than a thousand epinions reviews.

There are flaws in all systems. Considering the possibility of skewing those results, I personally do what I always do: trust individual reviewers. Amazon, epinions, and various reader reviews are interesting, but it’s my experience that they’re often useless because people have little or no obligation to accurately represent their views.

If it’s louder, you know it’s faster.

I don’t see why this is a shit storm compared to how bad hardware reviews seem to be in general. I especially love when there is 3% difference in results and they talk like it makes a real difference. Even more funny is where there is a 10%+ difference but the actual difference in framerate is like 3. :lol:

The amount of selective information is extreme it seems to me. They always couple an uber fast system with everything. It seems as if hardware sites strategy is to flood you with confusing BS so you forgot there are only a few products worth spending your money on. In the long run you can upgrade more, get better preformance, and save lots of money by going with the best preformance for value.

I used to have some sites I semi-trusted but lately they just seem all worse to me. Recently I found an article where they review a lot of products and half of them they said something like “this is the best we’ve seen”. Give me a break… :roll:

Easy – the reviews actually have meat to them. It’s not just “5/5 OMG TEH AWESOME”, the reviews are generally very well written and in depth. And I also pay more attention to the negative reviews than the positive ones =)

This depends on the website, doesn’t it? If Loyd Case or Jason Cross tells me some piece of hardware is good or bad, I’m going to give more weight to their opinion than a thousand epinions reviews.

It depends – does a single 8 hour review really give you the depth that the collective reviews of hundreds of owners will? I remember specifically taking Loyd to task for not using memtest as a measure of system stability many years ago – I’d try to provide a link, but their forum search is just utterly abysmal (you can’t sort by anything but relevance, and power search is ‘unimplemented’).

Even reviewers that I really trust and like (including Loyd and Jason) are fallible. Mike Chin at SPCR took a bit of flak because he unequivocally gave the Seasonic PSU’s the top ranking but the user forums were lit up with complaints of bugginess in the fan controllers leading to high noise and fan surge. But since Mike couldn’t repro it on his one manufacturer supplied unit, he wouldn’t even mention it in his review of the unit for a long time. In the end he finally had to mention it because the forums were too full of “I bought this unit based on Mike’s recommendation, but it does this weird thing, wtf?” and there’d be a million replies of “Yeah, me too”.

Again, unless these guys live with their test devices and multiple units, they can’t tell you about all the little nooks and crannies into which evil will crawl. I can tell you a lot more about the quirks of my Gateway M505XL and Gateway support than PC World can based on their “rigorous” testing consisting of weighing, measuring, and benchmarking it.

Amazon, epinions, and various reader reviews are interesting, but it’s my experience that they’re often useless because people have little or no obligation to accurately represent their views.

Sure, which is why you have to look for a consensus view based on multiple reviews, not just some random user’s posting. I might add that since these reviewers (the shills are pretty easy to filter out) aren’t getting paid for their reviews or relying on advertising dollars, you don’t get reviews like this (which is for the similar WAP55AG).

Now look at the PC Mag review where it got an Editor’s Choice. I don’t have the time to wade through the noise in Google, but I’m pretty sure most of the other big name magazine reviews and Web site reviews are similar. C-Net gives it an Editor’s Choice rating of 8.3, but the 98 user reviews on its own site aggregate to a 4.0 (Mediocre).

Look at the Amazon reviews for the WRT55AG which gives a consensus 2/5 based on 45 reviews. Look at the depth of the reviews and the specific problems the reviewers discuss – one reviewer has VPN issues, another has IPSEC issues, another has stability issues, several complain about support. The only ‘professional’ review I’ve seen that matches user stories is the one by smallnetbuilder.com/Tom’s Networking.

And he’s absolutely correct not to do so. In theory, he could say, “some users report problems,” but without verification the review ceases being factual observations and moves into the realm of “what people in forums say.”

If 10 people in a forum report problems, what does that really mean? What if I have a forum where people report tons of problems with ATI cards. Don’t I have some obligation to verify these errors? And if I can’t reproduce them, should I even bring them up? What if those were just NVIDIA fanboys spreading lies?

Honestly, I can’t believe a person that develops games wouldn’t understand the way forums can slant in one direction or another thanks to a tiny minority of posters (and they have a tendency to move, en masse, to do reader reviews, and vote in polls, etc.) If you visited some forums, you’d probably find that your games were buggy pieces of crap, while others think they’re the best thing since sliced bread. So if I’m supposed to take into account forum people in a review, who should I believe?

I can tell you a lot more about the quirks of my Gateway M505XL and Gateway support than PC World can based on their “rigorous” testing consisting of weighing, measuring, and benchmarking it.

Well, sure. And how long have you had it? Would you rather all reviews wait a year or longer? How would any user or “paid shill” review perfectly match your own experience over time? Any review–professional or otherwise–can only cover basic observed issues during a test period, not predict the future.

But don’t you think most reader reviews are based on superficial snap judgments, typically ones written at the height of emotion (either positive or negative)? Did they all follow some consistent testing methodology? Have they tested dozens of other units to have some point of reference for comparison?

I might add that since these reviewers (the shills are pretty easy to filter out) aren’t getting paid for their reviews or relying on advertising dollars, you don’t get reviews like this (which is for the similar WAP55AG).

Ah yes, all paid reviewers are shills, at least when their results don’t match your own. And the purity of the unspoiled reader review is a beautiful thing. (What makes yout think some of those negative reviews aren’t written by D-Link fanboys? Do network companies have fanboys?)

Hardware is a lot harder to test than people realize. You’d think it’d be easier to reach some sort of consensus, thanks to measurable data, but you still have to rely on your own observations.

Which is exactly what all of those individual reviewers are doing. Do they represent real problems those people are having? Absolutely. But every person doesn’t experience the same problems, or any at all. Such is the nature of testing.

I think there are some differences between HW and SW reviewing, in terms of what matters, and that ‘right’ answer lies somewhere between what Bacon and Steve are saying.

For HW, there are, roughly, only 3 things that matter:

  1. What does it do (i.e. the features) - usually easy to discern - reviewer just needs to confirm that they work as advertised (which may not always be true)

  2. Speed - fairly straightforward to determine through benchmarking. More important for some HW reviews than others.

  3. Reliability - Primarily the initial experience (i.e. could I get it working, and if so, how hard it was). To a lesser extent, ongoing perspective adds value (i.e. did it keep working, interfere with other HW, etc.).

A solid reviewer can excel at 1 and 2. However, they will be limited at #3, because their sample size is 1, or if they’re thorough, perhaps 2 or 3 (i.e. they try it in alternate systems, though I doubt this is ordinary practice most of the time). Reviewers are also more technically knowledgeable than the average user, which may impact their evaluation of this area (i.e. - they may know what things to tweak in their BIOS that the average user may not).

The ‘public at large’ is likely to be less thorough on 1 and 2, but they are probably a better source of information on #3 than a hardware reviewer. If a large percentage of users can’t get something to work right (say 30% of the users), then I don’t want to touch it, even though the reviewer may have fallen into the 70% that got it to work.

In software reviews, it really comes down to

  1. Is it fun/cool - very subjective, but a trusted reviewer is a good source of info on this.

and

  1. Is it buggy - there can be problems with SW as with HW, but in my personal experiences, I’ve had far less frustration with SW than with HW (as a percentage of total purchases). To wit - see my experiences in a different thread with my HP 6110 printer, and my unsuccessful attempt to replace it with a Brother all-in-one.

My solution? Try to buy HW at a local CompUSA or BestBuy that generously accepts returns, which is what I did with the Brother. Buying on-line you might save 10%, but just try getting your return accepted and credit card adjusted. I had two separate on-line HW purchases this spring where I was told to send it back for a refund, I sent it back, and got no refund. To the tune of $300 for the two of them. Grrr…

You consider it responsible to give your most glowing recommendation in light of numerous people – forum users you have a long term rrapport with – reporting problems just because you haven’t reproed it in your one case? At the very least pointing out that others are having problems that you can’t repro in your own sample would have been better.

So if I’m supposed to take into account forum people in a review, who should I believe?

Steve, come on, this is me you’re talking to. =) Yes, “forum people” can give the most slanted views possible on things, but this depends on the population, right? One would argue, somewhat plausibly, that the quality of conversation on QT3 is several steps above certain cough other forums, yes? The SPCR forums have some of the longest, most tenured posters, many of whom are articulate, conscientious, and intelligent. The quality of posting there is far beyond the typical fan site.

So it depends on the forum, yes.

In the context of games it’s still true. If a fan of a game has played it for 10x longer than the reviewer, then that fan probably has some insight the reviewer does not. People would be ignorant to both discount the posts or believe them entirely.

Any review–professional or otherwise–can only cover basic observed issues during a test period, not predict the future.

So we’re effectively in agreement – my point has been that layperson reviews can have value for this very reason and that professional reviews are very often superficial.

So what are we arguing about again?

But don’t you think most reader reviews are based on superficial snap judgments, typically ones written at the height of emotion (either positive or negative)? Did they all follow some consistent testing methodology? Have they tested dozens of other units to have some point of reference for comparison?

They don’t have to! My whole thesis is that if you take the sum wisdom of the reviews you will find a reasonable gestalt overview of problems. It doesn’t matter if Poster X has problem A and Poster Y has problem B and Poster Z has problem C, so long as I’m aware of these problems. If there’s a stronger correlation, all the more power to me.

I don’t see how this even in question – am I seriously making the wrong decision to go with two glowing Editor’s Choice awards vs. roughly 150+ detailed grievances of the Linksys router?

The point is that the aggregate is what matters.

Ah yes, all paid reviewers are shills, at least when their results don’t match your own.

I didn’t say that – come on Steve, of all people I would hope and expect you wouldn’t stoop to debate fallacies to prove a point. What I said is that there is an obvious potential conflict of interest with paid reviewers. Is it unheard of for games to get glowing reviews in exchange for exclusive covers? For hardware to get positive reviews because of complimentary copies?

And the purity of the unspoiled reader review is a beautiful thing. (What makes yout think some of those negative reviews aren’t written by D-Link fanboys? Do network companies have fanboys?)

I am not aware of network company fanboys, but then again, I wouldn’t be surprised if there were. The reason I don’t think those reviewers were reverse-shills is because the time period was spread out on the reviews, the tone was different, and the problems were different.

Keep in mind I’m talking about actual reviews, not just one liners with a star rating assigned to it.

Hardware is a lot harder to test than people realize. You’d think it’d be easier to reach some sort of consensus, thanks to measurable data, but you still have to rely on your own observations.

No argument there. In fact, you’re strengthening my point – multiple observations let you gather a more accurate view than a single data point. Do you seriously dispute that notion?

Last time I tried to return something at CompUSA because it wouldn’t work, they charged me a 15% restocking fee.

I had two separate on-line HW purchases this spring where I was told to send it back for a refund, I sent it back, and got no refund. To the tune of $300 for the two of them. Grrr…

NewEgg has never done me wrong, and Visa/Amex/MC will allow you to chargeback if necessary.

Actually, BTG’s point doesn’t even have to refer to reviewers or slants (and he knows it). The point is that reviewers have time constraints and often get product that has been better tested, carefully selected, etc. Getting epinions or something similar gives you an overview of common experiences with the product, things that potentially pop up and cause problems for people. Of course, the reviewers can’t mention this. But that’s BTG’s point. They aren’t necessarily bias. They are just one view based on one example of the product, and you must consider this when relying on the review.

I don’t think BTG is saying don’t listen to reviews. If anything, he seems to be saying, read the review from the trustworthy source, and then ALSO read the epinions to see if there are things a reviewer wouldn’t have caught but the masses do (thanks to extra sampling).

I’m going to touch on this point here.

Long ago, when I worked at HP (back in the John Young era, if anyone remembers that), the company did a study. They found out several interesting points:

  1. Anyone happy with a product will tell, on average, 9 people.

  2. Anyone unhappy with a product will tell, on average, 27 people.

This was in the context of customer support, and was the rationale for why it was good to keep squeaky wheels greased. Someone unhappy would tell a lot more people about their negative experiences.

If you look at the epinions user reviews of the Linksys router, you see many five star results and many one star results. If I were trying to make a decision based on those, I’d be baffled.

To this point, though, I think it’s still true. People will more often post their negative experiences than their positive one. That’s why these types of opinions need to be taken with a grain of salt. There is valuable information to be gleaned from them, particularly if the users are knowledgeable.

Too true. For many people a ‘positive’ experience is the experience they expect, and for them it isn’t noteworthy at all. Something has to be extraordinary in terms of service, stability, value, performance, etc, before the positives can truly flow. How many people would return to a store to tell them a product bought is working as it should compared to how many people would return to a store to tell them when a product didn’t work? Folks just don’t have that sort of time to spend on spreading positive vibes, and pointing out the negative is punishment to the manufacturer/seller for pushing their trash on us.

Has there ever been anything that hasn’t been condemned by someone, somewhere?

Sure, which is fine, because I’d much rather be a bit more cynical than optimistic when going into a buying decision.

That’s why these types of opinions need to be taken with a grain of salt.

That’s why ALL opinions need to be taken with a grain of salt. This is, again, an argument for using meta-consensus reviews.

You would think that given the 3:1 ratio of negative:positive that you cited that it would be nearly impossible to find something with close to universally positive reviews, but this is still very possible. There are plenty of books, movies, hardware applieances, etc. that have garnered lots of positive reviews.

And to state the obvious, you have to read the reviews, not just the scores. For example, the WRT55AG complaints are generally directed at the V1.0 unit, so some of the positive reviews actually cover the V2.0 but Linksys is notoriously bad about just revving versions without differentiating them clearly. I’ve also seen lots of generally positive reviews that bring to light significant issues that are glossed over because the reviewer is a ‘computer expert’ who wasn’t bothered by a lack of features or documentation.

How quickly you forgot the obvious fourth: value

How much does it cost, and what do you get for that cost relative to similarly-priced competitors?

I wonder if they similarly measured the “reach” of happy and unhappy customers today, if the gap would be even wider? I suspect with the ease of internet forums and such, there’s a geometric expansion of those numbers. The “reach” of a happy customer is probably like 20 people, and the reach of an unhappy one is probably a couple hundred.

I think it’s ethically dubious to report what are effecitvely rumors in a factual piece.

But consider this: What if we flip this around? If I’m finding an item horribly broken but everyone else reports it as being great, are you fine with me switching my 1-star review to a 5-star one? Is that being honest to the reader?

So we’re effectively in agreement – my point has been that layperson reviews can have value for this very reason and that professional reviews are very often superficial.

I’ve said all along they can have value, but how do you know the layperson review isn’t just as superficial? It’s not like you have to actually prove ownership of a product to review it.

And yes, if a year from now you want to find out what router to buy, it might be best to consult someone that’s been using the device for a year. You might find some long-term reviews on editorial sites, or maybe asking people who use it on a forum is a good idea. Reviews can’t take into account future firmware upgrades, or driver improvements, or other positive (and negative) long-term issues that pop up.

I don’t see how this even in question – am I seriously making the wrong decision to go with two glowing Editor’s Choice awards vs. roughly 150+ detailed grievances of the Linksys router?

That is easily the most extreme case I’ve ever seen. I seriously doubt there’s any single piece of hardware that skews that badly. I’m finding it hard to condemn every editorial site over one outlier.

No argument there. In fact, you’re strengthening my point – multiple observations let you gather a more accurate view than a single data point. Do you seriously dispute that notion?

I’m all for multiple views. I read professional reviews and reader reviews myself, but I think holding up the latter over those of professionals because of some notion they’re more pure, or more informative, or whatever, is naive.