The latest shitstorm in PC hardware reviewing

I’m a professional game reviewer and I don’t hold my chosen field in particularly high esteem. I find that people read game reviews most often to confirm their own opinions, to look at screenshots, and for entertainment. Deciding whether or not to buy the actual game seems rather low on the list for many - marketing has chosen for them a long time ago. Of course, it can be argued that game reviews are just the final phase in marketing a game, and from my experience I wouldn’t doubt that.

Some of this holds true for hardware reviews, but with key differences. For starters, the major items in hardware reviews are graphics cards and CPUs, both markets being dominated by two competitors who manage a relative performance and feature parity for successive generations. The rivalry and competition is intense in ways that game publishers cannot match. It is very personal for both companies since they are fighting head-to-head, rather than with a series of competitors. People buy many games per year, they don’t look at who the publisher is too often - they look at what the game is. People buy one graphics card or one CPU for their computer, and they do care who makes it. Furthermore, a publisher releases many games in a year, most very different from each other. Graphics and processor makers release one product per year or once every several years - and everything else they sell is based off this single design. Thus, the reputation an item gets before its release is vital - it will affect its starting position in a competition that will last the next year or even years.

NVIDIA or ATI, or AMD or Intel fanboys are rabid in ways to make the Counter-Strike or Quake fanboy quiver in fear. These guys strike up a holy crusade for their company with a side-blind loyalty of such intensity as to color their entire worldview. They’re like the Swiftboat Vets - reality does not interfere with their goals and opinions, they make their own truths. They will assault hardware reviewers for using the “wrong” driver verson in an article because the “right” driver could have eeked out 1.2% more frames in test X, which is obviously the definitive test because it has been the trusted standard/the hot new game/the most reliable/the most stressful benchmark. Fanboys are spread across forums and communities - not just in the hardware community but anywhere on the internet. Car forums, astronomy forums, game forums, sports forums - they’re everywhere, and they preach their religion.

Into this mess ventures the hardware editor. He is a rarer beast than the game reviewer - the knowledge requirements, the set-up costs and the difficulty of establishing trust with manufacturers are all significant barriers to entry. Many attempt it, most fail. Even when they succeed, that success is limited; there is only so much review hardware floating around and the stakes are very, very high for the companies involved. The pressure on the editor to give a “balanced” review is enormous, hardware writers can’t just skip over to the local store and pick up a GeForce 7800 GTX for $50 like they’re buying a game, and they can’t get early hardware for a day 1 review by downloading it off the internet.

Consequently, it’s generally rare that a stink is raised in the hardware community, but when it happens it’s huge. People may remember the “Quake3/Quack3” ATI driver optimization gimmick revealed on Tom’s Hardware, or the Rambus article on AnandTech, or sometimes game reviewers get sent on hardware trips and open their mouths on subjects they shouldn’t touch… cough … err… ahem, John, don’t say anything kthx. Sometimes authors make a mistake, other times they were fed information by one of the competitors, and sometimes it’s just a discovery during benchmarking and fooling around with settings. Inevitably, anything that doesn’t conform to the mold becomes the object of intense scrutiny.

Enter Hardware Analysis’ ATI R520 P/review.

This is actually a fairly tame piece. Well, the benchmarks are pretty bombastic because they show the upcoming ATI part failing to compete with the GeForce 7800 and barely outpacing the 6800 series. But the author immediately discredits himself by airing his grievances with ATI and by pointing out that he hasn’t actually done the benchmarks himself. He has very likely sunk his site and his career by publishing this. Even if he hadn’t drawn doubts about the work, odds are that these benchmarks are not accurate and when the final hardware is released, the author’s reputation will be in tatters. The only hope for him is that ATI’s part does flop. Even then, ATI will never trust him again and other companies will be very wary.

He could have done a lot more damage to ATI if he wasn’t trying to hedge. Claiming to have at least tested the hardware personally, if not actually having it himself, would have gone a long way. Removing any comments suggesting his bitterness towards ATI, on top of that, would probably hit at ATI’s bottom line. Remember, ATI has one product. They have many variants of it, but one product - a proper attack on their future release could easily have affected their bottom line and more importantly, stock price. ATI is already in a vulnerable position with take-over rumors floating around; further depressing the value of the company might make them come true. ATI’s choices would be limited - do they release the hardware to reviewers early to show the true performance, but then risk NVIDIA having weeks of advance warning of what kind of competition they’ll face and thus being able to muster a counter? Or do they sit quiet and face criticisms? They can take legal action, but what will that say about their hardware?

Of course, even in its rough form, the article has managed to cast doubts among the fanboys. While the ATI faithful are quick to assail the review, it has given the NVIDIA fans ammunition with which to convert the unwashed pagan masses who have not yet decided on a religion. Even the highly intelligent and well-informed crowd at Beyond3D is very careful about pronouncing this as an obvious fake, precisely because they are well-informed enough to not be able to dismiss this as a pure figment of the author’s imagination. Specifically, what seems to confuse everyone is the high performance of the X1800 XT in Doom III relative to its capabilities in other games and relative to the old X850. This dismisses the most obvious suggestion, that the hardware in the article is a mere X850 that’s been overclocked. In a further tease, the X1800 XT is performing just well enough to compare with the 7800 series, but not quite beat it. Then there is the rumored cancellation of the 7800 Ultra series, which might imply that NVIDIA has its own sources and decide that the X1800 line won’t be a threat, adding further weight to the truth of the benchmarks.

It’s all so interesting :)

I’ve never understood hardware vendor fanboys. I’ve owned a variety of video cards from nVidia, Ati, and 3dfx. I’ve owned CPUs from both Intel and AMD (although recently more of the latter). Whenever I’m due for an upgrade I do a little bit of research and then decide what to purchase based on which product will give me the best price/performance. I’m not loyal to any single vendor, nor do I hate any company (except Creative, of course, but we’re talking CPUs and video cards here, not soundcards).

Of all the things to get worked up over, choosing one hardware vendor over another – particularly when performance is very similar between the two – seems pretty stupid.

I’m with MarchHare. Any hardware I pick up, regardless of its purpose, is bought after as much research as I think necessary to get a good quality product (not necessarily ‘the best’). I don’t favour any particular brand or company, as none have supplied me with such overwhelming service or product quality that would make me a lifetime loyal customer.

I’m more inclined to like/dislike particular reviewers of software/hardware than the products/companies themselves. Issues with bias and pressure from the developer/publisher/manufacturer make me wary about immediately trusting any review. This is the reason I read many reviews from many sources (again, as many as I feel necessary) including user reviews to get an overall feel for how a product is being received. Any single review, on its own, is only a mild pointer suggestive of the product quality; it’s not the final word.

In terms of the ATI R520 article linked above, it is interesting only for the bias that is obvious in the first two paragraphs. The review would barely bubble to the middle as something I’d consider worthy of consideration without corroboration from other sources. In other words, this isn’t of much interest to anyone but the so-called ‘fanboys’ of either side in the ATI/NVidia debate.

Was HardwareAnalysis even that significant a site in the first place? I’ve never even heard of him before.

I don’t think so. I imagine they’d be a tier 3 or tier 4 level site. Anand and Tom’s are tops, followed by FS, ExtremeTech and HardOCP as far as I know. But my involvement with hardware is minimal.

:roll:

(Not at Jakub, but at this whole incident).

How soon we forget.

This has happened numerous times in the past. Small site leaks info and benchmarks on unannounced products. Firestorm ensues. Certain major sites get snippy. Hardware manufacturer gets defensive. Competitors make hay.

Real product ships. Performance isn’t much like the leak – usually better, occasionally worse.

What’s different today is that everything on the web these days seems to be under a microscope. Search engines allow even tiny sites to have a big impact. It’s not a bad or good thing, it’s just the way things are evolving.

Someone should tell everyone to step back and take a deep breath. But that’s unlikely to happen in today’s rather superheated atmosphere.

To paraphrase Humphrey Bogart, this don’t amount to a hill of beans. It will be forgotten within a few weeks.

I’m coming to the conclusion that the 7800 and the rumored resualts of the R520 just aren’t that big of a deal. Small speed increases for huge costs.

The 7800 is a good 40% faster than the 6800. Granted, you won’t see that at lower resolutions where everything is CPU-limited…

You also gave to remember that increasingly we are being locked in by the native resolutions of our LCD monitors. Even then, 300% the cost for that 40% increase. I’m not one to cry cheap but it seems the past two generations were a bit larger of a jump.
.

No different than Mustang owners vs. Camaro owners. It’s an inherent need for many people to justify the choices they’ve made and/or identify with a tribe.

As someone who was attacked personally on the Intarweb numerous times for simply having an opinion or, God forbid, benchmarks that didn’t match up with their world view.

As for the validity of sites – hey, ever since Tom’s Hardware posted bogus bullshit benchmarks for Q3 based on broken code in Q3Test it’s been hard for me to take any of them seriously. There are a handful of sites that I still respect – SilentPC Review, AnandTech, Arts Technica, and ExtremeTech.

any company (except Creative, of course, but we’re talking CPUs and video cards here, not soundcards).

I will say that I hate Creative, but for what I consider completely objective reasons, not because they ran Aureal or whatever out of business.

You are correct, sir. For people running 19-inch LCDs with 1280x1024 displays, there’s little reason to get a 7800GTX when a 6800GT (or maybe even worse) would be sufficient.

But with 20-inch and higher LCDs coming down in price, the desire to run at 1600x1200 to 1920x1200 increases.

As for the way this article was written, it’s practically asking for a libel suit. ATI would likely lose, but it basically establishes motivation and if it could be proven that the benchmarks were intentional hoaxes (like from an overclocked X850), yikes.

I do think we should all write more reviews out of spite. You know, since Vivendi didn’t let me go review FEAR like they let PC Gamer, I think I’ll review it today based on never having played the final! That’ll show 'em! I’m sure readers will understand and care!

But to match what this guy has done, you have to review the game without ever actually touching it. C’mon, Steve, you know you want to. :P

No different than Mustang owners vs. Camaro owners. It’s an inherent need for many people to justify the choices they’ve made and/or identify with a tribe.[/quote]

Grouping instincts are so automatic as to overide reasoning for many people. One thing you just can’t depend upon is rational thinking even in clever people. :)

Fanboy or Fanboi is a term used to describe someone who is utterly devoted to a single subject or hobby, often to the point where it is considered an obsession. The term originated in comic book circles, to describe someone who was socially insecure and used comics as a shield from interaction, hence the disparaging connotations. Fanboys are often experts on minor details regarding their hobbies, and they take these details extremely seriously. The term itself is often used in a derogatory manner by less serious fans of the same material. Nevertheless, self-labeling usages of the term have been noted; in the songs of the fannish parody musician Luke Ski, many characters proudly consider themselves fanboys. The term is usually applied to people in their teens or 20s. Within this group, common objects of deference for fanboys are TV shows, movies, anime, cars, video game consoles, video games, operating systems, MMORPGs, and software companies. The female equivalent is sometimes called a fangirl. Fangirls tend to be more devoted to emotional and romantic aspects of their fandom, especially shipping.

Stereotypically, fanboys are attributed with a sycophantic devotion to the creators and principles behind a work they are currently enthralled by, but will quickly move their attention elsewhere once something better or just newer comes along. A good example of this is Harry Knowles and his associates at Ain’t It Cool News, whose particular focus is on movies in the action, fantasy, adventure and superhero genres. Fanboys are noted for a very emotional attachment to their chosen subject, often taking negative remarks about it as a personal attack. They will readily engage in debates, but will fall back on emotional responses. For example, a “fanboy” may go out of their way to point out negative and often untrue statements about their obsession’s rivals; this is commonly known as FUD.

The stereotypical image of the fanboy is as an unkempt, often overweight, and generally unattractive figure of a young man who appears as a loud mouthed pseudo-intellectual. A popular depiction of this stereotype is the Comic Book Guy on The Simpsons. In the mid 1990s, the emergence of slacker culture changed this image somewhat. Movies such as Kevin Smith’s Jersey Trilogy (Clerks., Mallrats, and Chasing Amy), altered the image of the fanboy. As electronic entertainment gained popularity, the term became increasingly applied to video gamers and television addicts. As a result, a subculture emerged which readily labeled itself as fanboys. Within this group, more effort is taken to fit in with a perceived standard, resulting in a more outgoing attitude, even among those with insecurities. This outgoing stance is often felt to be abrasive by those not involved in the culture as deeply."


I wrote and edited hardware reviews for my large enthusiast-level gamesites many years ago, starting pre-3dfx and ending around the geforce2 days, so I have no recent experience, but…

Everything on the web has always been under a microscope. That hasn’t changed. Hardware vendors are still shady and deceptive, although ATI has really cleaned up their act over the years. ATI used to be total scumbags. The knowledge requirements to be a hardware editor are not excessive nor have they ever been. The setup costs are minimal. Shit, I used my desktop PC back then, these days you can buildout a test box with a pack of gum and $2.50 in quarters. Establishing trust with manufacturers? What the hell do they care, they just want eyes to read the gushing reviews. Used to be, if you’re a really successful site you’ll get free hardware. Used to be that smallish sites could still get hardware but only if they compromised their integrity and gave good reviews. I assume both still apply. If anandtech says your new card is shit, they won’t be banned from future product.

Thing is that hardware reviews are fucking boring. Both writing them and reading them. I must have done two dozen voodoo3 reviews, no exaggeration, and every card was identical to the fucking reference. I have an entire wall of voodoo2’s, tnts, tnt2, riva128s, etc, in my parent’s garage. They’re all the same damn card. That hasn’t changed either. Every card that deviates from the reference is either worse than the reference, or vastly more expensive but infinitesimally better. Who gives a shit? It’s obvious what to buy given your budget and degree of technolust. There are only four interesting reviews per generation, the topend ATI card, the topend nv card, the best buy ATI card, and the best buy nv card. What do you review the rest of the time? Addon copper heatsinks and bundles. Who cares?

You’re absolutely right. I think it’d be hard to write for a hardware site; I’d probably want to do as many tutorials and system building stuff as possible to keep from reviewing the same thing over and over again.

Fortunately with doing a game magazine, I can just do those high-end and mid-range cards, and the occasional noteworthy system (ones that are stupidly high-end, or good performers for little cash).

Well as the author states, there’s an NDA on reviews. I don’t know of any legitimate press that has samples from ATI yet. The author states they got their scores from other sources, but never states what those are:

“Unfortunately for ATI we have other sources to get detailed information from, for example the board partners that will be manufacturing and selling the majority of these new graphics cards.”

What if they’re from an Nvidia-only board vendor like BFG? Or Nvidia themselves? They don’t reveal the source of this info, the driver revision, or the details of how any of the benchmarks are taken. What’s this “custom benchmark” for Half-Life 2? Is it one of the most shader-heavy levels, or one of the least?

The Splinter Cell: Chaos Theory numbers are the most dubious - in all my testing, using Shader Model 1.1 path or Shader Model 2.0/3.0 path and HDR, parallax mapping, etc, the X850 XT outpaces the 6800 Ultra by a pretty good margin. Especially at high resolutions. They’ve got the X850XT running slower than the 6800 Ultra across the board, which makes me really wonder about their supposed R520 scores.

I can understand that this guy is trying to “scoop” everyone, and I think he’s kind of trying to get on ATI’s list of hardware sites they work with under NDA and stuff: “To ATI we’d like to comment that it is a two-way street really, we’d like to be briefed about your products and have early access to them, if that requirement is met we will have no issues with signing and adhering to NDAs. By cutting us off however we’ve just been given a new incentive to work harder to break the news when we get a chance, in our usual no holds barred fashion.”

I would take these numbers with a big grain of salt, and only add that:

  1. This is probably not the way to get in ATI’s good graces and get NDA info out of them.

  2. Graphics cards are about more than just 3D benchmark performance. There’s the feature set to consider.

What I hate is having one page of subjective information and then thirty pages of clicking through every fucking conceivable benchmark, very often with differences that are statistically insignificant but apparently define your manhood to geeks.

I hate motherboard reviews for the same reason. Hey, great, I can overclock it an extra 3%, but thanks for forgetting to mention that the power connectors are in a really shitty location, the drivers suck and are never updated, customer support doesn’t exist, you have to go through your OEM for driver updates, the fan is 98db, and it locks up every time you attach your new USB mouse.

Go look at the magazine reviews of the Linksys A+G wireless router. Inevitably 4/5 or 5/5, Editor’s Choice, etc. because they give it a 20 minute workout and that’s it. Now look at user reviews – 2/5 or 3/5, with complaints about support and compatibility, lousy firmware, random disconnects, etc. This is why I’m starting to trust epinions and Amazon reviews more than magazine reviews of hardware.

The complete lack of emphasis on stability has bothered me forever. I was using memtest86 as a benchmark of stability years ago and remember calling out some people on this and they poo-pooed my worries. When your idea of objective stability is “Windows booted and didn’t blue screen after an hour”, everything else is pretty much suspect.

There are only four interesting reviews per generation, the topend ATI card, the topend nv card, the best buy ATI card, and the best buy nv card. What do you review the rest of the time? Addon copper heatsinks and bundles. Who cares?

There are also the “quiet” cards, which I think are interesting. People liked to read about the Zalman fanless heatsinks and the Arctic Cooling things.

EDIT: Extarbags’s comments made me self-consciously fix my grammatical fuckups =)

[quote=“BaconTastesGood”]

What I hate is having one page of subjective information and then thirty pages of clicking through every fucking benchmark that’s conceivable, and very often with differences that statistically insignificant.

I hate motherboard reviews for the same reason. Hey, great, I can overclock it an extra 3%, but thanks for forgetting to mention that the power connectors are in a really shitty location, the drivers suck and are never updated, customer support doesn’t exist, you have to go through your OEM for driver updates, the fan is 98db

Go look at the magazine reviews of the Linksys A+G wireless router. Inevitably 4/5 or 5/5, Editor’s Choice, etc. because they give it a 20 minute workout and that’s it. Now look at user reviews – 2/5 or 3/5, with complaints about support and compatibility, lousy firmware, etc. This is why I’m starting to trust epinions and Amazon reviews more than magazine reviews of hardware.

The complete lack of emphasis on stability has bothered me forever. I was using memtest86 as a benchmark of stability years ago and remember calling out some people on this. When your idea of objective stability measurement is “Windows booted and didn’t blue screen after an hour”, everything else is pretty much suspect.

There are only four interesting reviews per generation, the topend ATI card, the topend nv card, the best buy ATI card, and the best buy nv card. What do you review the rest of the time? Addon copper heatsinks and bundles. Who cares?

There’s also the “quiet” cards, which I think are interesting. People liked to read about the Zalman fanless heatsinks and the Arctic Cooling things.[/quote]

Oh man, this post should win an award.

It’s interesting a software developer might forget that there’s a big difference between one person testing something, and their observations, and those of hundreds of thousands. Sorta like software bugs.

There are also the “quiet” cards, which I think are interesting. People liked to read about the Zalman fanless heatsinks and the Arctic Cooling things.

I like to test quiet stuff as well, though I can never tell how much readers care about how noisy their PCs are.