Quality Assurance in the Games Industry

After reading this piece about QA in gaming:

http://www.ntsc-uk.com/feature.php?fea=QualityAssurance

and realizing that gamers are constantly being fed SHIT that there must be some way to develop an online organization that tallies all the SHIT-BUGGED games that have been released by developers.

Almost like the weekly game sales listings that we see from NDP Intellect.

But this online organization should carry some kind of weight from the consumer market… I am not talking about a game review site… something that is concise that shows the track record of development houses that release buggy games…

EDIT: I guess I am referring to some kind of quality rating system for each game, that give consumers some idea about how stable/worthwhile a game is and give devs incentive to produce less buggy games.

suggestions?

Firstly, testers need to do more than just test for bugs. Checking for programming errors is one thing, but ensuring the game is fun and enjoyable is quite another and the two must co-exist.

I take issue with this passage because I don’t think it’s possible in the current state of the game industry (at least as I’ve experienced it). A “lowly tester” taking issue with game content is also known as a “C Bug” or “a fight with the Dev team” (a fight that a tester won’t win). Reports written by project leads to improve the general status of the game get whittled down into a few random fixes here and there that don’t actually resolve the problem. And the few times that I was sent to a development house to help “make the product better” - I was not exactly welcomed with open arms by a Dev team who knows exactly what’s wrong with their project and that they already don’t have the resources to eliminate the issues.

The problem is that there is too much of a demand for product to be thrust out into the marketplace ASAP (by the powers that be, I’m not sure about the consumers) and not enough demand for quality. So while I agree with you that there should be some kind of Better Business Bureau in the game industry, there is too much inbreeding and too many different places that blame can be spread for there to be a fair individual rating system.

how about an independent audit body… much like the ISO 9000 registration for operations or the ISO 14000 for environmental issues. ISO standards work pretty well for publicly traded firms from car manufacturers to hardware manufacturers…3M to Ford to Microsoft…

if the ISO registration process works fairly well for real corporations internationally, then something similar (but on a smaller scale) should function well for games.

the shareholders and senior managers of each software company WOULD take notice and want to have their companies ISO 20000 registered (for instance)… and if they continue to produce buggy games an audit will strip them of the ISO standard resulting in jeopardizing shareholder wealth.

fuck, I am brilliant.

any holes you can poke in this idea?

Until everyone uses the exact same hardware configuration, and shareholders stop demanding that games be out in time for Xmas come hell or high water, and consumers stop buying buggy games, none of this will happen.

Step 1: Think about ISO body for games.
Step 2: <mumble mumble>
Step 3: Profit!

If everyone would be willing to wait an additional 6 months or more for each game release then some kind of ISO certification might be feasible. No, I’m not joking.

The state of game testing is simply atrocious, but it’s just an extreme symptom of the black hole that’s the general state of software testing. We’re still in the phlogiston era when it comes to testing; little or no theory, only a few people pushing better practices, not much in the way of accepted frameworks, no feasible way to conclusively demonstrate that something will work; it’s just a general nightmare.

Gamers appear to have a limitless tolerance for horribly tested games, so the gaming industry probably isn’t going to improve until there’s some general breakthrough in testing.

The complete non sequitur in the first two sentences of that article didn’t exactly hook me. The author’s thinking is as bugged as the games.

Game QA will never be quite where it needs to be until publishers realize that paying one skilled test engineer is cheaper than six testers. I seriously believe that investing in test engineers would be cheaper in the long run, while producing higher quality products, to boot. It must be a byproduct of nobody quite taking the industry seriously (after all, we aren’t making pacemakers, right?) that formal testing practices are kinda unheard of.

Also, in my experience, publishers frown upon developers having internal QA departments – nobody wants to pay for them, yet they are more useful because they’re closer to the developer and have worked with them for a much longer period of time. I guess it all boils down to the publisher seeing a duplication of effort, since they have their own testers, and they think that they can do it better…

That being said, the article is off in la-la land. Only the console makers themselves have enough power to regulate what gets shipped, and they’re too concerned with the blood war they’re fighting amongst themselves to worry about issues such as game creativity. Not to mention the fact that testing a game and critiquing a game are two skill sets that don’t often intersect in game QA…

We can call it “Isonews”

If everyone would be willing to wait an additional 6 months or more for each game release then some kind of ISO certification might be feasible. No, I’m not joking.[/quote]

Gamers would be willing to wait, especially if the PR process didn’t ramp up until six months before release.

I’ve actually thought about starting a Game QA buisness. There are alot of things that inhouse QA just cannot cover. The 8 billion system configuations for example.The 500 doohicky controllers that people use for another.
An out of house look will also give an unbiased opinion of the game, which from experience, I don’t think you get that (or isn’t listened to) by an in house QA group.
QA in games has to be more than P1/P2 bugs. QA also has to be the critic. This is a hard thing to do if the guy you are bashing is your beer buddy.

Here is a short QA story:

QA Tester: We need bindable keys.
Dev:We use WASD! Everyone uses WASD!
QA Tester: I don’t. I’ve been playing Doom/Quake for years and I never use WASD.
Dev:We are not Fucking Quake!
QA Tester: But the guys buying the game play it, and expect to be able to set up how THEY want to play!
QA Tester:Oh and inverted mouse too.
Dev:Our last shooter used WASD
QA Tester: And that sold what? 20k copies?Look, No bindable keys and I bring the game back period.
Dev:@#@#$@#$@# and general abuse.
Yes this did happen. Not the best 1st days on the job for me, but I did get my bindable keys, even the ususal overlooked ALT and ;

Another helpful tip for any game developer houses out there- don’t hire teenagers. Over 21 only. QA has to be serious, and not waiting for the next break to play something else. BUT QA should be allowed, no, forced to play any and all games in the same genre and bring ideas to the dev team about what they find. Yes QA can also be a research department.
I’ve done QA in games and out of games. Sadly the out of games experience has been better- and actually pays well-more than double of my last game developer job. But the non games company has learned from bitter experience that bugs cost money.

Bingo: “The non-games company has learned that bugs cost money.” But in the games business, that really isn’t the case, or it doesn’t seem to be.

We get poorly QA’d games because in the end QA doesn’t matter to sales, or so the publishers believe. Who’s to gainsay them, when visibly buggly messes still sell as well or better than polished, superior products? There does not seem to be much correlation between quality (considered rather mechanistically) and sales.

I suspect gamers are to blame. Fans make up their minds long before the games are released, and have invested so much verbiage/energy/face into boosting games that when it turns out to suck they can’t afford to back down. Or, if that’s too harsh, perhaps it’s a matter of gamers not being able to resist cool-sounding/well-previewed titles regardless of the wisdom of waiting to see what the word of mouth and review cycle brings. Dunno. But whatever the reason, bugs don’t seem to hurt sales enough to get publishers worried.

Wombat- true, but I don’t buy the next game from a company that had me oh say HAVING TO RESTART THE GAME because the patches break save games- Fallout II comes to mind. That was the end of my Interplay love.
You will loose some sales in the future, and the costs saved by not having to keep programmers on patch detail is worth the small extra cost of QA.

I suspect the real loss is when you get casual or new gamers frustrated by games that won’t run properly or at all. Some probably bail on PC gaming forever.

The PC game industry should come up with estimates for lost revenue due to bugs just like they love to come up with estimates for lost revenue due to piracy.

Gamers would be willing to wait, especially if the PR process didn’t ramp up until six months before release.

Games are unique because people expect them to be out IMMEDIATELY upon completion. Movies, books, and music schedule releases based on other market considerations and the completed product may sit around for months awaiting a proper release window.

Consoles are generally better about this, particularly ports of foreign games. FF X-2 was release in Japan in March and probably could have come stateside a lot sooner than November.

How would that work?

I’m not a QA engineer, I’m a project manager, requirements analyst, and developer in an IT services firm. All of the formal test methodologies I know of start with testable requirements. They verify that the system as built complies with some predetermined standard of success.

“The system shall provide a means for the user to enter the billing date in one of the following formats…” That’s a yes-or-no kind of testable requirement. Is the billing date field present on the screen, and does it accept the proper formats, and does it reject obviously bogus values? If not, the system does not meet required functionality.

What does the formal test process buy you if there is no formal requirement spec to test against? How do you capture anything but the simplest of games in terms of formal requirements? And why would a formal methodology catch any more bugs than the informal ones do now?

My beta test team (speaking now as a game developer) is always finding situation-specific bugs that are nearly impossible for them to reproduce or even characterize. “I had been playing the game for half an hour and I was in this one mission where I shot the enemy destroyer with my lasers and then I went to launch a torpedo and the game crashed.” Can’t reproduce.

These kinds of bugs are usually due to something weird. Another enemy ship was destroyed by your wingman just at the moment you fired your torpedo and the weapon guidance system tripped over a stale pointer in the enemy ship list. Happens once every ten thousand torpedo launches on average. How would a formal test methodology help me find those weird can’t reproduce bugs any faster?

–milo

Doug and I were discussing this the other day. Example of how lousy the “science” of game testing is - theoretically, it should be perfectly possible to write automated utilities to verify the output of your game under different drivers; there’s plenty of tests to detect clipping errors, etc. Does anyone do it? No.

I suspect it’s more B (“not being able to resist”) than A (“make up their minds/invested so much”) in the sense that for the real sales numbers, I don’t think the online fan-driven side of things - or anything online, including reviews - is anywhere as important as the old-fashioned marketing side. The movie tie-ins, sequels, pandering to pop culture, etc. I don’t even think the magazines have a huge impact compared to this.

Of course I have no numbers and could have my head firmly inserted in my third point of contact.

It is possible, but it’s not easy and it doesn’t help with plot bugs and bugs of that nature.