Quality Assurance in the Games Industry

If everyone would be willing to wait an additional 6 months or more for each game release then some kind of ISO certification might be feasible. No, I’m not joking.[/quote]

Gamers would be willing to wait, especially if the PR process didn’t ramp up until six months before release.[/quote]

Publishers probably wouldn’t be willing to, though. You’re talking about adding on hundreds of thousands of dollars to the cost of every game.

If everyone would be willing to wait an additional 6 months or more for each game release then some kind of ISO certification might be feasible. No, I’m not joking.[/quote]

Gamers would be willing to wait, especially if the PR process didn’t ramp up until six months before release.[/quote]

Publishers probably wouldn’t be willing to, though. You’re talking about adding on hundreds of thousands of dollars to the cost of every game.[/quote]

Raph, publishers will have to though IF it became a standard. I think that is what we are arguing here.

It wouldn’t be a standard unless the publishers wanted it to be. :) That’s kind of what I am getting at.

More importantly, gamers have proven they’ll buy any slop you shove in front of them, so there’s no economic pressure to do so.

They couldn’t do that because it would lay responsibility were it belongs… :shock:

…well, partly anyway.

http://www.aqinc.com/

http://www.aqinc.com/[/quote]

I can’t see this working well in our case. We have an internal QA department that can walk down the hall and talk to a designer or programmer. You can’t beat that.

http://www.aqinc.com/[/quote]

I can’t see this working well in our case. We have an internal QA department that can walk down the hall and talk to a designer or programmer. You can’t beat that.[/quote]

internal activites are often politically manipulated… I have no faith in internal QA activities… look at the game industry in its current state. And IDaveC, no offense, but your company’s games are not a good example of high quality QA!!!

besides, it is often difficult to reproduce what EXPERIENCED outsourcing firms have been doing for years.

furthermore, there is more brand recognition when QA is outsourced as opposed to being an internal activity…

that’s just my 2cents

SolomonGrundy, I agree there are always some lost sales. But if the wastage was significant companies would have adapted by now I think. The fact that companies continue to churn out highly buggy software indicates, to me at least, that there is very little downside for them.

Haha, how many people here remember MAX2?

  • Alan

Nintendo has some of the strongest QA in the industry. It’s all done in house. They built their name on the back of nearly bug free software. It pays to have good people in house that you retain from project to project IMO.

–Dave

Del, hate to contradict you but you are really reaching on this. Give me some kind of evidence that shows that external QA is any kind brand feature. Are you proposing:

[size=6][color=red]Dolby Digital ™, Miles Sound System™, Bob’s QA Farm™[/color][/size]

on the box?

As for our QA I will and have admit that issues have got by them, but they are still one of the best QA departments I have ever run across. Our games are far more complex than most. Hell critical path in most games can be done in an afternoon, whereas in our games that’s one chapter. Also, I’d like to know where your experience with internal QA politics happened? Last time we corresponded you said you were a doctor.

Yeah, but that’s really only feasible if you’re at a relatively big developer with multiple projects going at once. That’s not the case with us here at Day 1 (one project, ~30 people). Fortunately we have access to Microsoft’s testing department when we need it, but I wish we were in a position to have access to in-house QA full time.

As for Absolute Quality (also joshingly known as “Absence of Quality”), I think their big selling point is their ability to get a console game through TCR/TRC fast. For small publishers up against hard deadlines, spending a few grand extra to trim a couple of weeks off of their cert time is a no-brainer.

Don’t fucking remind me. I tested that game.

Functional requirements testing works for games. It sounds dumb, but it’s very worthwhile as far as testing UI goes, not to mention game flow/campaign flow/mission flow. Cheat codes, certification requirements for console games, all kinds of things like that.

At a lower level, it’s useful for developers to do functional testing on bits and pieces of the game. For example, testing that navigation AI can successfully travel between waypoints is a perfectly useful test, and there are lots of other useful examples I could think of as well. The bonus, in my experience, is that (automated) tests like these tend to catch subtle, unexpected changes in behavior too. (i.e. something stops working all of a sudden because of a change in apparently unrelated stuff)

Oh, you can’t test everything, and you can’t quantify gameplay as tests – I’m not saying that anyone should try to do so. But there are plenty of things that are peripheral to the game design (but absolutely essential to the finished product) that can be tested…like UI, game rules/correctness, etc. It’s very much worth it to test these things, because even though they may seem simplistic, a) hey, sometimes you screw up the simple things, and b) sometimes they can point to larger problems.

Informal testing processes are terrible because everyone falls into a pattern about how they play a game. Everyone has their set way that they use the UI, advance their character, etc. and unless there’s some kind of test plan there, most people will, consciously or unconsciously, do the same things over and over. I’ve seen this happen repeatedly, both with developers (“it works fine on my machine!”) and testers, and I really feel that a well thought-out test plan helps immensely with coverage testing.

An alternate way of looking at things: without an organized test plan, you may find the exact same number of bugs, but way later than you would with a decent test plan. I’d prefer the level of confidence that comes from having a good test plan that’s consistently executed rather than pulling out my hair wondering constantly if everything’s still working. Games, as I’m sure you’re aware, are complex beasts now.

It won’t help you find them directly, but it does give you better odds at knowing that something’s wrong, especially if you have automated tests that produce consistent results. I guess my line of thinking here is that the more often you’re running the simulation, in realistic scenarios, the better the odds you have of finding such “random” bugs. It’s not necessary for someone to be at the controls to get the game to crash.

All this, of course, is separate from the idea of having better testing and test support at the coding end of things. Whole separate rant. :D

No offense, but it would not work well for video games. I worked as a tester for a ISO 9000 company and we were audited externally only annually. Two weeks before our ISO 9000 inspection/questioning, the management would go bananas and start making everyone study to ensure that we’d pass even on individual questioning. I made sure I knew what I was doing all year round, but by my second year, I began to see what a joke it was. There were our standards we had to live up to, and then there’s the ISO 9000 standards to live up to. Basically really just once a year. So the difference between our standard and the official audit standard was a means of profit for the company.

ISO 9000 standard practices wouldn’t make much of a difference in the games industry. You’d have to audit every single game to make this worth everybody’s while if you want accountability.

Hee. I saw these two threads juxtaposed in the index,

[ul][li]The return of Leisure Suit Larry?![]Quality Assurance in the Games Industry[/ul] and read them together as:[ul][]The return of Quality Assurance in the Games Industry[/ul]I thought this would be a very good idea. ;)
[/li]
.
.
.

[size=2](Yes, it’s a joke. I know that companies are trying, sort of. But too many buggy, unfinished products still find their way to store shelves. As a (non-game) software developer, I appreciate the difficulties–but as a consumer, I find the situation frustrating.)[/size]

Don’t fucking remind me. I tested that game.[/quote]
As did a number of others here on the forums (myself included). I doubt anybody else would even bother to remember it, proving that gamers really do have a threshold for what they’ll accept as far as bugs are concerned. The main problem is that really, gamers will put up with a lot of crap. I can only hope that as the console generation grows up and starts buying PC games, they will have less tolerance for shoddy QA work.

  • Alan

I (and hopefully the other people who tested it on this forum) lost all interest in purchasing it once I saw that they weren’t fixing a damned thing. Beyond even that, the methodology of their testing was patently fucking ridiculous. I’ve honestly never seen such a thing, in years of amateur and professional beta testing. “Okay, today you’ll be testing the helicopters. Tomorrow, you can test the boats. The next day, you can test some ground units, but only the ones with two legs and green boots, not the ones with brown boots. Those won’t be in the game.”