Why Vista took so long to build

Maybe your testers don’t contribute to the design. :)

Some of us prefer to have a high ROI…

I’ve never been in a situation where testers contribute to the design significantly, except insofar as saying that some feature may not be testable as designed. In the OP, there’s already a UI expert and a user experience person. I think that if you throw 3 QA people into the mix, you really do get a mess.

Not that QA can’t contribute–sometimes they have a broader view of possible conflicts with other features than the developers. But in my experience (both as a dev and as a QA person) it’s fairly rare.

Let’s take this shut-down menu thing as an example. You’ve already got two UI people, who would presumably trump the QA people if there’s a disagreement about the UI (otherwise why the hell are you paying them?). You’ve got 2 devs who presumably trump QA when it comes down to the actual nitty-gritty of the code design (b/c it should be a black box for QA). You’ve got a PM who actually decides what the features should be implemented, and who also (hopefully) has the broad picture of how this feature fits into the product.

QA’s job here is to ask questions related to their testing that might clarify holes in the design (“What happens if the user presses two shutdown buttons in quick succession–which kind of shutdown wins?” “What if we’re in the middle of operation X and we do a shutdown?”) and that will become part of their testing. To me, that’s how you get a good ROI from your testers–by focusing on their job.

As a related piece of trivia, someone on JoS pointed out Mark Lucovsky’s PowerPoint presentation on the evolution of Windows development from the fresh start on NT 3.1 to the introduction of branching source control for Windows 2000.

It’s really dependent on the structure of the group. Some groups the devs design everything and the PMs are train conductors. Some groups the PMs do it. Some groups the business development people do it.

Often, however, stuff like “for chrissakes, none of the customers I’ve talked to want a dozen buttons to choose from” isn’t on anyone’s list of things to do; test is in a unique position to take care of it, because the only thing they’re on the hook for officially is that the product works like customers want and expect. “Customers are exasperated by the feature” is going to make the product suck more than some edge case about wacking two buttons quickly.

The user experience people are on point for catching this sort of thing, but they’re often a spread thin catch-all for a dozen projects. The UI people I can see, but it’s not their focus to eliminate feature points people don’t want.

If your QA/Test people are just exploring the FSA for shit that breaks, I think you’re leaving money on the table.

What I dislike about this whole shebang is that I have to get the OS for work, but I don’t see myself really being able to trust it until the first service pack comes out. These kinds of stories don’t hearten me.

That’s generally been my experience, too. They’re involved primarily so that they can hunt down the vague “improved performance” requirements and make us clarify how to quantify it, what we can do to help them measure it, what it’s relative to, etc…

We’re trying to move to a model where QA is treated more like a customer of the product, though. They don’t define requirements directly, but their input gets factored into the designers’ long-term plans like any other regular customer’s feature requests.

Wait, what?

The QA/beta testers feedback is considered an “optional” fix?

Hey guys, the system crashes when you press these 3 keys at the same time.

Great, hopefully not too many people will push those 3 keys at the same time and maybe at some point in the future if we have time or feel like getting around to it, we’ll fix it.

Who do you work for?

I should have clarified that I was still talking about the design phase. They still hold the ‘this does/does not pass acceptance criteria’ hammer at the actual QA phase, of course.

It’s a tough call, although I think that the two UI people should be doing that. Otherwise why are they there?

I’m in a bit of different boat right now–I’m a QA person, and my particular team’s job is to act like customers (there are other teams who have more specific jobs trying to find edge cases). And we have no UI people. Even so, it’s very difficult to say “customers don’t want this feature” at the design stage–because the PM can say, “actually, we’ve checked with our 12 biggest customers and 9 of them want the feature.”

And, given the cost to implement any feature, I’d go with the PM over my own judgement on something like that. Sometimes, of course, a feature turns out to be confusing in the context of the whole app, but that’s usually something you have to at least have a basic implementation of before you can see it. (I’ve been sure something would be difficult to use, but when I actually came to test it, it was plain sailing).

As it happens, my company doesn’t write a UI-sensitive app, so maybe my approach is a bit different than the MS one. But I think that 8 people in a meeting, all sticking their oars in, would make for a fractious meeting–so if all those people are actually “contributing” to the meeting, I’ve got to agree with the OP that it sounds like a mess.

If they have customer data supporting it that’s a whole different ballgame, yeah. And nothing can be done in a meeting with more than 4 people or so.

My team has been obesssing about framing things in value propositions as of late. It seems a bit more useful than arguing over line-item features.

I kinda disagree with that. I’ve been in meetings (not in the game industry, but still) with more people than that.

To take an example off the top of my head…

Working out budget for a show at the Theatre I worked for, the meeting included the Director, the TD, the Master Carpenter / Shop Foreman, the Costume and Lightning Designers, the Props Manager, a lady representing Marketing, and a couple other people too.

Tons got worked out and decided on, everything that needed doing in fact got done, and a good time was had by all.

So it is possible to get stuff done in large meetings, just takes leadership and a willingness to stick to the agenda (imo).

From what I’ve heard, this is exactly the problem that made the original Longhorn untenable and caused basically an entire project reset - at the cost of quite a few of the original features.

Every interview I have read makes it seem like MS has learned their lesson. Vista took a big bite in moving more things into less inter-dependant components that run in user space instead of hopping back and forth to kernel space all the time. It was a bitch for Vista, but it should allow MS to iterate certain OS features more quickly without screwing up everything else. It’s not in an ideal state, but it’s better than the big monolithic monster it used to be.

Interviews with Ballmer and Alchin and the like make it sound like they’re going to go for - and they would never use these words - a more Apple-like approach to release schedules. Apple like to claim that they shipped 5 OSes in the time it took MS to make one, but that’s not really true. Apple has made five updates to OS X - a single OS - much like Microsoft has made two versions of Windows MCE, WinXP 64-bit, Windows 2003, etc. It’s just that it’s Apple’s “mainline” OS, y’know?

So the idea for MS is to make an OS update every 18-24 months, with more modest architectural changes each time around. There may be a lot of little end-user experience things, GUI work and the like, but low-level changes won’t be so grand.

The trick is going to be for them to avoid the “Apple Tax” where you have to upgrade your OS for $120 every damn year (which business and IT guys really hate) and to also avoid the Windows ME syndrome, where the new OS is hardly really new at all. It’ll be really interesting to see where Windows goes after Vista.