Insight into the QA process
View Single Post
Join Date: Dec 2007
04-04-2010, 12:20 AM
While QA is generally the "gate-keeper" at most companies, there are problems with this in an MMO environment.
I think that sometimes people forget how massive MMOs are. There are many massive software applications across various companies in the world (working for a large financial services company I've worked on several of them in a QA role), but very few of them compare to the complexity of an MMO.
In my own personal experience with QA and our standards, we had test cases for EVERYTHING in the application. Can you imagine the size of a test design for an MMO? It would be near impossible to put one together that covered 100% of everything. The same is true for most applications.
Test designs try to cover as much as possible and help to break testing out for critical parts of an application. Usually this works pretty well.. but there are other factors at work.
For example, you couldn't hire enough people to cover a regression test of every build. It's simply not possible. The cost vs. coverage would be so insanely high that the company would go broke. From past experience I know that QA is always being told to do less with more, and then turn around and get hit when they miss something that goes to production.
So generally QA will test functionality given to them by the developers in the release notes. The problem from this point on is making sure that the fix didn't break other things. With large complicated systems it's hard to find a bug that's totally unrelated to a change but cropped up as a result to a fix.
Most QA Analysts are the SMEs (subject matter experts) for their applications and some know more about the application than the developers do themselves. They support the app, test it and know the functionality in and out. However, even those SMEs can miss a bug. They're human.
So remember that it comes down to:
1. MMOs are more massive than most applications. More complexity means more chance for bugs to appear.
2. Time is finite.. it's not possible to do a full regression test every build.
3. QA is reliant on Devs to give accurate information on which bugs were fixed and, if possible, additional areas for the analyst to check.
4. Developers will unit test their changes, but generally in a small scope. QA is usually responsible for large scale testing of changes and no matter how well they understand the system a bug can still creep it's way in.
5. Fortunately for MMOs with test servers, a large number of players can quickly find bugs that the QA analysts may miss. The shotgun effect is very effective at finding those sneaky errors.
6. Just because a bug is found doesn't mean the build won't go live. A few simple bugs usually won't keep a build from going live because the benefits almost always outweigh the negatives of holding it back.