Lt. Commander
Join Date: Dec 2007
Posts: 120
# 1 Insight into the QA process
03-11-2010, 06:54 PM
Hello Everyone,

This response was originally part of this Thread, after significant request to have it stickied I decided it would be best to create its own thread in the Federation News Network.

-------------------------------------------------------------------------------

First off, I want to make the following explcitly clear, Cryptic Studios does not in any way "expect" our users to submit bug reports. This is an entirely voluntary process provided so that those users interested can submit both Feedback and any Issues they encounter.

-------------------------------------------------------------------------------

Now that we have that out the way, let me take some time to provide some insight into how QA (Quality Assurance) functions in general within the gaming industry and here in-house.
  • QA process differs from company to company, especially between a Publishing and Development house.
  • There are many types of QA: Game QA, Infrastructure QA, LIVE QA, Dev Support QA, Standards QA, and many more depending on where you work.
  • Each QA team generally has a specific area they focus on and do not detract from this unless there is specific direction or special circumstances to do so.
  • Resources are always limited: Staff, Time, Budget are just a few of the bigger examples.
  • QA most often, will not get the final say as to when something is released and in what state.

Those are rough basics you will find at most companies which maintain QA. That of course does not even delve into companies which utilize off-site (external) QA sometimes referred to as blackbox testers.

-------------------------------------------------------------------------------

Alrighty, with that, let us talk a bit about how we handle tickets from A to Z.

We encourage all users to participate in the voluntary process of submitting either feedback or issues (bugs) you may find on your travels through the Alpha Quadrant.

Once an issue goes to our C.S team, they collate all similiar issues and take as much information as they can in order to formulate an ideal bug to then forward off to DEV directly or in most cases QA to verify.
Tip - The more an issue is reported the more attention that issue gets
Once the ticket is in QA's hands it enters a queue and is processed in turn. The issue is checked for validity and that enough information is present to accurately reproduce the issue for DEV. If there is not enough information it is sent back to C.S and they are informed we need more information before continuing.

The issue is sent off to DEV which is given a severity of how bad the issue is from the QA perspective. A Team Lead and or Producer will triage issues submitted and prioritize them for their respective teams to address. The DEV will attempt to fix his assigned highest priority issue, upon doing so will send the ticket back to us.

QA waits for the fix to enter an internal build, at this point the issue is regressed and confirmed whether or not the fix has taken place. If the fix is confirmed the issue is closed out and we wait for the fix to make it into a build destined for either Tribble the PTS then Holodeck (LIVE) or straight to Holodeck depending on the issue.

If the fix fails, QA sends the issue back to the DEV and the process repeats until fixed.

Now please bear in mind that the above described process is a gross over simplification and many details and further processes were left out as this is internal information not available to the public. However the overall information indicates clearly the stages of issue submission to being fixed on LIVE.

-------------------------------------------------------------------------------

Now, onto the issue of bugs making it into release.

As previously stated QA and the company overall have many limitations (Time, Budget, Staff, Etc.). That means regardless of how Chuck Norris we are, there are going to be regular cases where an issue finds its way into LIVE.

This is no way means QA didn't find it, or that DEV hasn't been actively trying to fix it. All that means is that when the decision is made to move forward we like the Borg will adapt.

Many factors come into play, for example:
  • Would you rather a DEV artist fix those windows on the Galaxy or keep modelling his sweet looking new ship variant being built.
  • Would you rather a DEV programmer fix that super rare crash that affects 10 people or continue optimizing code so that the game runs smoother, or add a new tech feature for Territory Control.
  • Would you rather a DEV designer fix that minor typo is his mission text or keep putting together entertaining and challenging End Game content.

Some of you will be on one side of the coin because that issue is very important to you, others will be on the opposite because they just want new stuff. No company on the planet in the whole of existence has ever been able to make everyone happy, it is a theoretical impossibility.

However I will say this, we are human just like you (unless you are a certain programmer whom I still claim is a terminator re-programmed to program). We respond to the full range of emotions that you all do, when we get told our customers don't like something we feel bad. When we hear about a change we like/dislike we stand up nerdrage and go talk and debate the issue just like you would.

If you talk to us as adults, speak politely and with respect, be concise and clear in what you are saying, have some empathy and treat us just how you want to be treated. We are all the more enticed to interact and provide insight where we are able.

This should clear up a great deal for many of you.

Always remember if you have questions, concerns, feedback or issues to submit, a good place to get some perspective and DEV interaction is IRC.

I and a few others maintain a regular presence in #sto and #stoqa. You can always PM me or participate in discussion, I will most likely be busy helping others but if you are patient I will respond to you as I am able. This also helps to expedite issues as I can walk over to the DEV and tell them personally about user reported issues.

Q speed all.
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 2
03-11-2010, 07:15 PM
#stoqa can be found on www.startrekonline.com/chat (type /join #stoqa in the chat window after it's automatically loaded you into the default room #sto)
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 3
03-11-2010, 07:22 PM
It's all about respect.

Nice post! I hope it hits home for many of those who tend to forget that like us, you are all human (except for that dev you mentioned).

One thing I'd like to say:

Please try to remember that for every rude comment or abusive attack you may read, there are many more of us who are simply too busy enjoying the fruits of your work to take the time to comment.

But for what it's worth, know that there is heartfelt gratitude and powerful hope for what you have and will achieve in this most beloved fictional world.

Thank you all again for all the hard work. It is recognized and appreciated.

-Forjo
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 4
03-11-2010, 08:56 PM
Yay! This post didn't get lost in the bowels of Ten Forward. Good job with communication lately, trust me people appreciate it.
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 5
03-11-2010, 08:59 PM
As a former QA tester myself on various things including an MMO, I can confirm that yes, this is indeed how everything works!
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 6
03-12-2010, 02:17 AM
Thanks Falkoren. The decisions you talk about are the same ones that any of us deal with in our own jobs every day. Sometiems we forget that stuff when it gets in the way of our entertainment.

Almost every day at work I have to either take part in a decision just like that or be affected by the outcome of a decision like that. When I get home and sit down to relax and lose all that stress and frustration in the world of STO it is easy to forget there are real people with the same issues at work that I have to deal with on the other end. We should all try to remember that when we are frustrated by something in the game.
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 7
03-12-2010, 03:57 AM
Quote:
Originally Posted by Forjo View Post
It's all about respect.

Nice post! I hope it hits home for many of those who tend to forget that like us, you are all human (except for that dev you mentioned).

One thing I'd like to say:

Please try to remember that for every rude comment or abusive attack you may read, there are many more of us who are simply too busy enjoying the fruits of your work to take the time to comment.

But for what it's worth, know that there is heartfelt gratitude and powerful hope for what you have and will achieve in this most beloved fictional world.

Thank you all again for all the hard work. It is recognized and appreciated.

-Forjo
Can't say it better myself - so I'll just quote it
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 8 List of Known Bugs?
03-12-2010, 04:53 AM
Quote:
This is no way means QA didn't find it, or that DEV hasn't been actively trying to fix it. All that means is that when the decision is made to move forward we like the Borg will adapt.
Having been working in IT now for the last decade or so as a developer and dev lead I understand the process. I've been in several of these test cycles both pre and post deployment. I also recognize that there is no such thing as a 100% bug free application no matter what some project managers or product leads would like to think. And, sometimes, a go/no go decision needs to be made whether an application should be released with known bugs. Under these circumstances, when the developent and QA teams put together the release notes they include a list of known defects in addition to new features and bug fixes.

Would it be possible to include a list of known bugs in your release notes? The Engineering Report addresses this to some degree. Perhaps coordinating the updates of the Engineering Report with the release notes with a link from one to the other would address this. I certainly wouldn't expect every bug to be listed, rather those that affect the most players that have the most impact. For example, I would have liked to have known that the Big Dig episode was bugged before I attempted it. I would have avoided the Fire Caves on Bajor had I known my bridge offers would suddenly feel so despondent over my poor leadership as captain that they would violently take their own lives in molten lava. I still have nightmares. The horror, the horror....

And, thanks for you efforts and hard work!
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 9
03-14-2010, 04:07 PM
I have a query regarding in-game support from GMs.

I'm sure they're mostly all lovely and hard working people, but in the various tickets I have submit on a range of issues I have only ever received one response within the friendly side of 3 hours and even then it was just canned responses. It might be just because I am from Australia and thus there aren't many on, but it also might be a case of the issues I've reported are not something that GMs can address within STO.

Like you stated, each company does it differently, and I have seen comments in some threads that GMs are unable to help with issues such as re-spawning NPCs that in other games like WoW, a GM can do easily. It might simply be that such actions are inappropriate for STO, against Cryptic policy etc, however, at the moment the only expectations I have of in-game support are my experiences from other MMOs such as WoW and AO. That is not to say that their practices and systems are right, or even applicable to the technology or structure of STO/Cryptic, but rather I'm lacking a clear idea of what it is that Cryptic thinks its support will be providing us.

Can you provide the playerbase firstly with some kind of description of the hours in which GMs are operating in; ie is staffing moderately consistent for all 24hrs of the globe or are they all in the US and the graveyard shift is only five people?

Secondly, what kind of issues can the GMs actually help us with, and what kind of issues are we just going to get a copy and paste response that doesn't help us at all and wastes a GMs time in having to read and respond to when they can't help?

Thanks.
Lt. Commander
Join Date: Dec 2007
Posts: 120
# 10
03-14-2010, 04:59 PM
-----QA most often, will not get the final say as to when something is released and in what state.----

That bit right there is the main flaw with most MMO developers/producers...in the MMO world, the tail wags the dog, and you get unpolished games thrown over the wall...QA should be there as a gatkeeper, otherwise, spend your money something else.
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


All times are GMT -7. The time now is 11:41 PM.