Update:Blog - Updates: Trust Us, We're QA
This blog was written by Mod Reece and dates to 21 August 2009.
The forums have been bustling with questions recently regarding the process of QA with projects, and why some projects have bugs in them. Well, hopefully, this Developer Blog will explain the process of QA a little, so that you can better understand how content takes the step from a design proposal to being enjoyed by hundreds of thousands of players worldwide.
The step into QA
When a piece of content comes into QA, it typically takes no more than few hours to find errors, glitches, bugs and concerns. Worry not, though, as this is expected and it’s our job to discover these things! From this moment to the content being released, we work through specific test plans on testing each element of the content, known trouble areas and general exploratory testing.
Any issues found are then sent to the designated developer for fixing, be it anyone from a graphics artist to a content developer to web content. An example of the scope of issues found in a project: an upcoming quest currently stands at 123 bugs, has been in QA for one week and is being tested thoroughly to get it released in time to be enjoyed by everyone.
More than breaking things
While a large portion of the task at hand is to break and destroy content sent our way as much as possible, which can be extremely enjoyable and equally frustrating at times, we are also very opinionated and believe we understand the players.
When a piece of content comes into QA, it is inevitable that opinions on how to improve design decisions or gameplay mechanics will fly about, trying to make the content more appealing to players. This can be anything from making a onweapon look more deadly and suited to the style of RuneScape, to simplifying or increasing the difficulty of puzzles to keep the players engaged (but not bored or frustrated), and even completely redesigning core gameplay mechanics.
These feedback decisions are then relayed to the developers, and with the projects being their own, the decision rests on their shoulders. If the QA team feel particularly strongly about a decision, the concern can be raised to the leader of the particular department, who will weigh up the feedback given, the developer’s feelings, and the most important variable of all: time.
Priority
Time is extremely important when working on any MMO, and even more so when that MMO receives updates on an almost weekly basis (as opposed to the more typical three- month cycle). It’s rather remarkable, actually.
Unfortunately, given the strict schedule for release the game has, priority has to be put in place for what can be done in the time allocated. Being given two weeks to fully QA a quest, for example, means that the QA team working on it have to determine where to allocate time, as in those two weeks, the quest has to be tested twice - once in the work in progress (WIP) test environment and once in the release candidate (RC) test environment.
Further on from this, to maintain such a quick turnaround on releases, the QA team typically has anything from four to ten projects on at once, split between the team (which also have to be balanced alongside any fixes for previous projects or improvements that could not have been implemented in time), as well as fixes for bugs found by players and relayed to us via the Bug Reporting Forums or Bug Report feature on the homepage (we read and investigate every report!).
Why is content broken?
There are typically three core reasons why a piece of content contains bugs at release:
Time
This applies more to game ‘features’ than outright bugs. We know of a feature that might cause issues, but, unfortunately, the time it would require to try out and test alternatives may be longer than is available for the project. For issues such as this, we start to consider changes straight after the content is released and wait to see what feedback the players give, so we know the best direction to take. A good example of this is Mobilising Armies, which we’ve recently tweaked to iron out some issues players have had with it since launch (e.g. the number of briefing rooms).
Test Environments
As detailed above, and in even further detail by Mod Nexus in his ‘Upload Manager’ Developer Blog, the content has to be tested in both WIP and RC. WIP is the core environment that all developers use to create content for RuneScape, be it for an update scheduled for the next week, or one for next year.
Even a small change in WIP can have a large impact on the game. It is not uncommon for one change to affect just a single item or feature from a selection of millions - something that is nigh on impossible to locate even if given a year to find. Not only this, but when the piece of content moves from WIP to RC, it’s like moving content from a copy of RuneScape that is six months in the future to current-day RuneScape. The differences can easily break content, requiring us to quickly discover what, where and how this has been caused - as this move into RC means the content will go up as the next update, the clock is ticking.
Bugs Missed
Given the scope and scale of the game, along with the constant progression of updates, bugs are occasionally missed. Only a small percentage of bugs actually fall into this category.
When you consider that, for example, there are 713 head items, 211 cloaks, two graphical modes and 12 graphical settings, that creates 3,610,632 possible combinations. And that’s just regarding helm and cloak graphical issues; the concerns are far greater when considering there are currently 12,988 objects, 11,037 NPCs, 47,451 item locations, 2,531 interface sprites...the list goes on.
Rest assured that the QA team work tirelessly on getting updates in the best shape possible for players, each and every week. Personally, being a large fan of MMO games (and typically being subscribed to many at a single time), you are getting every second worth of your time and membership investment into RuneScape.
Hopefully, this will answer some questions and raise more. Feel free to ask any questions you have on the forums. Happy adventuring!
| Citing this source: |
{{CiteDevBlog |author=Mod Reece |url=http://services.runescape.com/m=devblog/view_post.ws?post_id=25&project_id=13 |title=Trust Us, We're QA |blogdate=21 August 2009 |date=~~~~~ |name= }} |