Quality Assurance Team

Quality assurance or "QA" is a team at Jagex made to assure that all new content within the game and on the website is as high quality as possible, focusing on the stability and usability of the content. QA works from a player's point of view and tries to remove any bugs that may arise while playing RuneScape or visiting the website. RuneScape QA serves 3 key games; RuneScape, OldSchool and Chronicle.

Role of QA

QA at Jagex assure that the game runs smoothly and is high quality for the players that play the game. They test new updates to make sure that there are no bugs, but are also responsible for maintaining all of the content in the live game through investigating player reports, and submitting feedback for post-release changes based on player suggestions. Some QA members have played the game as far back as 2002 and actively play today.

Something that makes Jagex quite unique within the games industry, is that they offer all of their Quality Assurance permanent contracts but it comes with a 6 month probation, whereas usually in games development testers are temporary on low pay only for the period of which the game needs to be tested, and are then dismissed until their next project. Discussions with QA at Runefest uncovered that QA appreciate their roles and have the same benefits, and rights as everyone else in the studio.

Scrum

Jagex has several Scrum teams that focus on creating individual, unique pieces of content. The QA team are embedded within the development team, meaning that they work directly next to the developers, designers and artists to deliver content as a small focused team. They are involved in the content from beginning to end, contributing toward the design and development, before focusing on the testing and maintenance of content for years to come. QA at Jagex are very player driven, always considering how content will be perceived when put in the laps of RuneScape players. Some QA members are skilled enough in code to create their own content, while being supervised by senior developers.

Release Team

In 2014 Jagex formed a Release QA team that handle all of the releases before being handed to RuneScape players. Any content that is required for a release is packaged together into a single build, so that the Release QA team are able to confirm that all of the required content has come together safely, and has not conflicted with other areas of the game. They perform both manual, and automated tests to ensure build stability, and also confirm any bug fixes for existing content that are due to go out into the live game. This team plays a vital role within the development process, and it is this team that allows Scrum teams to focus on the next big thing, without having to worry about re-testing everything in a new build.

QA Staff

In 2013 Jagex introduced a career path for the RuneScape QA team, allowing team members to progress from QA Tester, to Senior and Lead roles. This also included the creation of the Junior role, which is used to hire people that have limited, or no QA experience and are trained on the job. It is common for Jagex to place players into Junior roles as they have no previous QA experience. In 2015 the QA team were recognised as Analysts rather than Testers, as this was a more professional title and better recognised what they do at Jagex.

Mod Name	Role
Mod Cocoa	QA Manager
Mod Stacey	QA Team Leader
Mod Grem	Lead QA Tools Developer
Mod Dorando	Lead QA Analyst
Mod James H	Lead QA Analyst
Mod TomH	Lead QA Analyst
Mod Deadly	Senior QA Analyst
Mod Sarah	Senior QA Analyst
Mod Kristy	Senior QA Analyst
Mod B3	Senior QA Analyst
Mod Chaose	QA Analyst
Mod Manti	QA Analyst
Mod Ramen	QA Analyst
Mod Harrison	QA Analyst
Mod Cuppa	QA Analyst
Mod Giara	QA Analyst
Mod Curse	Junior QA Analyst
Mod Wolf	Junior QA Analyst
Mod Dolan	Junior QA Analyst
Mod Shogun	Junior QA Analyst

The Quality Assurance team's logo

Testing

Testing is a stage during Quality assurance's work where they test the new content when it has been developed and come up with errors in the coding and other things to make the game better and easier to use. They aren't just playing the game how the average player would, they try to test every possible thing in order to make sure that everything is possible.

“	"Imagine: if you're testing a racing game, you're not just completing laps all day. You're trying to drive the car outside of the track or crash it into things to see if the game breaks."	”
	— Mod LewisG - Behind The Scenes - Quality Assurance

Runeshark were invited to the office to perform an interview with two Quality Assurance members in 2012, where they spent some time talking about what it means to be QA at Jagex. Two quotes from the video include:

“	RuneScape is one of the only MMOs that release updates weekly, whereas most others release in patches where QA have several months to test content. RuneScape has over 10 years of content, and the RuneScape QA team sometimes only have 2 weeks to test content before it goes live.	”
	— Mod Cocoa - RuneShark interview

“	Of the handful of bugs that do make it into the game, the QA team have already found several hundreds of them before the content is launched.	”
	— Mod Stacey - RuneShark Interview

However, QA does play for fun just like a player would. This allows them to make sure that the content that they are putting into the game is actually content that a player would take time to play.

“

"During initial testing, you have to assume the mindset of a normal player, playing through as they would." Mod Martin - Behind the scenes - Quality Assurance

”

Playing the game is only part of what the Quality assurance team goes through during their part of the development process. They have check-lists to make sure that every single piece of the new content is working right and meshing perfectly with the content already in the game. Each of these tests has to be done many different times under a number of different conditions. Even after all of this testing opening the content up to millions of players brings out problems that were unpredictable for the QA team.

Players work to fix the bugs also by sending in feedback and bug reports. It would almost be impossible to fix all of the bugs the first time because there are 250,000 players for every QA team member.

Submitting a Bug Report

Players can submit bug reports to the QA team in-game, however more often than not the QA team receive bugs that do not have enough information to investigate, due to lack of detail. It is also common for the QA team to receive reports that are not actual bugs, due to it often being player confusion.

The QA team said that it is helpful, and easier to investigate bug reports when a player provides names of NPCs or Objects, provided the location where it occurred and provide step by step reproduction steps up to the point where the bug occurs.

Example 1: When I wear my Santa hat on my female character, my hair style sticks out of the sides of the hat

1. Use a female character with a long hair style
2. Give yourself a santa hat and wear it
3. Observe my long hair poking out the sides of the santa hat rather than sitting underneath

Example 2: When I perform an emote my character becomes invisible and I remain invisible until I log out

1. Use a male character as it does not happen with females
2. Open the emotes interface and perform the Thanks Giving Turkey Emote
3. While the emote is in progress click the ESC button to bring up the game menu
4. Wait until the emote ends and then click out of the game menu
5. Observe that my character is no longer visible

Live maintenance

Live maintenance is the Quality Assurance's job in which they track what the players are saying about new content by reading the forums and checking bugs sent in by the bug tracker. They alert the developers to the problem and the developers work to fix the bug. Good and bad feedback helps the QA team to learn what the players that play RuneScape enjoy.

The Quality Assurance team also investigate all player bug reports, and submit reports to the development team for fixing. They classify bugs as Critical, Major or Minor which the development team use as an indicator for its urgency, so they can shift workload to ensure fixes are released in good time. The QA team confirm the bug fixes in their development and release build, before confirming it can go live to RuneScape players.

QA disruptions

Quality assurance provides a unique opportunity to make the game better, but it can also slow down the development process severely. If QA sees something that they don't like they will send it back to the developers. The developers will fix it, but that isn't the end of the process. The fixed material will go back the Quality assurance team to be looked at again. Depending on how perfect the team wants the material to be, this process can go on and on.

Mobilising Armies

This process can be seen first hand in a case such as Mobilising Armies. Mobilising Armies was revealed in a behind the scenes (Update:Future PvP Updates) on 15 August 2008 along with other future PVP updates including Stealing creation. In November 2008 Jagex confirmed the release date to be in December. In December Jagex revealed that they would release the minigame in early 2009. In the Update:Behind the Scenes - February (2009) Jagex revealed once again that they were postponing the release date to later in the year. It was then revealed in the forums that it would be sent to QA again as of 5 May 2009. After all the delays, it was eventually released in July of that year.

References

• Update:Future PvP Updates
• Update:Behind the Scenes - October (2008)
• Update:Behind the Scenes - November (2008)
• Update:Behind the Scenes - December (2008)
• Mobilising armies