Microsoft Zone Archive/News & Events/Backstage at Asheron's Call
Original Link (now dead) - http://www.zone.com/asheronscall/news/ASHEbackstage.asp
Backstage at Asheron's Call
December 13, 2000
In this article, we'd like to better illustrate how we make and maintain Asheron's Call. This should answer several questions that players may have, though it likely has far more detail than most players care about. If you just like playing the game, by all means, do not consider this essential reading!
For those players who are interested, however, we hope you find this informative. The whole "AC Team," which includes both those at Microsoft and Turbine, jointly prepared this article.
It's been a year. What took you so long?
To some it may seem that this article has been a long time coming. We've never formally published anything of this sort before. Perhaps we have waited too long; but for reasons explained below, we have tried to give our players a sense of having one point of contact, and avoid giving details about the many dynamic groups working together to make AC. However, now that we have been producing AC for over a year, it's time to open the backstage door a bit and let you see some of the "what, how, and why."
So who are "we"?
Our intent has been to not distract our players with exactly who does what in producing AC. All those on the AC team, no matter where they happen to work, unite towards one goal: keeping our customers, the players, happy. If you go to an amusement park, you should not hear the ticket taker say, "Well, it's not my problem the Barf-o-Rama isn't working -- go complain to the maintenance staff." You should hear something like, "We are sorry the Barf-o-Rama is out of service. We'll try to get it working as soon as possible." We want to treat our players in a similar way (though sadly there is no Barf-o-Rama yet in AC).
Therefore, in nearly everything you read on the AC website and in the system messages, we use the term "we" to represent the entire AC Team, even if we are really talking about something that is purely an issue with the game code, the network operations, the community management, the website, the testing, or the like. This will not change. We still think it is the best way to serve the vast majority of our players.
However, we acknowledge that some players really do want to know the nitty-gritty about who does what. We also want to clear up some misunderstandings or incomplete assumptions that have cropped up during our silence. So in this article, we'll largely drop this "we" stuff and talk about the different functional groups that help make AC what it is.
What are the roles of Microsoft and Turbine?
Microsoft is the "publisher" and Turbine is the "developer" of AC. Though making AC is a labor of love for everyone, the general roles are most simply described in financial terms. Microsoft ("MS" from here on out) supplies people and money to help maintain and update AC. Expenditures include management, testing, and community staff in MS, as well as equipment, network costs, and so on. MS pays Turbine to continue to maintain and develop AC, as well as shares with Turbine a percentage of the revenue from AC sales and subscriptions. Turbine uses the funds from MS plus its own investments to develop AC, doing programming, testing, art, game design, and so on. As such, it's not very different from the relationship between a movie producer and movie production company.
Microsoft is a developer and publisher of software products and services. Nearly all the MS people who work on AC are located in the Seattle area, though there are critical people in sales, localization, customer support, and other functions who work worldwide. Turbine is a developer of software products and services as well, including games and game technologies. All of the Turbine people are located in the Boston area.
The most important thing to communicate here is that MS and Turbine are partners in a business venture. Our relationship is deeply collaborative, friendly, and oriented toward the exact same goals: make money by giving our players what they love. As in any business partnership, there are at times dynamic tensions among the various sub-groups. This is healthy. Everyone pulls for what they think AC needs most, and the ensuing discussions generate the best plans for action. What we all do share, however, is interest in each other's well-being, and devotion to making the best Online Role-Playing Game (ORPG) in the world.
As the publisher, MS in principle has the "final say" over the nature of the product presented to the players. MS makes the majority of the investment and desires a certain kind of product in return. However, Turbine is far from a mere contractor under orders from a dominating boss. Turbine has a great deal of its own money, time, and reputation invested in AC. Moreover, AC directly affects Turbine's image as a company. Therefore, Turbine has a great deal of influence on AC. MS realizes this and encourages it, not only because MS values Turbine as a partner, but also because MS recognizes that many important decisions concerning the development and evolution of AC are best left to those who love the game and know it the best. Turbine is full of those people, working and thinking night and day about how to make their "baby" better. That said, the MS people are not a bunch of suits behind mahogany desks. The MS game development group has many people who love games as well, and many of them have worked at other prominent game companies. They also play AC, talk about it in the hallways, devour the fansite traffic, and have their own passions about what is best for the game.
Though both MS and Turbine would love to add everything to AC right away and make it perfect, game development and publishing is a business. Contrary to popular belief, no software company -- not even Microsoft! -- has a barrel of cash in the hallway with a sign saying "take what you need." Turbine does have a budget. The game development group at MS also has a budget. All things considered, we think we do a great job within the constraints we have, and we plan to serve our players well for a long time to come.
How do MS and Turbine work together?
It's not easy to make an Online Role-Playing Game with the critical development and functional groups three time zones apart. Communication and a friendly relationship are the most important things to have. We are constantly in touch by phone and e-mail. A big conference call takes place each week that includes all of the functional "leads" (those who head up each functional group) in MS and Turbine during which we discuss anything and everything. In a different weekly conference call, the appropriate leads discuss bugs in the game. After each monthly event, we do a "postmortem" meeting by phone to discuss and document what went well and what did not go so well. When we can, we get on a plane and meet in person. (Teleporter vendors: please get in touch with us soon.)
Break it down. What are the different groups and functions?
So, in a nutshell, here are the functional groups. Labeling is always an inexact science, but this should give you a general idea of who is who. Then we can explain which of these functional groups do what various things.
"The Developers" is a general term that refers to pretty much everyone at Turbine who works on AC, including programmers, QA, visual and audio artists, game designers, and producers.
"The Publishers" is also a general term that refers to those at MS who make production and business decisions, including planning, budget, staffing, marketing, and PR. Others at MS fall under other more specific labels below.
"Programmers" are those who write game code, build technologies, and make development tools. They all work at Turbine. (Confusingly, some use the term "developers" to refer to just programmers. In our parlance, "developers" is the more general term described above. Also, some "programmers" prefer to be called "software engineers" or "coders".)
"QA" are those at Turbine who test the game. They make sure that a "build" of game code and data are in working order before it is sent to MS. They also find and confirm the fixing of many bugs.
"MS Test" are those at MS who test the game. QA and MS Test have somewhat different tasks, which are too detailed to cover here. In broad terms, QA largely concentrates on detailed content issues (that is, specific properties of in-game objects, places, monsters, art, and so on) while MS Test does detailed testing of game code plus a high-level check of the content. However, there is a lot of overlap.
"Artists" refer to those who make visual or audio content for AC. They all work at Turbine, though some of the art in the manual, website, and setup program was done by artists at MS.
"Game Designers" are the main designers of the game rules and functions as the players perceive them. They "spec" (specify) how various things in the game should work; they use tools to map out dungeons and landscapes; they write game lore and texts; they spec out quests and items; and, they also "tune" existing systems -- that is, they adjust the values and strengths of things in the game to pursue the philosopher's stone of "perfect play balance."
"Producers" are those who direct and organize the flow of work at both MS and Turbine, track schedules and budgets, monitor overall quality and customer issues, report to the business managers, and assure communication. At MS, the producers are called Program Managers or "PMs."
"The Community Lead" is a PM at MS who leads the in-game community "Online Team," which includes the Advocates, Sentinels, and Chancellors. (For more information on the Online Team, see this article.) The Community Lead also closely monitors customer service issues.
"The Operations Staff" are those at MS who maintain the AC servers. They monitor the server status 24 hours a day, and report problems. They also put the new "builds" on the game servers.
"The Zone Staff" are those at MS who keep the MSN Gaming Zone (which is itself a separate MS product) up and running. This includes the AC website on the Zone as well as the "lobbies" (those windows with the purple "play" button you click on to launch AC). They also made the code that downloads new AC "client" software to you, the player. When you see the "Downloading Zone Files" dialog box, it's the Zone's tech giving you a new AC "client.exe" or other files to make the Zone work.
"The Billing Staff" are those on the MS Zone Staff who concentrate on maintaining AC subscriptions and fixing related customer problems.
"PSS" are those at MS who provide product service and support. They are the people you communicate with if you contact Tech Support via the information provided in the front of the AC manual. They provide phone support and maintain the troubleshooter website to help solve technical problems.
"The Vanguard Team" is composed of volunteers who play builds of AC on a private test server before it is rolled out to the public. They are managed by Turbine, and share the same testing goals as the Turbine QA team. (Sorry, Turbine is not currently seeking any new applications for this team.)
"The Fire Team" is composed of MS employees who also volunteer to play test builds of AC. They are managed by MS Test, and also share the same testing goals.
Whew! Perhaps you now see why we tend to use the catchall term "we." Otherwise, it would really be distracting to always detail exactly who is informing you of various game issues!
How do AC bugs get fixed?
"Bugs" are those things in AC that don't work as the game creators intended. The first step is to discover the bug. We find and fix most of them before the update ever reaches the public. Those that "slip through," we typically discover thanks to customer reports, fansites, PSS, our Online Team, and the Fire or Vanguard Teams; however, sometimes they are discovered by others on the general AC Team.
Once found, the bug is carefully described and put into a bug database called "RAID." Both MS and Turbine can read and edit this database. The first stop for the bug is "triage" (borrowing a hospital analogy). Every day, the producers, QA, and MS Test look at bugs in triage and assign them to a developer to fix. At the same time, the bug is rated with a "priority" (where on the big list of things to fix the bug ranks) and a "severity" (how badly it affects the customer when it hits). Everyone generally agrees on priority/severity the first time around, but if anyone thinks a bug should be a different priority, they can send it back to triage for reconsideration. The goal is to have everyone agree on a bug's priority, and that is almost always accomplished. Developers then fix the bugs in the order of their priority. If faced with bugs of equal priority, developers generally tackle those of the greatest severity.
So the developers, always hard at work, look at bugs in RAID assigned to them and fix them as soon as they can. Very serious bugs can get fixed right away (more on this below). Bugs of urgent priority get fixed in the next monthly update, while bugs of less urgent priority may or may not get fixed as soon.
A developer, upon fixing a bug, "resolves" it and assigns it to QA or MS Test to confirm that it is in fact fixed. If the fix is confirmed, the bug is then "closed."
It is very rare that we disagree over when or how to fix a particular bug. We nearly always achieve our goal of decision by consensus. If a bug needs more discussion, it goes to "supertriage," which is a weekly phone meeting of all the relevant leads at MS and Turbine.
Usually we fix bugs in the next monthly update. However, if a bug is especially problematic for a large number of players, or threatens the stability or integrity of the game, we may want to "hotfix" it, meaning we fix it before the next monthly update. Once again, we nearly always all agree on whether a bug merits a hotfix or not, and if we don't, we try to keep discussing it until we have a consensus. A hotfix eats up a lot of time from a lot of people who are otherwise working hard on future monthly updates. Even a couple days of delay can throw off our pace for the entire month. Not only does the hotfix need to be coded and built, it also needs a full testing cycle since even a tiny fix can break something else, with major consequences. Next, our Operations Staff needs to get up in the wee hours of the morning to put (or "prop") the new build onto the servers. During the prop, the servers are all shut down and nobody can play (that's why we do it in the wee hours of the morning). When the worlds are restarted after the prop, there is a chance for instability. For all these reasons, we try to hotfix only when we can make and fix the bug quickly, and when it is important enough to risk stability and the loss of time.
How do new things get added to AC?
Each monthly update starts off as a "spec," or documented specification. It starts in brainstorming sessions among the game designers, drawing from ideas within Turbine, MS, and from the players themselves. The game designers write up specs that describe what the next month should bring. This process begins several weeks before the public gets the update.
The PM at MS comments on and approves the specs, representing what the publisher wants in the product -- that is, happy players of all types, as well as good opportunities to further publicize and sell the game. Likewise, the game designers choose the direction and focus of the updates, balancing all the things they want to introduce. The PM and the game designers discuss and refine the specs until everyone is satisfied with the plans for the next update. Meanwhile, the Turbine producer makes sure that the work slated to be done for that month stays within the development budget, and the testing teams make sure that the update will not be too hard to test.
This is a deeply collaborative process, and what comes of it is a spec better than any one group could have done on its own. The updates are both a science and an art, both a business and a performance. Each of the functional groups share a respect for the roles of the others, and the goal, almost always achieved, is for everyone to be happy with what is coming next to AC.
As the developers create new code and content as described by the spec, QA and MS Test start testing it out. They report bugs and make suggestions. Soon they get help from the Vanguard and Fire Teams. This work gets increasingly intense. In the last week of testing, only quite serious bugs will result in changes to the build, because any fix, no matter how small, might break something else.
Then comes "prop day," when the new build is put on the public game servers. The Operations Staff shuts the worlds down in the early hours of the morning and begins the process of updating all the server files, which takes at least a few hours. Meanwhile, the Zone Staff prepares the new client.exe file for download. The servers are started up and checked, though players cannot yet enter the worlds. Once things look good, the players are let back in.
Then, we all watch and wait. We devour all the news on the fansites and in the game chat. Any problems are either put on the docket to fix next month or, in urgent cases, set up for a hotfix.
How is the Online Team run?
The Online Team consists of those people who help the AC players while they are online. For details of this team, see this article.
The Community Lead at MS oversees the day-to-day operations and policies of the Online Team, and takes care of unusual or ambiguous cases. The Online Team follows very specific documented guidelines which describe what kinds of behavior we want to stop, what we will and will not do to help players, and generally how to give our players service that is, as much as possible, consistent, fair, pleasant, and free from the possibility of abuse or misunderstanding.
These guidelines, as well as revisions to them, have been mutually agreed upon by Turbine and MS. Turbine did important early work on community design, which MS further developed when it took over the task of managing the Online Team.
How are the servers maintained and the worlds kept going?
The AC datacenters are in Washington state, running on hardware leased by MS from a major datacenter service provider. The Operations Staff manages new props (see above) and maintains backups of critical data. Monitoring the servers 24 hours a day, they also send alerts to the AC Team when there are problems with the servers or the datacenter. They capture and report data as requested by the programmers or producers, so the programmers can obtain clues as to what may be affecting AC's performance. Some problems are in the datacenter itself, such as problems with server or router hardware. Some problems are bugs in the AC server code. Slow performance may be due to general Internet lag, a datacenter overload, or due to the fact the AC servers are simply doing too much work. All such problems are taken very seriously. However, they are also the kinds of problems that are very hard to diagnose and solve in a stable and effective manner, taking careful iteration to improve over time.
How are new worlds brought into being?
Each game world's server is capable of doing a certain amount of work. The more players there are online at once, the more work they need to do. When the worlds approach peak populations high enough that they threaten to overtax the servers, we start planning to bring a new world online. We are also carefully pursuing code optimizations to get even better performance out of the game servers we have, which helps us stay on budget.
How is the AC website on the Zone updated?
The text of the articles published on the AC website on the Zone can be supplied by many sources within either MS or Turbine. An editor at MS works with the PM to assure the quality and consistent messaging of the articles. The editor uses tools supplied by the Zone Staff to publish new articles online. The monthly newsletter is composed by similar means.
What about all my other questions?
We hope that this article has been informative, but we know it may not answer every question you have. As we see more questions come up in fansites and game chat, we'll try to answer them here. Please be aware that if we have anything officially true to say, we'll say it in a published article or system message. Beware believing speculations or opinions you read online! No single person ever has a complete picture; we try to get several AC functional groups to buy into what we publish on the Zone to make sure we inform you, our players, as fully as possible.
We all look forward to many more great updates. We truly feel that Asheron's Call is improving all the time, and that we continue to break new ground in the ORPG genre. We feel very lucky and privileged to serve you by inviting you into our ever-changing fantasy world.
Here's to another great year!