Designing A Production Process: Part 2
In Part 1 of this series, I broke down the day-to-day production process we put in place under leadership of Jeff Morris while developing Star Wars: First Assault. It's not a prerequisite for this post, but it's a good place to start if you want to put what I'm about to discuss in full context. Or, maybe just read it later if you like what I have to say here :)
For Part 2, I'm going to focus on the tools we used to track our tasks/bugs (issues) and the processes we put in place to funnel issues into the database. SOUNDS EXCITING, AMIRITE?! I know. Like I said in Part 1, if you're looking for video game glamor, unfortunately you came to the wrong series of blog posts :/ However, if you're here to learn how you can make a meaningful impact on the quality of your game by designing industry leading production processes for your team, then you're all goody! Jeff Morris taught me a lot about that and I'll do my best to pass on the knowledge here.
Issue Tracking Software
Let's get this out of the way first: all issue tracking software sucks to some degree. Just choose one of the standards (Hansoft, JIRA, DevTrack) and stick with it. You're always gonna hate something about the one you have and be tempted to think the grass is greener in another solution. I think the bottom line is database management is tedious and prone to exploding in complexity if you don't think hard about how you organize and structure it. Is there a clear definition for every field? Are you trying to cram two fields into one? Do you really need to add a new field? Can you nuke some fields you thought were useful, but are now just noise? Should you make this field mandatory or optional? You need to consider this stuff carefully or you're going to make navigating your database and creating useful filters a pain in the ass.
Every issue database is going to require some unique architecture, but I'll highlight a small subset of fields here that should exist in any project and that drive me crazy when they're not utilized properly.
Task Priority
I'll be honest; I'm hesitant to say that what I'm defining here for task priority is what's best for every team. Even on Star Wars: First Assault, we had general guidelines, but didn't have an official rule set. That said, this worked extremely well for me as a strike team lead and it's somewhere for you to start. The underlying philosophy behind every process we designed on that project was iteration. In addition, we allowed for changes to accommodate specific needs of individuals. So, if this doesn't work as well as you hoped, tweak it, evaluate it, tweak it again, evaluate it again...into infinity.
P0: This task needs to be completed ASAP in order to unblock someone else from doing their work.
P1: We can't ship without completing this task.
P2: We can ship without completing this task, but quality of the game will suffer significantly.
P3: We can ship without completing this task and that's probably fine.
Bug Severity
Understand that I'm coming at all this from a developer point of view. That is to say, I'm not coming at this from a QA point of view. These are bug severity definitions that I think work well during the non-shipping development phases of your game. Maybe you should consider adjusting them when you hit alpha? Don't be dogmatic and always be evaluating if what was once right for your team, now no longer makes sense. The only constant is change in our video game worlds.
Critical: The build is crashing consistently and often due to this bug. The entire team is affected. Drop everything and fix it.
High: This bug is blocking someone from progressing on their work, severely hampering productivity or our making it impossible to evaluate a feature. Needs fixing sooner rather than later.
Medium: This bug represents an extreme lack of polish and it's very obvious that something's broken, but generally not affecting anyone's ability to get their work done.
Low: This bug is pretty minor and a good chance the consumer won't notice, but you know it's there and you would love to get it fixed if time allows for it.
One thing to note is that if you get a bunch of High severity bugs stacking up on your plate, that can really represent a Critical situation for your team. I can't emphasize enough how important it is to make bug fixing a regular part of your routine. Build time for it into your schedule during every phase of development (see Part 1 of this series).
Discipline
This is kind of a no brainer, but every issue should have a mandatory discipline field associated with it (e.g. Gameplay Engineering, Level Design, Environment Art, Audio, etc.). However, what isn't a no brainer is having an explicit discipline field. Too many times I've seen discipline values mixed in a field with other specifiers like the feature or strike team it's associated with. These all need to be separate values. You need to make sure field properties only describe one specific aspect of the issue.
Also, you might want to break this up into 2 fields; one for the core discipline (e.g. Engineering) and one for the sub-discipline (eg. Graphics, Gameplay, etc.). It can help with creating filters and reporting.
Feature
With very few exceptions, if any, every issue in the database should be associated with some feature in the game. Whether that be a weapon, character, level, menu, etc. This should be another mandatory field for you to fill out. It's going to be most useful for strike team leads who should be creating filters with this field to evaluate and plan work across all disciplines for the features they're responsible for. Sometimes QA Leads can get a little protective of their issue database and push back on adding fields like this that are developer-centric. If that happens, push back on them harder...as nicely as possible ;)
Similar to the Discipline, you may also want to break this up into 2 fields; one for the core feature (e.g. Spartan Ability) and then a sub-feature field (e.g. Thruster Pack, Spartan Charge, Ground Pound, etc.). To reiterate; it helps with creating filters and reporting.
Just a few more things about issue tracking databases...
Take the time to keep your issue view clean and organized. Hide columns you don't care about and put them in some logical order; usually keeping the most frequently referenced information to the left so you don't have scroll over all the time. Also, not a bad idea to come up with a template for your average team member and make sure it's applied to their account when they first come on board. On more than one occasion I've been helping someone at their desk and their view didn't include task priority. WTF?! HOW DO THEY EVEN VIDEO GAME?
Teach and encourage your leads to create filters. If they'd rather you just do it for them and you have the time, that's totally cool, too. They just need to feel empowered to slice and dice the database in ways that are most useful for their individual priorities. Having great data doesn't matter if you can't examine it properly.
For all that is good & holy, DO NOT have separate bug and task databases. Your team should only have to learn how to use one issue tracking database and they should only have to go to one spot to check their tasks & bugs.
Maintaining a Task Backlog
“If it’s not in Hansoft, it doesn’t exist.”
So, now that you've got your issue database setup properly, time to fill it with stuff! You need to make your sure that your issue database the "single source of truth" (i.e. there aren't other to-do lists floating around somewhere) and that the backlog represents all the known work so you can properly evaluate what your ship date is trending towards. To that effect, you need to make sure that every task in the backlog has a work estimate associated with it, too. Not necessarily right when you enter the task, but as I mentioned in Part 1, during your next sprint planning phase you should scrub every task to ensure that they all have estimates at that time. This definitely isn't fun and you're going to get resistance from people, but you need to do it. Also, remember that you don't need to be spot on immediately. Take your best guess now and then you can update the estimate when you learn more about the task later. That's the most you can ask of anyone.
Managing an issue database takes constant grooming. It'll get unwieldy, unreliable & full of lies very quickly if you don't scrub it regularly. Discipline is the name of the game here. Like, really difficult to maintain, floss-your-teeth-every-day type of discipline.
Managing Bugs
“I’m gonna go make sure that was a valid dump.”
Managing bugs is a lot more straight forward than maintaining a task backlog so I'm not gonna spend much time talking about it, but I did want to touch upon one topic. That topic being, when to assign a bug to the person responsible for fixing it. Often times production will attempt to "protect" their team by not assigning bugs out until they want it worked on so people don't get overwhelmed with a giant list. I strongly advise against this practice. On Star Wars: First Assault, Jeff always had bugs assigned as soon as they went into the database. There are a few reasons why I think this is a better practice. For one, assuming you have work estimates on bugs, you can add that up with task estimates in order to predict your ship date and balance workloads better. Second, that bug might be super quick to fix and you should give the person responsible for it the option to do that. Lastly, everyone should have an accurate picture of how buggy their work is at any given moment and not assigning bugs out is kind of lying to your teammates.
If someone is getting overwhelmed by a giant list of bugs, then they should think about dedicating an entire sprint to catch up when the normal dedicated "Unscheduled Work" time is not enough (see Part 1). Not to mention, you should be ensuring that your team has filtered views of their issues so they can focus and just look at ones assigned to the current sprint, if they want to.
Processing Playtest Feedback
“Playtests are the most important meetings you can be invited to.”
On Star Wars: First Assault, we had 2 playtests per day. This generated A TON of feedback and we wanted to implement a way to capture that and act upon it, if deemed necessary. First, to collect the data, everyone filled out surveys before they left the playtest lab. 10 minutes before the hour was up, we had everyone switch their screen display from the console over to the survey on the PC that was waiting for them in a maximized window. Each survey was tailored for feedback on a specified feature. The surveys were changed up on a weekly basis, based on the focus that was usually set by our Lead Designer, Ed Kay. They consisted of no more than around 10 multiple choice questions and always included a comment box.
We mined a lot of gold out of that comment box. QA read through each one and pulled potentially actionable feedback into the database in the form of a bug, also making sure there weren't dupes of that feedback. For example, "Assault Rifle feels really weak against the SMG". That's the type of "bug" I would receive as a designer. However, that didn't mean I had to take any action. It was totally valid for me to just close it "By Design" and move on, but at least my finger was on the pulse of the team.
The last thing I want to say about playtests is that they need to be mandatory for everyone. Seeing your work in that context is imperative. You need to be playing your game on the regular. At least once a week, but hopefully more. Jeff actually kept a list of when each person on the team had last been to a playtest and used that as ammunition to deny requests to skip them. The only reason you might allow someone to skip a playtest should be if they have a Critical bug that needs fixing ASAP, but even then maybe they should take an hour out of their day to play the game. If someone does skip a playtest, then they get an invite to the very next one. There is no escape!
Almost done...
Dang! You made it to the end again. Thank you. It means a lot to me that designing a good production process means a lot to you. Stay tuned for Part 3 where I'll dive into long term planning and handling dependencies. SPOILER ALERT: It doesn't involve MS Project. Burn that shit in a hot fire.
@Ryan_Darcey