Tech Dev Blog #10: KILL THE BUGS!

By @ 08/30/13 in techblog

“Kill the bugs!”, or, “How to do QA without a QA team.”

The bugs attack Whiskey Outpost. Mmm... whiskey.

The bugs attack Whiskey Outpost. Mmm… whiskey.

I am, of course, lying. It’s not like my brother and I were the only people who ever played TownCraft before it was out. We had some incredibly diligent testers along the way, including one who gave up his time for a pittance, for a days at a stretch if we needed him.

But it’s still not a QA department. When you’re an indie developer, it may not be too difficult to find a few friends & family members to play your game – especially once you hit that magical point where it starts being fun – but that’s not what QA is, really.

Seriously, good QA means repeating the same thing over and over to find obscure bugs. It means not just playing the game while munching popcorn, but getting people who are not only good at testing every little facet of your user interface and game, but also good at writing technical descriptions of the problem – and how to re-create it. They have to stress testing your AI and scenarios for all manner of discrepancies and woes. In the case of an action game with a linear path, it may mean repeatedly doing the same section until you’re so tired of it you just want to jump in front of the nearest Shambler while crying for mummy.

So, without a team of paid, professional game-breakers at your disposal, how the heck do you get your game to a playably stable state? Well, after much thought and putting together all the things I think we did right and wrong on TownCraft, here’s my list of things I would do again for our next game.

Notes to past-self, from current-self:

#1 – Prioritise your bugs in a good system, and categorise them

As you add bugs to your ever-growing bug-list, add priorities. You know the one – it’ll be the big document which, when you open it, starts liquid pouring from your eyes for some reason you can’t figure out? Yeah, that. You can break it down as much as you like, but make sure YOU know how important they are just by looking at them.

These bugs should be places in a good bug-tracking system. Even if there’s just one of you, trust me – it’ll make your life easier to use a dedicated back-tracking system instead of just a scrap of paper or a text file somewhere.

Part of the reason for this is to be able to list your bugs as belonging to certain parts of your engine. Is it user-interface? AI? Rendering? Mark them as such.

#2 – Change the priority of your bugs – often

You know that bug where that button stops working until you re-open the window? That tiny one? It’s no biggy. Doesn’t crash the game or anything, and doesn’t really affect gameplay.

If you keep seeing it, it’s more important than you’d think. By the second or third time you notice it or have someone comment on it… change the priority. Seriously. The importance of the bugs on your bug list should be in a constant state of flux.

#3 – Have bug-hunt sessions… for specific parts of your game

You know those bugs that will take forever to fix? Marked as ‘critical’? The ones that make you hyperventilate? The ones you NEED to fix before you can push out the next beta build?

Yeah, well, they aren’t the only ones. In fact, they may seem important, but they can also lead to ignoring a ton of tiny, insignificant-seeming bugs which will result in people playing your game for the first time and thinking “Boy, this feels buggy and cheap”.

My solution for this was to set aside an afternoon a week to simply grab all the bugs in one category (AI, pathfinding, rendering, UI) and actually start with the simplest bugs first. Why? It’s psychological, y’see. When I’d just knocked off 5 simple bugs in half an hour, the idea of tackling that big beast that I thought would take me hours doesn’t seem so bad.

And trust me on this – nothing is more exciting than seeing your bug list shrink, even if they’re all tiny low-priority bugs. And when your entire, say, crafting system, goes from buggy to solid in just an afternoon, even if other bugs remain in your game, you’re going to feel pretty darn good.

#4 – You need a brutal, diligent tester

You may not be able to afford a QA department, but I think almost all of us can find that one friend who’s good at breaking things. That one who constantly finds – no matter what he or she does – that one obscure UI bug in Microsoft Word. You probably spend half your time saying to him or her “What bug? I haven’t seen that one.” But they’ll find it.

You want this person, and you want them playing your game often.

But the thing is… they’ll probably only find a relatively small number of bugs in your game, despite your best efforts. Which is what brings us to…

#5 – You periodically need about a thousand short-term testers, too

As good as Captain Breaksthings is, and as much as you might play your own game, here’s a secret:

Humans, including and especially the both of you, are creatures of habit. The way you use a certain bit of your UI and the way you behave around your NPCs is going to be conditioned by having played the game a great deal. That minor bug that you know about? Without even realising, you’ll be working around it, and artificially making it, in your mind, a low priority bug.

But it may not be.

So, periodically, you want a large volume of people playing even just a few minutes of your game at a time.

This may seem difficult, but it’s easier than you’d think. We routinely took our game to the local IGDA meetups and let anyone and everyone play the game. Sometimes, some very helpful bits of advice from the aspiring and experienced game developers helped. Other times, just watching them play your game can help you re-think the priority of your bug list – or add new things to it.

We launched our game at PAX Australia, and while I think the total number of sales we made that weekend would not even come close to covering the cost of going down there and attending, one thing which, to my mind made it worth while was this:

We got to see about 30-40 people a day playing the first ten minutes of our game.

By the end of that long weekend, my bug list hadn’t grown, but it had entirely shifted orientation. Tiny bugs were suddenly listed as important – because so many people found them, despite our staple pool of testers and half the dev team having literally not seen the bug in weeks.

#6 – Your game is NEVER FINISHED

So your game is out. It’s actually selling copies and people are playing it (creepy feeling, right?).

Yeah, well, it’s not done. Not even close. Even though we were fairly happy with out 1.0 release, especially, as we kept telling ourself, that it was the work of a tiny indie team and yet still seemed pretty slick to us.

Every bug people report has to go straight in, and more than that, you should engage with them. Make it personal. Your game may technically be out of beta, but the best thing you can do is not only respect that the people emailing or tweeting or otherwise contacting you are paying customers involved… but remember that they, too, want your game to be better. So when they either politely or rudely tell you about a bug, the best thing you can do is eat some humble pie, thank them… and make those bugs your priority.

If you can’t afford a QA team, then you should relish every interaction you have – leverage every single player who engages with you. That doesn’t mean you should ask them a thousand technical questions or get them to try and track down bugs for you – they are, after all, enjoying themselves with your game, or trying to! But it does mean you should get as much information from them as politely as you can, at least so you can replicate the bug and get that shit sorted, post-haste!

Because many, many people (if you’re lucky) will be playing your game for the first time not on version 1.0, 1.01 or the like – but 1.0.5, 1.0.6 or maybe even 2.0 – and so while you may be telling yourself that it’s just the work of one or two people on the core programming side, an indie team, and it’s *okay* that it’s a bit buggy… nobody else knows this. Many – even most – of your players won’t know or care that it was made by two people in their basements. They’ll simply notice if it’s buggy or not. Don’t make excuses – just fix bugs.

Make your game so polished nobody will ever realise that you didn’t have a team of twenty professionals testing your game for a year before release.


Comments are closed.