Anatomy of a Bug

The Bug

Recently we had one of those bugs.  Any developer who’s been around long enough knows what I mean.  It’s the kind of bug that skitters across the freshly waxed floor just before opening night.  Then it vanishes and you wonder if you really saw it at all, or if it’s simply a caffeine and sleep deprivation-induced hallucination.  A manifestation of the anxiety that always accompanies you when you’re getting ready to ship:

What if we missed something?

What if we missed something really, really bad?

We had just finished adding some “Awesome Sauce” to Warheads for Android—shards, more explosive explosions, fireballs, a cool surface fog effect, and a planet (Mars in this case).  It looked g-r-e-a-t and it played really well due to some hard work we had put into re-writing the shard rendering code.  We’d tested the heck out of it and we were sure we’d gotten everything.

I did the final builds—the ones we upload to the Amazon App Store and Android Market—and did one last play just to marvel at the work of greatness.

The game froze.

At first I thought for sure it was just some cosmic ray striking at just the right moment to flip a bit and lock things up.  So I tried again, and it played great.  In fact, I played 17 waves and felt pretty good that my original assessment held up.  It was just a fluke.

Then it froze again—on Wave 18—and I knew we had a problem.

The Hunt

This wasn’t your run-of-the-mill null pointer exception.  The only thing we knew for sure was that this freeze had never been reported with the published version of Warheads, and that we could only get it to happen with the new “Awesome Sauce” version.  To make matters worse, when the game froze the only way to unfreeze it on our Nexus One devices was to pull out the battery for a hard reset.  It only happened rarely and when it did happen, all we got was a cryptic OS log error:

waitForCondition(DequeueCondition) timed out

 (identity=6, status=0). CPU may be pegged.

One of the first things we did was search for that message.  We didn’t get any stack overflow-style “here’s your answer” results, but we did see that other people had encountered this issue that it could be something that only occurred on HTC devices.  Since we had just rewritten the shards drawing code, we suspected that immediately.  However, we soon discovered that the bug could be reproduced even with the shards rendering turned off.  So it had to be something else.

We began to mistrust the OpenFeint code because the freeze seemed to loosely correlate to sending a high score to the OF service.  We also knew that the problem was that the gl rendering thread was deadlocked so we thought maybe the OF thread was somehow interacting in a bad way.  In the end, though, we were able to repro the problem with OF disabled.  So it had to be something else.

This took a couple days, and by this point we were really sick of playing the game over and over again to make the bug happen.  So we decided to have the game play against itself.  This was something we had done in Zombie Armageddon and it was a huge help because, while Reabs and I were both pretty good at the game there’s only so much you can stand before you start to get a little demoralized.  The game could play itself for hours and never get bored.

After about 4 days of poking and prodding and hoping that we had found the issue only to have our hopes dashed again, we gave up.  We had spent close to 35 hours working the problem and could not find the solution.  The update had other features and bug fixes we wanted to get out there so I finally made the call to ship it (version 2.4.0) without the Awesome Sauce.

I was really, really disappointed by this because I knew people would enjoy all the extras we had added in to the paid version.  But at some point you just need to walk away.  We shipped Warheads 2.4.0 without the Awesome Sauce.  No one but us even knew what could have been in that release (until now).

The Kill

“I have a hard time letting go of problems that vex me.”

That’s what I wrote in my email to Reabs which included a build of Warheads that I sent literally moments before I dashed off to the airport for a Memorial Day weekend trip to California.  As with many vexing problems I’ve encountered in the past, I solved this one while sleeping.  Really.

I can’t really say exactly how the epiphany came about, but when I awoke in the morning I was inspired to look at the various textures we were using for the Awesome Sauce to see if there was something unusual about any of them (too big, for example).  Others who had encountered this particular problem had commented that fiddling around with textures seemed to make the problem go away.  I also recalled that different Android devices supported different texture formats. 

I inspected the new textures we used and there it was—one texture was not like the rest.  One texture, the one for the “fog” on the planet surface, was not square.

I know.  OMFG!  Not square!

You wouldn’t think that would cause 35+ hours of headache and to be honest, it shouldn’t have mattered at all.  It’s a bug in the platform to be sure.  But once I removed that one asset with the non-square texture, everything worked great.  The game happily played itself well past level 70 with no freeze.

At this point we had another choice—try to resolve the problem and save the for feature, or just punt.  We chose to give up on the fog and launch everything else.  It’s really too bad and this is one of the downsides to the Android ecosystem—we believe this particular issue only happens on certain devices, but we don’t want to exclude those devices from our potential market and we don’t want to have to support multiple versions of the app.  In this case, the deficiency of a small group of devices has caused everyone else to miss out.

This is an image of what it should look like (with fog).  Still, it’s pretty awesome even without the fog.

You can see the latest version of Warheads on the Android Market, complete with Awesome Sauce, here.  It’s a free trial, so go check it out!

Technorati Tags: android,warheads,bugs,indie games,Open GL

Laser tower upgrades for the Walled Garden

There’s already been a ton said about Apple’s recent change to section 3.3.1 of the iPhone Developer Program License Agreement, including this well-known post by John Gruber aka Daring Fireball.  Admittedly it’s hard to add to the conversation, but I’ll try.

First off, if you have not heard about this particular issue you probably can safely skip the rest of this post and go check out something more amusing like xkcd.

Still here?  OK, let’s go then.  The story so far goes something like this…

Apple: Starting with the next version of the iPhone SDK, you need to use Apple’s tools and technology if you want to build iPhone apps.  You can’t use middleware or cross compile tools like Adobe’s Flash CS5, MonoTouch, Unity, etc.

Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

Adobe: WTF?? We just blew a crapton of money making Flash CS5 cross-compile into iPhone!  Go fuck yourself, Apple!  By the way, who’s your lawyer again?

Unity: We really don’t think this is going to be a problem for us.  Really.  We’ve got some good friends at Apple and even though they can’t comment they say they think Unity is really cool.  Plus we talked to Steve.  One time at a party that is.  OK, well it was the WWDC but it was like a party.  And we know Steve heard us even though we were in the ninth row because the nice security person told us it was time for us to leave Steve alone.  But really.  Not a problem.  Everything is going to be OK.  We’re going down to Cupertino this week!  Did we mention that we have venture funding?

Novell (MonoTouch):  Oh no.  Not again.

Apple is purposefully creating a monoculture of iPhone development.  They want to breed the best iPhone apps possible, and I’m sure they will get some amazing, incredible apps built on their amazingly incredible platform.  But as anyone with a Golden Retriever or other pure bred animal knows that purity comes with a price. 

Most people (including me) acknowledge that it’s well within Apple’s rights to make this call.  Apple has never been shy about standing up for the user experience by keeping the riffraff off the streets of the iPhone/iPad/iTouch.  Well, except all those farting apps.  But Steve thought they were kind of funny so that’s OK.  Folks like Gruber point out that this move will ensure that only people who are serious about iPhone development will be making apps.  No shitty Flash ports.  No argument from me on that.  Too bad for all those awesome Flash games like Machinarium, though.

Here’s the problem.  With this change Apple has basically declared that there is no such thing as an “independent” iPhone developer.  We kind of already knew this because of the whole App store approval process, but that whole SNAFU seemed to be on its way to become more liberalized and tractable.  Now there can be no escape.  If you are developing apps for iPhone & co. you are sharecropping for Apple.  Apple will tell you what apps you can and can not make.  Apple will decide whether your apps meet their standards… after you’ve finished them.  Apple will decide what tools you may use, what services you may use (don’t get me started on Section 3.3.9), what middleware you can use, what payment terns you will receive.  And if they decide to alter the terms of the arrangement, they will.  You can take it or leave it because they don’t need you.  There are a hundred others waiting in line for your land if you don’t want to work it.

Personally, I hope that some kind of incredible middleware tool comes along that enables some amazing new games and is not available on the iPhone.  It’s not that I harbor any ill will toward Apple.  On the contrary, I worked there in the ‘90s and admire them for how far they’ve come.  I also own an iPhone and two Macs.  I just value innovation over purity. 

This whole situation leaves developers like citizen12 studio with a question.  Do we make our way in the great open-er frontiers and badlands where we can make use of great middleware technologies like Unity and Flash, or do we want to become sharecroppers in the Walled Garden of iPhone and hope that we’re one of the lucky few who make it?