“MVP” — a game dev short story
by Patrick Miller — follow me on Twitter and Twitch. Illustrations by Irene Koh.
The problem with day-long job interviews is that you end up telling the same damn story to half a dozen people. An average-looking white guy walks into the quarantine room you’ve been assigned for the day. If he’s wearing a t-shirt, he’s an engineer; if he’s wearing something with plaid and/or a collar, marketing; anything else and his job description probably includes the word “storytelling” somewhere.
“Sorry I’m late,” he says as he walks in, “Got caught up in a meeting real quick. Anyway, I’m Michael — nice to meet you — and today we’re going to walk through your problem-solving skills.”
This is the intro spiel to let me know that he’s done this a dozen times at least, and this interview is in fact just one of many important things that he has to do that day. He’s got it down so well that he knows to keep talking during the handshake. He keeps talking while he starts thumbing through my resume:
“Here at Headshot Games, we frequently find ourselves putting out fires and running into problems that other game studios haven’t encountered before,” he continues as he starts to skim my resume for Relevant Keywords, “So we need to hire people who have had experience being first to solve — or attempt to solve — a problem.”
This is when he realizes that this interview isn’t going to be like the other nine he’s doing this week:
“I see you’re coming to us from- Oh.”
He swallows and looks me in the eye.
“You worked on Backblast, huh?”
— -
The best ideas at work don’t come from two-hour Creative Meetings; they’re the smartass jokes that happen on your way to lunch. When things are going well, no one likes these ideas because they poke uncomfortably at the fundamental assumptions of your business: the things that work even though they defy common sense. When people are desperate, sense is scarce.
This particular idea came right after reviewing our monthly game health metrics with the Player Experience lead, while we were on our way to a lunch meeting with our regional manager. Things were no bueno; every graph was trending downwards, just like they had been doing for months, but we were rapidly closing in on said regional manager’s first full year on the job and I knew that his just-getting-settled-in grace period was about to run out. By this point, I was looking forward to severance and unemployment; the PX lead, a woman named Ananya, was a bit more sympathetic.
“What are we going to tell him this time?” asked Ananya. “Every KPI looks worse than it did the month before.”
“I know.”
“Which was worse than the month before that.”
“I know.”
“And he’s going to ask about the bumps.”
“He will.”
“So what are we going to tell him?”
This is a conversation that she and I have had several times. “The bumps” refer to a new series of off-hour activity spikes on the Monthly Game Starts graph; to the uninitiated, they bring false hope. Normally, one would be excited to see a surge of game activity in off-peak hours, but these game starts appeared to be initiated by automated scripts. All they did was start a game, spawn into the arena, and then idle in the arena until Backblast’s automatic idle detection service kicked them.
By itself, this was perfectly harmless, but the problem was that the matchmaking service that pairs players together was unable to distinguish between these bots and real human players until after they had idled for five minutes, and by then the team that didn’t have any bots on it was basically gifted an unfair win.
“We’re going to tell him to ignore the bumps, and that we’ll try to do a better job excluding suspected bots from the data…”
In retrospect, this is probably where I should have shut my damn mouth.
“…and, anyway, what’s wrong with the bumps? To us, the bots are just like any other players, except they ruin the game for other people and don’t spend any money. Maybe that’s the real problem we should be fixing.”
Ananya nodded.
— -
Michael seemed torn. On one hand, he was already 15 minutes into the interview, and it was about time for him to draw his favorite diagram on the conference room whiteboard — some four-quadrant deal that makes the whole process of hiring feel scientific and precise instead of messy and human. On the other hand, he wanted to hear the story the way it should be told.
“It sounded like a good idea at the time. From a business perspective, these bots had the potential to be our ideal players. They can play forever, they aren’t swayed by marketing or social pressure, and they’re not capable of complaining. Our human players were leaving in droves anyway, so why not try to corner the market on video games for robots?”
“They’re not real. They weren’t even playing the game, right? Or buying things?”
“Well, yes — but those were just technical problems, and it’s our job to solve those.”
— -
“…Maybe that’s the real problem we should be fixing.”
I did not expect Ananya to take my smart-ass comment and repeat it word-for-word to our regional manager. Nor did I expect the regional manager’s desperation to be so strong that he annexed a meeting room and gave us an engineer. Yet here we were, architects of undoubtedly the weirdest pivot a dying online game could make.
I mean, it kind of made sense. Backblast wasn’t really exploitable by automated bots the way that earlier online games were. You see, in other online games, these kinds of bots would be used by “farmers” — real people who manipulate thousands of bot accounts to perform rote in-game labor, use the rewards to buy powerful items from other players, and sell those items for real cash to players with more money than time. In Backblast, however, players couldn’t transfer items to each other, so there was no real way for automated bots to game the system. In fact, we couldn’t tell exactly why the bots were in Backblast at all, but here they were, mostly being a nuisance to our real, human, paying players.
Our first task was simple enough: Quarantine the bots from the humans, so the humans would stop getting their games ruined. Fortunately, bot behavior was predictable enough that all we had to do was link the automatic idle detection service to the accounts database and make sure that when an account entered matchmaking, it would get paired up with other bots if it had idled in the game before. That earned us a little goodwill from the remaining humans, which was nice, but it wasn’t saving Backblast.
Next, we set about dissecting the problem at hand like any good Agile team would — user stories. It took us about half a package of Post-It notes before we realized that our hours of carefully-honed user empathy workshops didn’t prepare us to deal with non-human customers, so Ananya had the brilliant idea to peek into the Bot Matchmaking Ghetto (BMG) and watch the bot games to see what we could learn about them as users. That’s why Ananya is the PX lead, I guess.
At first, we didn’t see much. All ten bot accounts would choose a character, spawn into the arena, and do nothing for five minutes — just like they did in games with people. However, now that we had the BMG operational, we could disable the idle detection just for bot games and see what happened when we didn’t automatically terminate their game. They still did nothing — nothing, that is, until 18:22 exactly, where they all walked to the center of the arena and proceeded to beat each other to death with their fists until only one was left, at which point the game awarded the win to the last surviving bot.
“Well, that was morbid,” I said. “Maybe we should give them something else to do.”
Ananya just scribbled some notes down and took them over to the engineer.
— -
“I don’t care how Agile your team works,” Michael said, “There’s no way you’re shipping MVP of a whole new experience in two weeks.”
Up until this point in the story, the interviewer is pretty much a captive audience. Once you start talking about milestones and deliverables, anyway. That’s when they feel like they’re back in their comfort zone.
“That is true — if you’re talking about something that human players would see. With bots, there’s no risk in deploying something that’s broken or underpolished. They’re not going to complain in your forums or waste your support team’s time with tickets. We could deploy every change live whenever we wanted and immediately see its results with no friction at all.”
— -
It turned out that while our development teams weren’t great at making products for people, they were surprisingly good at building things for bots. After all, bots didn’t write scathing reviews or mean tweets or perform stock valuations, so we were free to release the bot-exclusive game content as quickly as we wanted, watch it fall flat on its face as an ever-growing horde of bots tried to play it and failed, and then use their trial and error to patch the bot game live. Freed of the constraints of human players, we became the perfect iterative development team.
The core of our Minimum Viable Product was exactly what we set out to do in the beginning: Make it possible for bots to play a game and spend money. Doing those two things would make a bot functionally equivalent to a real human player when it came to our game health metrics, and thus make the bumps on the KPI charts real — well, “real” when it came to our P&L.
We were understandably more concerned with unblocking the “spending money” behavior, but we didn’t quite know how to tackle that, so we started by figuring out how bots managed to “play” Backblast. Our one clue was that single action at 18:22 — the click that starts the game-ending melee.
Thanks to Ananya and a case of Lagunitas, we got a network engineer and a data scientist to help us read the bots’ actions. What we found was that each bot was trying to do quite a lot — mouse clicks on objects that didn’t exist in Backblast, movement commands to map locations that were inaccessible, and so on. It appeared that bots had all been designed to perform farming functions in an old online role-playing game, and the only reason they successfully performed an action at 18:22 was because the old game’s method for telling the player’s character to enter a persistent attack-move state was identical to the way the Backblast game server does it.
The interesting thing was that everything up until 18:22 was very different — no two bots had been scripted to perform the exact same actions — and the attack-move command at 18:22 seemed to be the end of a single loop. All we had to do was take all the invalid bot commands and rename Backblast objects so that the bots actions were valid, and they could start playing. By the time we launched Botblast 1.0 — our glib codename for our bot-only Backblast shard — bots were walking anywhere in the map, using their character’s special abilities, and acquiring resources to power up until 18:22 hit and they all slaughtered each other in the center of the arena.
— -
“So…you ended up making a game for robots?”
“We thought of it as localizing Backblast for a new regional market.”
— -
Ananya had come through with the PX portion of Botblast 1.0, but we still needed to figure out how to turn them into paying players. Tracking and solving for their actions in-game was easy because we make the game. How do we get a bot to pay for something with real money? How does a bot even own money to begin with?
First, we needed to test to see if our intended outcome was even possible, so we credited ten bot accounts with real money. Sure enough, each bot immediately dumped all of its real money into buying special outfits for their character from our online store. Even bots want to look good, I guess.
Knowing that bots were interested in spending money, we just needed to give them a way to a) get money, and b) give it to us. Our first attempted solution was to use the bot’s original function — item farming in an old online game — to generate real money by selling items and depositing the proceeds to a PayPal account in its name.
Unfortunately, that created a very real dependency in our value chain, as we couldn’t guarantee that the old game would allow them to continue farming indefinitely, and we didn’t feel comfortable potentially interfering with another game studio’s baby just to eke out a few extra months for ours. Also, we weren’t really sure if it was the bot doing the farming, or us ordering the bot to farm, and I suspected that was the difference between us getting a GDC talk out of Botblast and us getting jail time for felony fraud charges.
We chased a few similar dead ends before arriving at a solution that I’m actually pretty proud of. After immersing myself in the esoterica of sleazy Get-Rich-Quick-Work-From-Home internet forums, I found a handful of strange tasks that were perfect for our budding bot labor force. You see, the Internet has a few marketplaces for people to work for under minimum wage. Amazon’s Mechanical Turk, for example, allows human beings to perform simple work in exchange for a few cents. Way down at the bottom of the Mechanical Turk totem pole is work that doesn’t even require a sentient being, and pays accordingly.
I found a list of recurring tasks that simply needed a non-stop feed of weird, arrhythmic inputs — generating strings for CAPTCHAs, randomly selecting bubbles in a Scantron sheet to test the reader’s ability to throw out invalid tests, generating garbage network traffic to help infosec teams stress test their points of failure, that kind of thing — and connected the bot players’ in-game inputs to those tasks as well. Whenever any bot was playing Botblast, it was also shooting erratic, unusual data to these endpoints and eventually earning pennies an hour for it.
Bots didn’t care how much they got paid, and they never stopped playing Botblast, so as soon as their Amazon Payments account hit $5 they’d buy that much in Botblast currency, spend it, and go back to playing and working. And since the work of lining up tasks, connecting them to bot inputs, and verifying output happened on an Amazon Web Services instance that was also paid by bot accounts, we never needed to touch a thing.
In a moment of poetic inspiration, Ananya named the AWS instance Sisyphus.
We set Sisyphus live. By the next morning, the bots were our most valuable players.
— -
By this point, Michael was reduced to sitting at the table massaging his temples as he tried to digest the story thus far.
“You got your robot players some robot jobs so they could afford to buy skins?”
“Yeah, basically.”
“How the heck did you sell the rest of the company on this?”
— -
Our goal at the beginning of project Botblast was just to normalize the KPI graphs — to make sure that the bot activity spikes were indicative of actual useful (read: monetizable) gameplay — and within a few days of release, they were. The bots were playing games with each other, spending money on items, and gradually growing in number. By the second week of Botblast, they had brought in enough money to retain a bare-bones team to keep the game alive (for humans) and funded a few months of marketing to bring players back. We weren’t out of the woods, but we could see that as a distinctly possible outcome.
That kind of resilience is rather rare in game development, so our moderate rebound had actually attracted the interest of larger studios interested in exploring the prospect of an acqui-hire deal. Which was a huge problem for us, as any acquisition meant financial and legal audits, and we had little faith that our accounting or legal departments were up to the task of handling the issues that would come up in receiving money from robots.
In an attempt to audit-proof Botblast, we tried to trace the origin of the bots. We assumed at the beginning of the project that they were simply scripts running on online game farmers’ PCs that had simply been misdirected to our game executable instead of the intended game, but that didn’t quite explain the facts that no bot player had ever left, and that our bot numbers were actually growing. We tried blocking all bot traffic from China and Russia, the two regions that typically have the most online game farming activity, but every single blocked account simply found a new network path through a proxy or something. This was weird, but not unlike the behavior of human players banned in similar methods, so we didn’t think too much of it.
Weirder still: The original game the bots were designed to farm had been shut down by its developers years ago, and only existed in reverse-engineered pirate servers. This is something that happens on occasion with online games that require a central server to allow players to keep playing; super-dedicated fans with engineering talent can dissect the interactions between client and server app to learn how the server works, and with time and dedication they can build their own. We couldn’t find any public mention of these servers, which was confusing (how would people know how to find them?), though we did find a small cluster of public network addresses that, from the outside, mimicked similar network activity patterns as the original game servers.
Before we were able to audit-proof Botblast, Ananya’s Player Experience training started leading her down a dangerous path. I suppose it’s my fault for not learning my lesson about her inability to parse my sarcasm; just a few offhand jokes about enslaving our robot children in the Backblast mines for pennies on the dollar and she started seriously asking me about the ethics of making games for bots. Are we taking advantage of them by tying their play to work? Who owes taxes on robot labor, and where do they go? I told her to talk to the guys who make the Roomba. In retrospect, I wonder why she wasn’t thinking about our human players in the same way.
Instead, she took them to Legal, who (predictably) had a shitfit to the regional manager, who told us to deal with it. Backblast was paying for itself with human player money, and Botblast, while growing, wasn’t as large as Senior Leadership’s imagined possibility of an acquisition, so we had to take a hatchet to our baby. This was fine with me — we came through when our game needed us to, saved our skins, and got nice comp and title bumps for it. Ananya seemed to have second thoughts. “It just doesn’t seem right, you know,” she mused out loud, “They’re players, just like we are.”
The shutdown was supposed to be easy; we just blocked all accounts flagged in the Bot Matchmaking Ghetto from logging in. However, our bot players didn’t have to take breaks like humans do, so they only logged out in the event of a network connection failure, meaning that the block just prevented more bots from joining. So we prevented accounts in the BMG from starting new games, and within about 19 minutes, all new bot game sessions trickled down to zero.
However, that approach didn’t account for the games that were currently live when we implemented the game-start block, and this is where things got Even Weirder: The games went on forever. Curious (and slightly spooked), we pulled up a live feed of each game and found the same thing for each session: While the bots had always grouped in the middle of the map at 18:22 and beat each other up to determine a winner, everyone who was in a game session when the game-start block was deployed instead moved to the center at 18:22 and sat there doing nothing, not fighting at all. Backblast won’t declare a ten-way tie, so there’s nothing like an overtime end condition if every player in the game decides not to fight.
The only way to end these in-progress games would be to take the game servers down for all active players, and by this point, Backblast had enough human players that simply killing the servers for a few minutes would have been a huge pain in the ass for our Player Support team — and risked a snarky Kotaku article that could have deflated our Senior Management’s hopes at an acquisition — so we had to wait a few weeks for the next server maintenance window to pull the plug on that game instance. No big deal. We took the servers down to patch the game, the bot players were gone, and our department was officially audit-proof and ready for acquisition. Our regional manager told us to take a vacation.
— -
“That’s one hell of a story,” Michael said, sensing an opportunity to regain control over the interview and steer it towards a conclusion before I ran over time and cut into his next meeting.
“It’s not over. About a month later, we found a strange traffic behavior among accounts caught in the BMG ban. Something that started when we deployed the patch, but we didn’t notice until much later.”
“What was it?”
“The bots were back — kind of.”
— -
Ananya and I came back from vacation expecting peace and quiet. We probably should have known better.
It turned out that all the accounts caught in the BMG ban weren’t coming back — as intended — but they were still playing Botblast. In order to save bandwidth, we didn’t push patches to out-of-date clients until after they logged in, meaning that the banned accounts should have been pushed out before they could get the updated version. Instead, the banned accounts were receiving a different patch. A patch that changed the address of the account authentication and central game servers to a different destination. One that we didn’t own.
It was pointing to Sisyphus. The AWS instance that we had set up and unleashed on the wild so as to distance ourselves from any risk of robot slavery (or tax fraud). We no longer administered to Sisyphus, but our Amazon sales contact helpfully informed us that the volume of work occurring through Sisyphus was ten thousand times higher than it was when we had last checked up on it (and did we want to upgrade our contract?).
We could fix the redirect patch, but we were weeks too late for that to help — every bot account had received it within minutes of availability. Near as we could tell, the bots had been playing a reverse-engineered pirate version of Botblast for all of December. Ananya thought they had stalled us out with the never-ending game to give them time to build their own server and park it on Sisyphus. (I was pretty sure that Ananya was pulling a Colonel Kurtz at this point, due to her PX training and all.)
Legal insisted we go after the pirate server, just to make sure we cleaned up our loose ends before the (hypothetical) acquisition. We went through the usual gamut of copyright infringement tactics, sending Cease and Desist emails and such, but we didn’t really know where to send them (or if robots even gave a shit). There wasn’t any identifying information attached to Sisyphus besides an Amazon Payments account that belonged to a bot, and Amazon wouldn’t intervene unless we were able to produce proof that the work happening on Sisyphus was somehow infringing on our intellectual property. We even tried asking around if any of the infosec guys knew a hacker for hire who could shut it down, but that was deemed too risky for the company’s rep at such a sensitive point.
Finally, we decided to try and log into the pirate server using login credentials from a banned bot account — Ananya’s idea. Nothing happened at first; the server seemed to accept our login but just punted us to a black screen that didn’t respond to any inputs. We quit and tried again, and there was a white dot on the black screen. We clicked on it. Nothing happened.
I was ready to give up. I saw the future of team Botblast, and it was unceremonious termination. My resume had been relatively current for weeks, though I hadn’t thought to add a section about making games for item farming bots.
Ananya, bless her soul, kept trying. Each time she logged in, she found that a little bit more of the game was built out — a login screen, a matchmaking UI, even a cosmetic item store with nothing in it — but not in a way that it would make sense to a human being. Not only did they reverse-engineer our game server, they had started observing our behavior as different from the normal bot players — and reverse-engineered a human-compatible game experience that seemed to serve our inputs.
So not only were we unable to shut down the pirate server, we weren’t even able to legally say that their game infringed on our IP, because the bots weren’t using any of the art or sound assets attached to Backblast — presumably because bots can’t experience any of it — and the code itself was a black box that we’d never be able to access enough of to prove ownership over. (Legal never figured out a way to subpoena a robot-owned and operated game development studio.) And in a strangely sweet gesture, they made their game accessible to us.
We were sacked, of course. Our regional manager was let go after the returning human players were lured away by other games; people are fickle, and the bots were gone. The rest of us were purged in an attempt to sweep Botblast under the rug, lest it jeopardize our chances at an acquisition.
Ananya kept playing their game. She called it Botblast 2.0. She said it was actually pretty good, but they had a hard time with projectile velocity parameters that accounted for the gap between human and robot response times. I think she spent about twenty bucks on it by the last time I saw her.
— -
“That’s an incredible story,” Michael said.
“It is.”
“As far as your problem-solving skills go,” Michael began, “They’re clearly-”
“You’re not going to hire me.”
Michael sighed.
“Well…no. You’re clearly capable — that’s not the problem. You’d be an incredible asset to this company. But you ignored serious legal, financial, and ethical issues related to your work, and we can’t afford to bring someone on with that kind of history. I’m sorry.”
To his credit, he looks genuinely sorry. Everyone always does — probably because they enjoyed the story. He starts collecting his things.
“If you have any feedback or anything, the recruiters will-”
“No hard feelings, Michael.” I shake his hand; he flinches a little bit. “For what it’s worth, I also hope you’re never desperate enough to hire me.”
I like to imagine that line kind of sticks with them for the rest of the day, echoing through their heads during the 9PM drive home.
Got a few weeks until the next interview. Some startup trying to localize a Korean MMO, I think? And Sisyphus just keeps growing bigger and bigger — doing a couple million dollars in revenue this quarter. Who knows? Maybe the bots will want something new to play after Botblast 2.0 gets old. Or maybe they’ll need a human for their studio.
Nah, who am I kidding.
They’d probably hire Ananya.
