Jan 12

2018

How to Get the Most out of Automated Testing—Part 1

Posted by: Rachel E. Towers | Comments (0)

As part of our support for the Choice of Games Contest for Interactive Novels, we will be posting an irregular series of blog posts discussing important design and writing criteria for games. We hope that these can both provide guidance for people participating in the Contest and also help people understand how we think about questions of game design and some best practices. These don’t modify the evaluation criteria for the Contest, and (except as noted) participants are not required to conform to our recommendations–but it’s probably a good idea to listen when judges tell you what they’re looking for.

If these topics interest you, be sure to sign up for our contest mailing list below! We’ll post more of our thoughts on game design leading up to the contest deadline on January 31, 2018.

Coding in ChoiceScript is relatively simple, but that does not mean it’s always easy. In addition to writing a compelling story with a number of branches and making sure that the story flows naturally no matter which choices are made, you must also add a level of game design—of tension and uncertainty—to give a sense of accomplishment to successes and of replayability in finding more. This means considerations for things such as game balance must be made. If the game is too difficult, it narrows the options available and eliminates possibilities, while if it’s too easy, second and third playthroughs can lack a sense of tension. This is why we test our games. While testing by hand can be a useful tool for shorter games, or to quickly check something, the longer a game becomes, the more tedious and less useful testing by hand is. This is of course where automated tools come in.

There are a few different versions of the testings tools included in ChoiceScript, but for our purposes the html version is the easiest to use. All you need to do to run it is open up the folder you have your copy of ChoiceScript in, and open the html version of the tool in a web browser. (Like everything related to ChoiceScript development, we recommend you use Firefox.) If you would prefer to use the command line version, instructions on running that can be found here.

Now, if you have any errors that quicktest or randomtest are able to catch, the tools will run until they find them, then stop and say what the error is. These errors usually specify what exactly the problem is, but if you need more detailed help with one, our forum has a category dedicated to coding problems, with a number of questions already answered.

Of course, sometimes we know exactly what the problem is, but because of the limits of randomtest, it’s impossible for it to make a reasonable choice. Say, for example, we want to ask a player to input a password they learned earlier in the game. This is a barebones example of what we might use:

What is the password? (Hint: It’s the name of a fish.)
*input_text answer
*if answer = “swordfish”
	*goto correct_answer
*else
	*goto wrong_answer

When we think about how randomtest would run this, we can easily see the problem. Randomtest obviously has no way of actually understanding the question! (As a quick aside for the curious, when presented with *input_text, randomtest always says “blah blah”.)

For these situations we have a special variable called choice_randomtest. This variable never has to be created or set, it simply has to be tested like any other variable, but instead of a value based on player it choice, it simply returns true when the game is run by randomtest, and false when played normally. This will let us create special code to handle situations such as the above password. Of course we can make randomtest test always pass by simply checking *if (answer = “swordfish”) or (choice_randomtest), but let’s do something a little more complex, and have randomtest know the answer only some of the time. To do that, we’ll need to create a temporary variable, randomize it, then use that to set the answer. That would look something like this:

What is the password? (Hint: It’s the name of a fish.)
*input_text answer
*if choice_randomtest
	*temp coin 0
	*rand coin 1 2
	*if coin = 1
		*set answer “swordfish”
*if answer = “swordfish”
	*goto correct_answer
*else
	*goto wrong_answer

We now see that randomtest “guesses” the answer with a coin flip, returning the correct answer half the time, and failing the other half (with “blah blah” as its guess). We can of course tweak things even more, and make this much more complex (maybe it’s a really hard puzzle and randomtest should get it one out of ten times), but this is a good indication of how choice_randomtest can be used to reflect actual game play where randomtest otherwise wouldn’t.

That brings us to another common use for choice_randomtest. Some games include complex looping choices, such as requiring a player to confirm a selection, or there is an option which gives a player extra information, but doesn’t otherwise change the game. In randomtest, these kinds of choices add time to the playthrough (making randomtest take longer to run), and playthrough word count to the total, but don’t really accomplish anything. (This is also a small part of what we mean by coding efficiency.) For example, here is a simple loop where a player has to confirm a choice. (The confirmation is in a subroutine so we can more easily see how it all works together.)

*label take_a_shortcut
Which shortcut do you take?
*choice
	#Go through the rose bushes.
		It’s filled with thorns.
		*gosub confirm
		*goto rose_path
	#Climb the wall
		It looks very high. A fall might hurt.
		*gosub confirm
		*goto wall_path
	#Wade through the river
		It is running very fast and you can be swept away.
		*gosub confirm
		*goto river_path
	#Give up on 
		*goto give_up
*label confirm
Are you certain this is the path you want to take?
*choice
	#Yes
		*return
	#No
		*goto obstacle

Looking at this, we can see that when making a choice, randomtest would normally select any given choice about 25% of the time, but when given an opt-out, that one option, the easy path, suddenly becomes a lot more common. For example, if you run this through randomtest, it will give up about 40% of the time, close to double what would happen if there weren’t a confirmation, and almost certainly a lot more than a real player would (because a real player is more likely to assess the risks and take the path where they are most likely to succeed). But with one little change we can make it possible for randomtest to not opt out.

*label confirm
Do you continue?
*choice
	#Yes
		*goto obstacle
	*if (not(choice_randomtest)) #No
		*goto take_a_shortcut

In this case, we’ll see that when randomtest makes a choice, it no longer attempts to back out. This returns our run percentage back to reasonable 25% all around. This also has a side effect of speeding up randomtest. Again, we might want to fine tune this for more specific circumstances, but the general idea remains the same.

Looking at this, we see that we can force randomtest to arbitrarily make any decision and run any code. Sometimes we might use this to test specific possibilities in a game by inserting choice_randomtest into our code (and removing it once we’re done testing). For example, say we have a spy game, and while out on mission the player only has room for one hidden tool, and must make the important choice of whether bringing along a lockpicks, a pistol, or a disguise kit, and we lock and unlock certain paths based on what’s selected. Of course we want a player to be able to succeed no matter which tool they pick, and to be certain of that we might need to test what happens with each tool. So we would lock the other ones by an *if not(choice_randomtest), then running randomtest.

Of course, this has to be done with some care, as if randomtest makes decisions a player wouldn’t (or especially one that a player couldn’t) it becomes useless and detrimental to testing. Bad information can be worse than no information at all.

Subscribe by E-mail