2011-02-17 : Ben Lehman: Playtesting: Stop
a guest post by Ben Lehman
Before I start, I just want to thank Vincent for giving me some space on his blog to discuss this. Being able to guest-post on anyway is pretty swell. Oh, and also, understand that the "you" in this essay is addressed from me, an amateur game designer, to other amateur game designers. If you're not a game designer, have fun reading some inside baseball. Okay, now on to the essay.
You need to stop playtesting. It's not just that you're doing it wrong, it's that you're doing it wrong in ways that hurt your games. Furthermore, by promoting a culture of design in which playtest is held up as the be all and end all of the design process, to which all other elements of design must bow down, you are causing at best cause other, newer designers to feel inferior and inadequate and at worse cause them to mutilate their own designs in ways similar to which you have mutilated your own.
Playtesting is fucking dangerous, and you need to stop doing it, stop talking about it, and stop using it as a substitute for the hard work of game design.
(I'm not saying don't playtest, btw. There is a very specific place for playtesting in the design process, which we'll get to at the end. I am saying that you need to stop playtesting because it is a terrible tool. If you were to see someone pushing a power drill towards their eyeball, you'd say "STOP DRILLING!" and not take the time comment about that the drill does have some appropriate uses.)
Here is a short sampling of the things that I regularly see people use playtesting for, but are terrible, no good, horrible ideas: Identifying rules and textual errors, mathematics and probability analysis, marketing and advertising, developing or finishing your game.
Let's take each in turn, shall we?
(A brief aside to define playtesting: it means an appropriately-sized group of people sitting down to test an unfinished game, putatively to assist in its design process.)
Textual Errors:
I hear people talk, sometimes, about "textual playtesting," where they use playtesters to test the coherency and completeness of their game text. This is a terrible idea, and results in mangled and incomplete texts and overconfident authors.
No one actually reads RPG texts. No, seriously, they're just not part of play. Whether or not your rules cohere in play has much more to do with your game's similarity to other games that they've played, and very little to do with the contents of the text. Your text could be totally complete and clear, and many RPG groups will muck it up anyway. Contrariwise, your text could be riddled with procedural and textual holes, and most groups that would playtest for you could make it work correctly.
Furthermore, since the effectiveness of a draft text is largely rooted in the group's prejudices, "textual playtesting" by groups who know who you are (and thus are willing to playtest your game) is not only likely to result in a chopped up, incoherent text, but is going to highly prejudice your text to be just like other games that you and your group of playtesting connections are familiar with. If you, like me, are interested in innovation, you can see the problem.
The right thing to do about textual errors is twofold: First, follow an effective and coherent didactic strategy; second, hire an editor who will provide you with structural feedback. Once your rules are already firmed up, take a look at good textbooks and board game rules and Jack Chick tracts, look at the different ways that teaching in text can be made to work effectively, pick a strategy, and pursue it with gusto and consistency. Then, turn it over to an editor who understands both good style and your goals and take almost every one of their suggestions for re-ordering and rewrites.
After this, I guess it couldn't hurt to turn it over to players for a round or two, as long as you don't compromise your didactic goals and run it through editing again, from scratch, if you make any changes. Personally, I'd just skip it: I think a competent editor knows better than any random group of gamers.
Rules Problems:
Okay, so sure, your game text shouldn't be playtested. But you should definitely test to look for rules holes and rules problems, right?
In a word, no.
In general, role-playing game rules are pretty simple. It's possible that your game is as complicated as Magic: the Gathering or Power Grid, in which case, yeah, exhaustive testing is the only possible way to discover that one particular subrule interferes in a bad way with some other particular subrule. Even then, it's not exactly a good tool: it's the best of a lot of terrible options.
But if your game is simpler, like say 3rd edition D&D (and most of my audience of game designers is producing games which are way simpler than 3rd edition D&D), you should have determined and fixed all rules holes well before they reach playtesting, not only because it is insulting to your testers and wastes their time, but because playtesting is simply a bad way to find rules holes.
(Let's define what I mean by "rules hole." I mean a gap where procedure of the game falls through: there's simply no way to proceed in the game. Also, I'm discussing "rules problems" which are rules that have detrimental effects on play: most commonly infinite loops or insufficiently thought-out mathematics, but also other things, like division of play responsibilities in such a way that it violates the Czege principle.)
Let's unpack that. Why is playtesting a terrible way to find rules holes? Simply put, unless you test exhaustively, you're not going to find all the rules holes and problems that are present in your game. This is almost tautological, but playing can only test the combinations and interactions which come up during play. All other possible combinations and interactions and rules uses are going to remain untested. Any rules holes or rules problems in that set are going to remain, often with detrimental effects on your game once it gets out in the world and all those unexamined holes are revealed through play.
For finding these problems and holes, though, playtesting isn't just inefficient and incomplete, it's also ineffective. Even considering the long odds that a particular rules hole will reveal itself during a playtest, most role-playing groups are not willing to simply leave a rules hole as a rules hole. They will patch it with their pre-conceptions, prejudices, and group social contract. Furthermore—and there's some interesting social context stuff here which I don't want to get into in this essay—they are not likely to even remember that the rules hole came up: once they've papered over it, it may as well not exist. The chances of actually getting them to report it to you, the designer are nearly nil. Even if you, the designer, are playing, the chances of you remembering it as a hole are low.
The right thing to do about rules holes is to take your rules set, once it's finished being written and modified and so on, and critically evaluate every possible rules interaction, looking for holes. Use a pencil and paper, if you're like me, or a blank text file, or even just your own head and the shower (although this can be dangerous: see above about memory holes.) Yes, this is a huge pain in the butt and it takes a long-ass time. It's also some of most important work you can do as a game designer, and if you're not willing to do it you should take up some other creative activity. The procedure for finding rules problems is similar, but requires even more critical faculty on your part.
Once you get good at this, you can start taking short cuts: "okay, so let's look at all abilities that work like this: can any of them cause unexpected interactions?" But ultimately you're going to have to do this a lot.
It's possible that your game has a lot of complicated interactions and nested cycles, in which case you may not actually be able to handle this. In this case, there are a couple of strategies. First, you could simply run through most of the common of the interactions, confirm that they basically work, and hope that no exotic ones come up in later play. Second, you can really get into your game, understanding its internal logic and appropriateness, which lets you review much faster, and does involve a fair amount of play (even solitaire, see below.) Third, you can change the mechanics of your game to include a cybernetic control system with the human players, which I will get into more in the next section.
A great tool for all of this is the imaginary play session. Just sit down and take on the role of several different players, with different personality and goals, playing your game. Figuring out how it works, from the inside, is a huge step towards the "logic and appropriateness" above.
Testing Mathematics and Probabilities:
This is, in many ways, a subset of the above, but I see people getting this wrong in such blatant ways that I think it deserves a particular call out. Playtesting is a terrible way to test the mathematics and probabilities of your game, which includes things like if the resolution system "feels whiffy" or whatever. Playtesting is absolutely not the place to determine this stuff, because no amount of playtesting will produce statistically significant results, and most games' probabilities are easily calculated anyway.
You know, I could write several paragraphs trying to explain that, but I don't know if I can do it in a clear way. Let me say it again in all caps: NO AMOUNT OF PLAYTESTING WILL PRODUCE STATISTICALLY SIGNFICANT RESULTS.
If you need me to clear that up for you, just ask in the comments.
The right practice, here, is to determine the results distribution of the game through probability calculation. I just do this with a pencil and paper, counting up probabilities on my fingers and applying my college probability class. If you're more spreadsheet / programmer oriented, you could do it with the monte-carlo method: have a computer run millions of tests of your mechanics and determine the probability distribution that way. If you're not into math or programming, just get a friend to help you: a lot of folks, including me, will do it pretty gladly.
It is possible that there are enough complicated loopy bits in your game mechanics that they elude rigorous calculation. Bliss Stage had this problem. The solution I implemented is to give a human player (in this case the GM) a throttle which they can use to accelerate or decelerate the mathematically unstable parts of the system (in this case, number of interlude scenes and, to a lesser degree, trauma spending strategies.) An additional advantage of this method is that it is called, I shit you not, a "cybernetic control system" which makes it 10 million times cooler than any other rules subsystem. (Dogs in the Vineyard also does this, with the Give rules.)
The absolute best practice in this case is to use simple enough mathematics that you don't need a lot of calculation to determine the probability distributions. Apocalypse World does this quite admirably, as do many other games.
Marketing:
You know all the stuff Vincent has been saying lately about the social context of game design, about your target audience, and how your game speaks to them or fails to speak to them? You know what won't help you find, develop for, speak to, or interact with your target audience? Playtesting.
If you don't have a target audience in mind, or if your target audience is something inaccessible or lame, your problem begins well before playtesting, and I really can't help you. Nor can anyone else. But if you know your target audience, and so on, and so forth, playtesting won't really help you unless your playtesters are composed of your target audience. Furthermore, even in that case, playtesting isn't the beginning of a conversation with your target audience. The beginning of the conversation is publication, and the continuation of the conversation is play.
Again, let's unpack. As a designer, your goals operate at both social and personal levels. At the social level, you're looking at the place you want your game to occupy in society and, in particular, in your subcultures. At the personal level, there's the aesthetics and premises you want to convey to each player of your game.
Playtesting fails at the social level simply because you can't "test" your games social impact: it's presence at that scale is basically a one time thing. There's no parallel societies in which to release your game once you've tested it in one.
(There's a possible exception here for people who write in languages other than English: you can use your native market as a test-bed for the global market. But if you publish in English, your first publication is going into the global market, with no takebacks.)
Playtesting fails at the personal communication level because a game is either not yet communicating what you want it to, in which case it needs further development but definitely hasn't succeeded in your artistic goals; or it does effectively communicate your goals in vision, in which case it's done and needs to be published.
In either case, playtesting is a means to prepare for an audience, not a means of finding or engaging with one.
Furthermore, in the land of sales and moneys, playtesting is pretty terrible advertising. While public playtesting can be a good way to generate "buzz" for your game, it doesn't often generate the sort of buzz you want (it's mostly buzz in a small group of insiders), in general, by the time that playtesting reports are good publicity for your game, playtesting itself is largely superfluous. Either playtesting is revealing problems with your game, in which case it's not good publicity, or its time that you should write your rules text, edit it, and publish it, in which case, stop playtesting.
There's a sidenote here, which is that there's a culture where endless playtest is, itself, a social entrance and a means of gaining social cred. To some degree, our over-playtesting has created this culture: you're not really "in" unless you're playing some unfinished game. For reasons both above and below, this is a terrible, terrible thing for actual game design. (I'm hesitating on elaborating here: I can probably unpack this more in the comments if anyone wants.)
Development:
Playtesting is an awful means of revising or developing your game rules. A lot of people seem to think that the process of playtesting is about revision, but in fact most rules revisions should come well before playtesting (see above), and the few remaining rules revisions should come well after playtesting. Never, under any circumstances, should a playtest group be revising the rules of the game. Neither is it a good practice to revise the rules of a game during a playtest.
The exact wrong people to tell you how to change your games rules are the players of the game, who are in the midst of the game's emotional and social manipulations, and cannot clearly see your design goals nor your own style. Allowing playtest groups to dictate rules changes violates your game's direction and focus, and results in confused, chopped up game texts full of a hodgepodge of different techniques from various people's favorite games. It drives against coherency and against enjoyment.
Furthermore, the amnesia of the game group (which was mentioned above) means that any rules changes that emerge from play are likely to be negotiated through the social consensus and prejudices of the group. It is highly unlikely that these rules will emerge in a state where they make any damn sense to anyone who wasn't a member of said group. So in addition to the scattering effect of letting playtesters develop your game, whatever rules changes that are made are likely to be superstitious garbage.
The best practice here is to design all rules changes yourself, in the context of your own understanding and testing of the design. No one knows the creative vision and focus of your game better than you. If you are unable to work out a solution to your rules issues, remember that playtesting is not a solution. Probably the best idea to to either let the game lie fallow for a bit, or to discuss the issue with trusted fellow designers, members of the target audience, or good friends (ideally: all of the above.) Generally speaking, in discussion, your own ideas are going to come to the surface and they'll fit much better.
("But what about playstorming!?" Go ahead. Ask me that. I dare you.)
Finishing Your Game:
I've saved the best (as in: the worst) for last. This here's the thing which I've read which most makes me aggravated: "I've taken a lot of time to carefully playtest the game. No rush to publication here! I took seven years testing it." This is absolutely bullshit. And it's not just, like, horrible self delusion that sitting around with your finger up your ass is making your game better. Nor is it just that you're absolutely wasting your own time, as well as the time of all your playtesters. This sort of thing is corrosive to the culture of play and design and you all need to stop this bullshit right now.
Rule of thumb: playtesting is like being engaged to be married: after a year you're just wasting your time. At full press of playtesting, a year is the absolute longest it should take to get a game into shape (it's okay to take longer than that when you add in writing the text, editing, layout, printing, marketing). If it takes longer than that there are a few possibilities: Your game is done and you should stop dwaddling and just publish already; you are not a good enough designer to finish this game right now and you should work on other projects and come back to it; you've been slacking off on your design process, which is fine but slacking off is not a fucking virtue; your game will never be good and you should stop wasting your time with this project and move on to others; you are artificially extending your playtest because you think it gets you more social cred to have games in development than actually ever finish a game; your game really is quite complicated and it just needs a little more testing to be great.
The reason that this is so harmful and corrosive is that it serves to make game design seem remote and inaccessible to novice and amateur designers, which is exactly the wrong thing for them to learn. "Oh, someone like me doesn't have the resources / time / commitment for the seven years of playtesting that real pros have to do. I guess I can never be a real game designer." Or, worse, they get caught in a cycle of endless fruitless playtesting, rather than finishing projects and learning and growing. Even worse, letting a playtest group mangle their game as described above. This is what happens when we let procrastination be turned into a virtue.
The best practice here is to shit or get off the pot. Failing that, stop acting so sanctimonious about it.
When to Playtest
Despite all of the above, playtesting is actually an important part of game design, it's just that it is a limited tool and, when misapplied, it is detrimental to both games and texts. So what is playtesting good for, anyway?
Playtesting is necessary for revealing problems with the parts of your game where the mechanics and processes of your game interface with the players at the table, and most particularly with their imaginations and social interactions (this is a broad definition of "mechanics and processes:" including such things as who speaks when, the game's setting, character-player relationships, and so on.) It is only useful for revealing problems, not resolving them, for the reasons noted above. It is useful for the parts of the game that rely on imagination and social interaction because these are the two things which you can't account for procedurally, and so problems there are invisible to your individual testing and calculation.
I should add a word about fun here. Included in this is whether or not the game is any fun (or, as I prefer to think of it, satisfying). For role-playing games, satisfaction is an emergent social property: we like what the game is doing to our imaginations and social interactions. There is, of course, a taste element to this, and because of this you should keep in mind the social context of your design and your target audience, just as you should throughout the whole design process. But there's also a lot that isn't a taste thing. Many games, particularly those in development, simply do not have that spark of satisfaction, and playtesting is required to reveal that.
Even then, playtesting is not a particularly satisfactory tool for determining these problems. Even with a great deal of playtesting, done appropriately, it is possible you will miss certain creative or social interactions which will cause problems with your game in play. But the problem is that there's simply no other way of knowing. Playtesting isn't miraculous, it's just the best of a bad job.
Of course, fixing it involves a lot of design work and creativity. The process is still a little misty to me, although I know it involves looking at how the moment-to-moment process of play is interacting with the goals for play, as well as a healthy dose of critically examining said goals altogether. Ultimately, I and most other designers rely on bolt-out-of-the-blue insights to resolve issues at this level.
The Takeaway
More playtest does not magically make your game better. Neither does it fix most problems that your game might have. It is not a replacement for skilled writing, a creative and technical vision, editing, inspiration, or game design work. It is a very specific tool for finding very specific problems, and attempts to use it beyond that are detrimental both to your own game design and to design culture in general.
It is also important when we embark on a playtest that we respect our playtesters' time and attention. Relying on playtest to point out textual, procedural, and mathematical flaws which could be more easily spotted with simple analysis is degrading to our playtesters and also simply bad game design practice. Stop it.
1. On 2011-02-17, Mendez said:
I will ask the obvious question. Ben, what does playtesting do, if not these things?
2. On 2011-02-17, Marco said:
I've felt the focus on "actual play" reviews was misplaced for almost exactly these reasons.
-Marco
3. On 2011-02-17, Erik Amundsen said:
And one great weight lifted from my shoulders. All this time I thought I was just playtesting wrong.
4. On 2011-02-17, Bret said:
"(I'm hesitating on elaborating here: I can probably unpack this more in the comments if anyone wants.)"
Please unpack.
5. On 2011-02-17, Grover said:
Sometimes drills are useful and appropriate tools. You talk about how, when playtesters find holes in rules, they patch them, and then forget patching them. How is this a problem? If your rulesystem has a hole that can be patched that seamlessly (a group which was trying to find holes failed to find it) then that hole doesn't really matter. Playtesting should find the holes that actually interrupt play.
6. On 2011-02-17, Todd L said:
Ben is trying to fight sanctimony with sanctimony.
Bad idea.
The whole rant should have been written as 'we' shouldn't do this, or 'we' should do that. Not a finger-pointing 'you'.
A few years back, pro-playtesting advocates framed their points in absolutist terms.
Pity the poor aspiring designer who followed the advice of his betters, only to now be admonished with brimstone bluster from the opposite side.
Or don't pity them. Maybe they had it coming for being fanboy sheep?
The pro- and anti- playtest t each-other, rather than at the student body. Firey sermons are crowd pleasers, such as indie design manifestos and seminars; but the approach is more Glenn Beck and less 'bed practices manual'.
Gawd. Who died and made me the arbiter of style, eh?
7. On 2011-02-17, Todd L said:
'The pro- and anti- playtest camps should address their points to each other and not at the student body.'
8. On 2011-02-17, Luke said:
Be a good writer.
Hire an editor.
Have a vision and stick to it.
Don't overdo any one aspect of the design process.
Did I miss anything?
I think your points about testing the game's mechanics and text in play are overstated. An RPGs mechanics are more than just statistics. They must have the proper flow and feel in play. This flow is something that you just can't simply write and wish into existence. As a designer you must witness how the rules are applied in play and design around that interaction.
And a game book is both an instructional text and a reference manual. Using it in play helps disover what's missing. This is a vital aspect of playtesting. The designer is just as likely to suffer amnesia when designing as the players are when involved in testing. Balancing design with play helps find the blind spots.
9. On 2011-02-17, Eppy said:
One year to finish a game? Bah! That is exactly why playtesting is for suckers. Now, if you were playstorming, you'd have that game done in a week. That's how real designers do it.
10. On 2011-02-17, jaywalt said:
Ben, you have some good points here, but do people who've done playtesting really think it solves problems by itself? In my experience, it's not much different than just playing some "finished" game that has some problems. You still have to go out and actively fix the problems in some way, either at the group or rules level.
Generally, I don't sit down to playtest games; I sit down to play them and have fun, no matter the state they're in. If I don't expect them to be fun, I don't play those games! And sometimes the result is "this game isn't sufficiently fun yet; guess it still needs some work." And I feel the same way in that regard about my own games, other indie designer's games (even ones that are supposedly "finished"), and D&D4E.
And then, somebody has to take it upon themselves to figure out how to make the game fun before I can get excited about playing it again. That person can be me, some indie designer, some guys at Wizard (hey, Castle Ravenloft is hella fun! It's nice that they finally "fixed" 4E!), or whoever else.
11. On 2011-02-17, Ben Lehman said:
Thanks everyone!
Mendez: See the "When to playtest" header.
Erik: You were seriously in my head as I wrote this.
Grover: If rules are patched without any sort of record of their patching, it results in mangled and incomplete text. Additionally, most gamers are terrible at patching rules in such a way that they work outside of the immediate context of situation and playgroup.
Todd: "We" would have been inaccurate, no matter how warm and fuzzy. Additionally, every header in my essay includes best practices (usually the last paragraph or two.)
Luke: I agree wholeheartedly with your first point. With your second point, I think a good editor is way better than any amount of playtesting.
Jonathan: You don't playtest. Okay. Why are you commenting?
12. On 2011-02-17, Tyler Tinsley said:
ben I like you, but...
By this point I have started and stopped writing this comment a few times tyring to find the right way to put this. It is now an embarrassingly long reply.
TL:DR
making games is really hard. Play testing is an important tool but it must be used with skill and inside of a functional development process.
While I can tell you have put a good amount of thought into your arguments. I think I'm having a disconnect because the development process your using and seeing used is vastly different from other genres of table top games.
I will outline my process and you can tell me where we differ. My process is what I have been able to learn from board game designers and figure out on my own
Concept
Each game has a unique spawn point, lately I have been fairly existential, I only want to make a game exist if it's going to do something other games don't yet.
Goals
I quickly write out my goals, sometimes this is in the form of ad copy. If I can get the pitch down then all I need to do is deliver on that pitch.
Notes
I write notes and outlines of how the game will play, nothing at this time is what I would consider "final text", it's mostly to remind myself of the rules at the table. these notes are developed and researched, math and other things found out.
Initial build
This is when I make everything required to play the game, for storygames or rpgs this would be a character sheet a rules outline and whatever scraps of setting i think will help. for other games it's decks of cards, boards and bits.
let me again stress how important it is that nothing at this point is considered "final text".
Playtest
I play the initial build with my group, I know them very well, I understand their tastes and styles and know how to frame their feedback and make it useful.
I do this because it's almost impossible to recreate the brain space and atmosphere of an actual game. This stress of play always finds things that can be adjusted.
Development and playtest cycle
Any changes I feel necessary are made to the outline and components. At this point I play test again. I do this enough until I get what I want from the game.
Initial draft
Few games ever make it here. Once the game is where I want it, I start writing final stuff. Expand my outline and write rules into a book and generally prepare the game for consumption as well as I can.
product testing
I hand the game to a few people who have yet to play it and see what they think. Their feedback still must be filtered but these few people will still catch the most glaring errors in the presentation. I revise a little at this point to make sure the presentation is effective. When i'm happy i release the game.
——————
The process I have seen many story game and rpg designers use is for lack of a better word total bullshit. I know this because it was the process I used when I started. Process outlined as such.
Make a game
write your first draft of the rules, with diagrams. Make all the components and everything necessary to play.
Play game & change stuff
Crap! it sucks! change stuff. repeat until the rules document is completely incomprehensible because your constant edits have trashed it.
for some people this step is optional.
release the game
At this point you have the choice to internalize or externalize your failure. It's either "they don't get it" or "your terrible and should stop making games". The truth is neither.
GAMES ARE HARD! no harder then you thinking, they are REALLY HARD! I make games because WATER COLOR was too easy. Failure is required, a necessary step. The key is what you do with it.
13. On 2011-02-17, kleenestar said:
Wait a second ... I have to take a step back here. Do people play-test without being physically present to observe, or (worst-case scenario) video-taping the session? Seriously?
14. On 2011-02-17, Weeks said:
Ben, have you used playtest both right and wrong?
Also, could you write about what you got out of the Polaris playtests?
15. On 2011-02-17, Ben Lehman said:
Hey, Bret: IOU one post about infinite playtest for social cred. I'm trying to think of a way to write it that won't be highly devastating to some of my personal relationships.
Weeks: Most of the examples from my own design are positive: I came into RPG design with enough background in fiction writing and poetry to understand how revision processes should and shouldn't work.
Yes, I should write about Polaris in my copious free time.
16. On 2011-02-17, Lugh said:
Wow. As someone who has been a playtester for quite a few games, I can just say that clearly your problem is simply that you're doing it wrong. Or, at least, the "you" that the OP is addressing his rant towards is.
First, as a playtester, I don't *want* to receive a finished, polished, completely analyzed version of the game. I've gotten that a couple times, and then my comments were ignored because it was too late in the development cycle to change things. At that point, you don't want playtesting, you want approval.
If you're not getting proper playtest reports with notes written during the game, you need to train your playtesters better. The playtest process should never rely on comments written from memory after the game is done.
Playtest *is* important for textual errors and rules holes. Because every writer/designer will have blind spots. This is the same rule as never editing your own work. You won't see what you either forgot to mention the first time around, or that you assumed was obvious.
As for using your playtesters to revise your game, I *think* you have a point there. But, it's terribly muddled. It is a bit clearer in your "When to Playtest" section. Use playtesting to *REVEAL* problem areas. Then, as designer, fix those problem areas. You seem to have experience with designers, what, pasting notes from the playtest report directly into the text as new rules? I think that's pretty clearly "doing it wrong."
Your point seems to be that we poor design monkeys have put our eyes out with the drill far too often to be trusted with it anymore. Some of us still have a pretty good idea of how to use it, though.
17. On 2011-02-17, Simon Rogers said:
This could be paraphrased as "make every reasonable effort to get your game right before you begin the essential process of playtesting."
18. On 2011-02-17, stefoid said:
I think the value of playtesting is inversely proportional to the RPG experience of the designer.
Im a good computer/boardgame designer and I write great mechanics and have nifty ideas, but my couple of RPG attempts have sucked on ice because my experience with what different groups of people do at the table and why, is limited. I got immense value out of a very early playtest that was a complete disaster. An experienced RPG designer wouldnt have made all the blunders and uncritical assumptions I made in the first place, so they wouldnt have got that value.
So if youre a newbie at RPG design, like me, dont heed Ben's advice—playtest the shit out of that thing ASAP, it'll be a real eye opener. (dont use a drill)
19. On 2011-02-17, Gabriel Verdon said:
[quote]This could be paraphrased as "make every reasonable effort to get your game right before you begin the essential process of playtesting."[/quote]
This.
20. On 2011-02-17, Gabriel Verdon said:
To elaborate, games are emergent systems. No matter how much of a genius you are, it is impossible to imagine what will emerge from the rules systems you create. This is why in videogame development you create prototypes and vertical slices.
Without playtesting you are just stumbling around in the dark.
21. On 2011-02-18, Luke said:
Benfolds,
I hire three level editors for each project: a content/game editor (thor), a copy editor (Rich Forest) and a proof reader (Johanna). I then take volunteer editors from 6-12 people for each project.
And I can confidently say that editors do not catch everything. There are aspects of game play and interactions that cannot be caught from reading. Playtesting—or usability testing in software development parlance—is an important part of the process of design for many reasons.
22. On 2011-02-18, David Berg said:
Ben,
I'm digging some of your "how to do it right" advice, but the Rules Problems part is tough for me to get a handle on. Could you share any examples (e.g. images of your own work or links to others') of:
1) evaluate every possible rules interaction, looking for holes. Use a pencil and paper
2) short cuts: "okay, so let's look at all abilities that work like this: can any of them cause unexpected interactions?"
3) run through most of the common of the interactions, confirm that they basically work
4) really get into your game, understanding its internal logic and appropriateness, which lets you review much faster, and does involve a fair amount of play (even solitaire, see below.)
5) include a cybernetic control system with the human players (accelerate or decelerate the mathematically unstable parts of the system)
Are you talking about something like this (Delve) or this (Poison'd)?
Thanks!
23. On 2011-02-18, Moreno R. said:
Wait... I thought that the problem these days was FINDING playtesters...
24. On 2011-02-18, Marshall B said:
from kleenestar:
Wait a second ... I have to take a step back here. Do people play-test without being physically present to observe, or (worst-case scenario) video-taping the session? Seriously?
I do it, and have found it useful. The trick is knowing what feedback to ignore from it.
Ben,
I think I agree with you. At least, while I was reading, I was thinking, "Well, yeah, what else would you do" in my head. However, all the things you say NOT to playtest for—I've gotten those out of playtesting. I don't look for them, but you can get them.
But it's quite possible I'm misunderstanding. Here's what I do:
1. Design in my head a model of a game: what the game is supposed to do, and how I want it to do that, in a holistic manner including all subsystems and their interactions. This step takes days, weeks, or months, depending.
2. Once I can visualize the game all at once as a functioning unit, write an alpha draft (I call them "scratch rules"). This usually takes a day or two.
3. Read the scratch rules over and over. Think about them. Don't think about them at all for a while, get plenty of sleep, go to work. Think about them again, and make changes. Repeat until changes stop happening. This takes weeks to months to years.
4. Playtest, mostly to make sure that when actual people sit down at an actual table to actually play the game, it runs the way I imagine it—things flow like they should, subsystems interact properly (e.g. from The Rustbelt: does Psyche raise questions re: Pushing, does the Price raise questions re: giving, does char-and-sitch-gen result in fit characters and fit adversity).
5. Think about playtests. Don't think about playtests for a while, get sleep, go to work. Make changes to text. Playtest again. Repeat a couple-a few times.
6. Get people to playtest from outside, knowing that I will ignore 90% of what they tell me as irrelevant or due to goofs on my part (esp. re: documentation), but being very excited when that 10% happens.
7. Write a new draft. Think/don't-think/rewrite for a while.
8. When changes stop happening (or at least ones that I consider at all significant), publish.
I've only published one major game, but that's only because I haven't got off my ass to do #7 for about three games.
Does that sound about right to you?
25. On 2011-02-18, valamir said:
I had a very wierd experience reading this.
At first I was all "playtesting doesn't work"? That's hooey
And then I was all, "yeah that's a lot of good points Ben's making"
But from there I was all "but making the leap from 'these things are important' to 'and you can't get them from playtesting' seems pretty unsubstantiated.
So my conclusion:
Every place where Ben is saying "do these things they're an important part of design"...bold that...underline it...and do it.
And every place where he says "don't try to use playtesting for this"...scratch it out...erase it...and ignore it.
Replace it with..."when you use playtesting for this, it should only be after you've done these other things* and you should by this point have some specific ideas of what you're looking for in the playtest as a result of having done those other things...cuz otherwise...you're playtesting wrong and wasting your time...and THAT'S when you can mangle your design".
So the premise of this article I think is completely wrong. It shouldn't be "stop playtesting because you're doing it wrong". It should be "keep playtesting, but here are the things you need to have done first in order to do it right."
Rewrite it like that, and it gets two thumbs up for me. Leave it as is and its just another internet design meme waiting to be passed around and misused.
*not referring to the practice of Playstorming here, which IMO is really its own seperate animal.
26. On 2011-02-18, anon. said:
POSTING ABOUT GAME DESIGN: STOP
27. On 2011-02-18, Daniel Solis said:
Rob Donoghue tweeted this quote, others requested it as a t-shirt. Figured some folks in this thread would be keen on it, too. Available in a variety of colors!
28. On 2011-02-18, Evan Torner said:
I do believe that Ben's actually ranting here against playstorming, but put in the following defense mechanism...
("But what about playstorming!?" Go ahead. Ask me that. I dare you.)
...so as to denigrate that cultural practice entirely, while then ratcheting up the hyperbole on the practice of playtesting that, at best and at worst achieves mixed results.
But you cannot title a post - Playtesting: It's Kind of a Mixed Bag, Really - and farm a lot of high-profile attention from the indie game community, as this one does. And playstorming as a process is even more unstable, sorta like Calvinball.
That being said, I am one of those meticulous composers of AP reports on behalf of the gaming community, in part, because I've perceived an increasing demand among all sorts of RPG communities to see their practices documented. What Ben states about the craft of RPG writing/design is clear and sound: do it well, and filter out the voices that seem confused or have their own agenda vis-?-vis your product. But AP, like it or not, has become an integral part of the publishing constellation as well, and playtesting forms a cornerstone in that arc: what does this game look like when it's played, and how do those people who aren't the designer receive it?
Otherwise we get into 19th Century Romantic notions of the independent genius artist and their craft, etc.
29. On 2011-02-18, Ben Lehman said:
Hi Bret!
I figured out how to talk about the endless playtest for social cred cycle. Yay.
(As a warning to folks, I'm going to do a thing here where I talk about social interactions and social groups in rationalist terms: reward systems, strategies, and the like. Some people get very upset when I do this. Are you one of those people? I'm going to rely on your maturity and tell you to stop reading now: what I'm going to write next is just going to make you feel uncomfortable.)
So I think we can all agree that in the story-games.com extended social circle (although this dates back to the Forge and extends out to RPGnet) there's a fairly large amount of social cred and power attributed to people who are game designers. Essentially, there's a large social reward for "being a game designer" particularly an experienced one and also particularly one who hasn't become a local "bad guy" the way that Joshua became re: Shock:. Because of this there are a lot of people who want to be game designers because they lust after the status that it will provide them.
There are two problems with this. First, finishing a game, particularly finishing your first game, is really fucking hard. Like, way harder than most people will do strictly or even mostly for social status. Secondly, someone might say mean things about your game, and if a dogpile starts you go from high-status to pariah almost instantly.
There is a strategy which overcomes both these problems: putatively developing game(s), but keeping them infinitely in playtest. (This is far from the only viable strategy in this environment—I can talk about some of the others but they're really beyond the bounds of this post.) Give your game some cool evocative bits, some weird-ass system bits, and a nice title (the title is key). Talk it up, a lot, about how long you're playtesting it for. Make sure people post their playtest reports publicly. Attempt to use playtesting to substitute for any of the really important elements above (including marketing, creative and technical inspiration, and game development.)
This neatly solves the two issues with being a game designer, socially. First of all, you never have to go through the actual, grinding work of finishing a game. Yes, your game will never have an audience larger than 50, but as long as it's the right 50 forum-goers, you don't care. Secondly, as long as your game is "in playtest" no one can say anything bad about it: all they can do is give feedback, which must be phrased constructively. "Well, it's still in playtest" is an iron shield against the critique and dogpiling which could cost you status.
All you have to do is change games from time to time or make sure to migrate your game to the newest system trend (it's narrativist! now it's d20! now it's jeepform!) so you're never yesterday's news.
And so we have games in continuous playtest for years and years and years, and also a socially toxic environment where new, actually inspired designers are looking up to and taking advice from ignoramuses.
30. On 2011-02-18, Ben Lehman said:
31. On 2011-02-18, Jesse Burneko said:
I keep hearing about this game-design as social status thing. Who is doing this? Why am I so blind to it?
Am *I* doing it?
I have a crap ton of "Alpha" games on my website. You know why they're not done? Two reasons.
1) I lack the social courage/leadership skills to organize a consistent period of play-testing.
2) I lack the discipline to do all the work that isn't the design part (like writing, layout, art co-ordination, etc).
I own those failures. I hope to get over them some day. But I make no excuses (beyond owning up to my failures).
So why are they there at all? Because I needed to communicate the ideas in them in SOME way. I'm not looking for status or attention. I *might* be looking to see if the ideas call to someone else but that's a quest for like minded creative companionship.
Who are these status seekers?
And Joshua A.C. Newman became a "bad guy"? Did I miss a memo? I thought I kept on top of Story Games/Forge culture but I keep reading posts like this and thinking I must be a blind man in a dark cave.
Jesse
32. On 2011-02-18, anon. said:
GAME DESIGN: STOP
33. On 2011-02-18, Vincent said:
Hey anon? If you have a point, please make it instead.
34. On 2011-02-18, Will said:
Ben, that's a hell of an observation re: social status and the infinite playtest. Do you think folks do that by design or by falling into it? I'll be sure not to do too much playtesting of any future RPGs, lest I seem from the outside to be status-fishing, I guess.
Let me ask you, why did you choose the voice and approach you chose for your playtesting article? The antagonism here is a didactic choice? Which design culture says playtesting is that "to which all other elements of design must bow down[?]" If a risk of playtesting is that it may cause new designers to feel inferior... why is it the right approach to cast your article in a way that suggests the reader, as another amateur game designer, is inept and inferior?
I personally think playtesting is an area where nuance and the ability to take in and critically digest outside voices is valuable?that it is not a "terrible tool" even if can be misused.
35. On 2011-02-18, johnzo said:
re: eternal playtest as social farming.
Ben, I'm feeling like I'm part of this problem you're describing, because I award my attaboys to the designers who make fun games, and whether those games are "finished" or not is mostly irrelevant to me.
So, for the sake of the scene, should I refrain from praising long-beta darlings like Danger Patrol and Geiger Counter?
36. On 2011-02-18, Will said:
I should say, these are serious questions. I take this subject seriously and I take the way we talk about it seriously.
37. On 2011-02-18, Evan Torner said:
A ha! Your Comment 29 appears to be the real protein lurking behind the initial essay.
And I can say I agree 100%.
38. On 2011-02-18, Ben Lehman said:
Johnzo, Jesse, Will:
I am not your conscience. You have to make your own decisions re: your own actions. I will say that I don't think that there's particular malice aforethought in any of this: people respond naturally to reward systems by optimizing them, and I don't care to cast moral aspersions on this process.
By pointing out that it's going on, I hope to give people the tools to recognize and critique the infinite playtesting loop. I'm not saying "you yourself need to do X or you're a bad person."
Will: read this.
yrs—
—Ben
39. On 2011-02-18, Mendez said:
Do you think the cycle of endless playtesting for social cred is deeply intertwined with the ills of playtesting for the wrong design reasons which you cite in your original post, Ben? At a glance I would surmise that those are two vastly distinct issues but I might be misreading you. It seems that, although I have never noticed it in my limited sojourns through the community, you have discovered an epidemic of harmful playtesting; but is this epidemic in fact two epidemics springing from different sources (social anxiety and misguided design)?
40. On 2011-02-18, Will said:
Ben, I'm sorry if I was being distracting or tangential with those questions. On a post advising people about the importance of skilled writing, the value of editing, didacticism, and the role of critique, I honestly thought them relevant. I still do. I think how you say something makes a difference.
For what it's worth, I agree that playtesting is not magic, that it can be handled skillfully or not, and that it is not something "to which all other elements of design must bow down," for a designer must be sturdy enough and sure enough to maintain and protect her vision in the face of cloudy feedback and suggestions that threaten to drag focus away from important goals. That's a valuable message. I'd argue about specifics?"textual playtesting" does not require surrendering to a crowdsourced editing job, to my mind?but whatever.
I'd have liked this a lot more and understood it better if it was restructured.
41. On 2011-02-18, Ben Lehman said:
Will: We all make decisions with our communication strategies. It's far more important for me to communicate with Erik (and others) than it is for me to not offend you, for any value of you.
Also, note that this essay (right there in the first paragraph) is not addressed to you, Will, in any way. If I were to write a parallel essay addressing workers at 2nd tier RPG companies such as Atlas or White Wolf, I would title it "Playtesting: for the love of God consider doing some of it for once in your damn life."
yrs—
—Ben
42. On 2011-02-18, Will said:
Fair enough. I'm not offended. Sorry I read something intended only for other people. Don't know how I did that on the Internet.
That second-tier playtesting joke's funny.
43. On 2011-02-19, Will said:
Seriously, I'm sorry I took focus off your message. Go forth and conquer.
44. On 2011-02-19, David Berg said:
Ralph, thank you. I was thinking that, but couldn't figure out how to say it that well.
45. On 2011-02-19, Joel said:
This post is chock full of fascinating stuff for me, that hits the core of my experience and angst with RPG self-publishing. I'll note for the record that the confrontational, even accusatory, tone is difficult for me to engage with, Ben, and I think I AM one of the people you want to communicate with...? Like, countering harmful moralizing with equally fierce counter-moralizing isn't too productive, maybe? Still and all, I AM going to attempt to engage with you because these issues are important for me and I want to learn. If that validates your decision to use that tone, then so be it!
I'm going to focus on telling my own story about my experiences with the world of Indie design and publishing. I entered the arena of game design in Game Chef 2008, and produced a game that I liked, and wanted to continue developing. I worked it up into an extremely ugly ashcan for the Portland area's Gamestorm convention the following March. I had exactly one playtest beforehand. I ran one game of it at the convention. I played a several-session game with friends after the convention, plus a semi-drunken one-off game with coworkers. Some folks bought the game at the con, ran it, and emailed me feedback. I ran it at Go Play NW that year ('09). One other person bought the game, played it with their group, and gave me feedback. I ran it for Gamestorm '10. I haven't played it since.
That game went on the backburner when I got a new bolt of inspiration for a game, one dearer to my heart, in fall '09. I struggled with implementation for several months, considering and discarding several procedural ideas. I had nothing playable. Then in March '10, as Gamestorm was once again rolling around, I had a eureka moment where one key element presented itself and everything fell into place. I worked furiously to write down the resulting game, which had been rattling around in my brain for months. I playtested it the week before the con, and arrived at the 'Storm with a new (also pretty ugly) ashcan. I didn't have any scheduled games for it, since I didn't know it would be finished, but it demoed well, and some friends organized a spontaneous game of it with no prompting from me, which I then got to watch.
I played the game with friends several times over that spring and summer, and ran two games at GPNW. I got myself a copy of InDesign, and bugged friends to give me a crash course in layout, and produced a very pretty (if I do say so) handmade book to take to PAX Prime in August. I ran some games there, and it sold well. I've been selling a steady trickle ever since, an d playing a game here and there. I consider the game complete. If I want to refine it, it won't be in the form of revision or rewriting, but rather low-cost companion booklets that expand play options, recontextualize the procedures, and explore the game's themes. Similar to the Sorcerer supplements.
meanwhile, my other game is due for a heavy revision (and, given newly acquired skills, massive prettification) before it can move forward. I've learned a lot since its inception, both through playing it and my other experiences other play and design. I aim to get under the hood again soon.
++++++++++++++++++++++++++
So, why the extended anecdote? Because my story IS my argument. I don't see any profit in my going "Hey, Ben's right!" or "Aagh, Ben's wrong!" Instead I'll look at how my personal story matches up with his claims.
First, when I began designing, there was definitely a Playtesting Shame Culture circulating around this corner of the internet (for a certain value of "this corner"). There was a lot of polemic about the moral responsibility of the designer to playtest heavily, about making sure you had "blind" playtests, about picking the customer's pocket, etc. I haven't heard a ton of those polemics lately, but my gut says it's down to a dull roar, but the Shame Culture still exists. It had a huge impact on my. Not that it impacting my design practice per se; it just helped me internalize a load of guilt about not being able to "properly" playtest, and fear that I'd be found out and decried on Story-Games or the blogoblags or wherever. I've had that fear right up until...well, up until yesterday.
When I read Ben's article.
I felt such relief to hear a I have been terrified that if anyone in the Indie crowd knew how "under"-playtested my games were I'd be shunned, called out, condemned. That may have been melodramatic on my part, but I don't think the fear is unfounded. Now I feel it's important enough to take the social risk and lay my design journey bare.
The fact is that for both games, I playtested as much as I was able in light of my own time and energy and the pool of willing participants. If I could have played the ashcan versions twice as much, THREE times as much, before writing a revision, I would, but not so I could "spot the holes" or test the mechanics. More so I could be comfortable and knowledgeable with how play of the game feels, so that my instincts for running it and playing it are finely honed, and I can reflect that in the text. Because yeah, game-mechanical interactions CAN'T be tested in a sufficiently large statistical sample. It's like saying "I rolled percentile dice three times; the most likely outcomes are 17, 52 and 86." But getting a game to fit you like a second skin, so you can trust in the emotional experience of play—that you can get through playtesting. I'm fairly happy with where I am with that on my finished game, but more is always nice.
It helps that my games, especially the completed one, are relatively simple mechanically. Still, my unfinished project, to be sure, has some kinks to work out in the numbers department. But that's definitely a work of tinkering in solitude. I'm not going to tweak the numbers, play it with friends, tweak 'em again, play again, and so forth. That really WOULD take ten years. The more talky rules, though, they definitely bear testing with people. The "OK, so after you roll that, person A says this, and includes a bit about B, and..." stuff needs testing, not because of "balance" or anything, but to see how it feels coming out of people's mouths. Again, it's about getting comfortable with the game's emotional core.
++++++++++++++++++++++++++
So there's where I stand with respect to all that. Let's see, what else? Oh yeah, trading on social status. Totally true! That's not a dirty thing, just a thing. So I met Ben at Gamestorm '08, played his Bliss Stage demo, totally wanted to know him and be cool in his book. Before that, I found Anyway, then the Forge, totally wanted to know Vincent and Ron and be cool in their books. And so on. At first this was just a matter of playing cool games and finding smart ways of talking about them online. Later, with the impetus of Game Chef,it also became a matter of designing good games and getting them played and talked about. So far, they haven't been talked about much, but they've been purchased some, and played some, and that's something. In any case I totally want to be thought smart and cool by a bunch of internet people (including you folks!) on account of my game design, and that's just fine.
That sort of thing can turn ugly, yes. Status is one of those things where you can't do it JUST to do it; you need an authentic statement to make, or some sort of personal integrity, or it IS kind of hollow and unsavory. And a delicate touch is needed; more delicate than I'm being here, I suppose. I deliberately haven't mentioned the titles of the games I designed, because that sort of status-fishing feels too overt for this discussion. But yes, I'll readily admit that I seek status with the self-publishing crowd...why wouldn't I? I hope that I'm making authentic statements in the process.
I will admit that I hadn't picked up on the strategy of milking playtesting for status; I was more concerned with getting finished and felt my status was fleeting until I had a solid game. Maybe I'm naive or something. But I can totally see that pattern. Especially now that I do have one finished game, it's easy and tempting to be like, "oh, this game I've got in development blah blah blah..." when maybe "in development" means a paragraph scribbled on a scratch pad a year ago.
OK I've been typing for a long damn time so I'm just going to stop. I hope I'm coherent here. I didn't know how else to approach the topic; I'm still trying hard to process all this info flowing at the speed of Internet.
Peace,
-Joel
46. On 2011-02-19, Ben Lehman said:
Thank you, Joel. That's pretty great. I mean, except the shame part.
I am really glad that you pointed out that there are totally healthy and good expressions of social status.
47. On 2011-02-19, stefoid said:
On the flipside, status will get you playtests
48. On 2011-02-19, jenskot said:
"Now that we know the bad way... what's the good way to playtest?" was asked on Story-games in reaction to this thread. My response...
My playtesting (Usability Testing) experience is with interface design (websites, applications) and information design (instruction booklets, posters) but this is how I would playtest RPGs...
Answer these questions...
1. What is your game? What problem does it solve? What makes it unique?
2. Who is your game for?
Without the above, you don't know what you're playtesting. You don't know who should playtest your game. Additionally, you won't know which playtest information to take seriously and what to ignore. And finally, you won't know if you even need to playtest.
Why are these answers important?
- If you don't know what your game does and doesn't do, how do you know what feedback to listen to? You can't succeed, if you don't have goals. You won't know what direction to walk unless you know where you're headed. You won't know when to stop unless you know what your final destination looks like.
- If your game is just for your personal gaming group, why bother with outside playtesting?
- If your game is just for you, why does it matter what anyone else thinks?
- If strategy and tactics are important to your game, why send a playtest copy to freeform gamers?
- If you don't know what problem your games solves, how do you know what rules to write?
You can't please everyone!
Ultimately, no matter what you do, thousands maybe millions of people will disagree with you. You can't please everyone. Fortunately, if you know who you are trying to please... you don't have to care what everyone thinks. If you reached 1% of people currently playing D&D... that's 60,000 people. A lot of design, not just game design, is about focus and eliminating anything that distracts from that focus. With focus, you know who and who not to listen to.
Once you identify what problem your game solves, you can nail down whose problem you're solving. This is your audience. This is who you want playtesting your game.
I can't find playtesters!
If you can't find anyone to playtest your game, that might be a warning sign to go back and revise what problem your game solves and for who. If people don't want to playtest, maybe it's because your game doesn't actually solve a problem they have. Or you've misidentified your audience and are talking to the wrong people. Or your audience is very busy and needs fair compensation to make playtesting worth their time. If you can't compensate them, maybe the project is too small to warrant all the time, energy, and resources dedicated to it as a commercial project and would be better off released non-commercially.
Every roadblock you hit is a sign that you may need to rethink your strategy. It's a gift. It will force you to step back, sharpen your focus, and ultimately learn from your experience to make you even stronger.
Even if you find playtesters, be realistic about your expectations...
Sometimes people volunteer to playtest and never come through. If 1 person out of 3 who volunteer actually deliver... I would consider that a success! If you need 6 playtesters, find 18.
Create your playtest...
Decide what you need playtested. Think small. Start with the absolute bare minimum rules your game needs to achieve its goals. Identify the top 1-3 things you need tested. Ignore everything else. Create scenarios that will allow playtesters to focus and test these top priorities.
Design your scenarios so what you're testing isn't obvious to the playtesters. Your scenario might be, "create a character" but what you're specifically testing is "how long does it take", "is stat allocation frustrating", "does character creation give the GM enough information to design an adventure." Don't tell your playtesters what you are actually playtesting.
Take caution that your scenario doesn't influence your playtesters actions. Don't ask leading questions or make leading statements. If you want to test "how long does this take", in your scenario, don't say "character creation is super fast". Don't influence!
Run through your scenario before you give it to playtesters. Make sure it's clear and error free, otherwise you tests may fail for reasons unrelated to your game. Check your grammar, make sure any jargon is clearly defined, give clarifying examples, and cut out anything that isn't related to your test that could distract from your goals.
Running playtests...
Make sure your audience is uninitiated. Don't test the same scenarios with the same people. Ideally, don't run multiple tests with the same people at all. And run each test with 3 separate groups of testers.
Record all your tests. It may not be possible for you to be present for all your tests. And even if you are, in the moment you may miss something.
Ideally your scenarios are prepared so playtesters can run through them without your involvement. Your goal is to observe and not interfere. If your rules are in the pre-alpha state (more ideas than solid procedures), feel free to guide the playtesters through your scenarios but recruit a third party to observe the test or absolutely make sure to record the test so you can make observations later. Don't multitask.
Train your playtesters to think out loud. If you're recording audio, you'll miss all their revealing facial expressions. Talk to them, make them feel comfortable, run through practice scenarios first (not your actual real scenarios) so people get used to thinking out loud. If people are quiet, ask them what they are thinking. If people talk out loud, nod along and give them encouragement. Reassure them that the playtest is a safe space, they can say anything, just be honest. Make sure they know that they aren't being judged. I know this will sound funny... but it's probably best not to tell them you're the game designer! That alone will bias the test.
If you're taking notes while people playtest, do so in a way that people can't see you.
Don't ask playtesters for their opinions. Especially after the test is over. People are unreliable self-reporters. Plus their job is to playtest, not observe themselves playtesting or trying to fix your problems. Your goal is to make observations by studying what people do, not what they say they did. In playtesting, actions speak louder than words!
Creating reports, identifying problems, solutions, and more tests!
Review your recordings and notes to create a report identifying all the problems (even if you don't think they are actual problems). Take special note of any problems or unusual behavior that happen more than once.
If you can, get a second person to create a report by reviewing the recordings and identifying the problems independently of your observations. Compare the two reports.
Compare these problems to your goals.
Decide which problems actually need to be solved.
Fix these problems.
Create new scenarios to confirm your fixes work.
Run additional playtests with new people.
Playtesting your text is more important than playtesting your rules!
When someone buys your game... all your rules become guidelines to be followed or discarded at the whim of your audience!
And even if they decide to follow your guidelines, the rules don't matter unless they understand what the rules are, how to use them, why they are important, when to use them, what will happen if you don't use them... and then actually remember all this when it is time to play!
This is where your game text comes in and why playtesting text is more important than playtesting rules. Your rules have no power at anyone's gaming table. You must persuade players to use them (this is true even in boardgames). Your text is your only tool of persuasion. Use it wisely!
Your text needs to do more than teach. Your text needs to turn its advocates into teachers to spread its message. Your text has to arm its readers with the tools to teach other people who will never bother to read your text! This is where handouts, character sheets, and reference sheets all come in. They not only help people who haven't read the rules to learn... but also remind everyone what the rules are, when, and how to use them.
I don't know how to write my rules...
But you probably know how to verbally teach others how to play them. So do that and record yourself. Then listen to the recording and create an outline. What order did you explain things? What questions did people ask? What rules needed extra explanation? What examples really helped people understand how to play? What confused people?
And then do it again but this time, as scary as it might sound, video tape yourself doing it! Why? Because when you're teaching people how to play, you're probably using body language and visual cues (pointing to specific rule pages, charts, character sheets, rolling dice) to help people understand what you're talking about. And your text can do the same thing with the help of images.
And hire an editor!
Editing isn't just fixing grammar and spelling mistakes!
Editing is also about content. What's the best way to express an idea? What order is the clearest and most effective way to express your ideas? Which ideas need to be reinforced with examples? Which ideas aren't important and need to be cut out?
Editing is not enough!
You need to playtest your editing just like you would playtest any rule changes. Grab someone and watch them read your rules out loud. If its' painful, don't turn away... find out why it hurts and fix it.
If I do all this, will my game sell?
Does a contract protect your rights? Not absolutely. My friend Chris who's an IP lawyer once described contracts as a +2 circumstance bonus in a d20 law roll. Nothing is guaranteed but just like in D&D when you're facing a red dragon, you're going to want every bonus you can get! You might attack unprepared and get lucky rolling a natural 20! Or watch in despair when that die reveals that cruel critical miss... a natural 1! But the game isn't about just 1 roll. It's about trying again and again. Learning from mistakes. And arming yourself as best you can.
I hope that helps!
It all sounds harder than it actually is. And even if you can only do 25% of the above... you're ahead of most game designers! Hell, if you just clearly know what problem your game solves and who your audience is... that alone puts you way ahead. Focus on doing what you can. Be honest with yourself. Remember that getting started is the hardest part. And doing the first 10% is the first step in doing the next 10%.
Rock,
John
49. On 2011-02-19, anon. said:
GAMING: STOP
50. On 2011-02-19, Josh W said:
Ben, maybe we need to create a "social version" of a game that we release to friends/designers we respect, and what we ask of in return is for them to write about their play.
This is not a product, it is not something with any expectation of merchantability or consumer entitlement, it is a gift with a request attached. "I made this, and I wonder if it might be the sort of thing you like, if you wish to accept this, accept it as the invitation to a conversation"
So why do this hippy thing? Because you want people to interact with your stuff, talk to you about it, get interested in your stuff and help you explore your ideas, but you want at some point to say "Ok people, now I'm a product-maker, and here is a thing worth paying for".
The ashcan price seems like a way to remind people people that this thing deserves money at some point, and to form a token gesture of respect for you as a creator. And the fact that it is a physically different document means that people buying the later properly priced one don't feel short-changed. Playtesting also forms this sort of, "it's not free stuff, it's valuable stuff I'm letting you have for free" vibe.
So say you have a version where you say;
"I want to hear about just about every game you play using this system, even if it's crap, post it and send me a link or just email it direct".
That's the price.
But it's not meant to be about the quantity, like doing lines, but the commitment to a designer-player relationship. To playing it intensively and thoughtfully. Plus you don't just give it to anyone, you either let people request a copy or suggest a copy to them. Your not distributing, your giving a personal gift.
You can honestly say your happy with your game, but you want to let it brew up a community before you release it as a game. If the insights cause you to change your game, then it's sort of like playtesting, but the point is that people feel free to release games for the cred and the conversation, and then differently, for money.
I think this is a good thing, allowing the personal and the profesional to coexist.
On the other hand, I've got little time for refuge in mediocrity, where people don't really try so as to keep their potential seeming magical. I say shoot your cannon balls, then reload with better ones.
51. On 2011-02-19, Judd said:
Ben,
I'm not sure that was being fair to Will. He's a gamer and thereby, could publish an indie RPG if he wanted to.
I've done some freelancing, does that make this article not for me or am I indie enough for it because I published Dictionary of Mu?
I dunno, I call shenanigans on that one.
With love,
Judd
52. On 2011-02-19, Ben Lehman said:
Judd: I have a target audience in this post. I point out what my target audience is in the first paragraph. I am talking about real problems faced by people in that group, not all the hypothetical people who might hypothetically someday maybe be part of that group. Given the response from people in that group, I think I hit my mark, more or less.
Will is in a different group. There's no condemnation there: I'm not saying that anyone who is a developer for White Wolf and Atlas is wicked or whatever, just that they face a very different set of issues than amateurs. It doesn't mean he can't comment on the essay. It means that commenting as if I were talking about him, specifically, when I am actually not talking about him, specifically, is disingenuous in the extreme.
yrs—
—Ben
53. On 2011-02-19, Ben Lehman said:
Josh: More barriers to publication is not a solution.
John: I'm not sure the Story Games "let's take this and invert it" is the best route to discourse. Like, that's an essay that I have no idea how relates to mine at all. It could just be that I haven't had my coffee yet, though.
yrs—
—Ben
54. On 2011-02-19, David Berg said:
Joel,
Thank you so much for giving us a concrete example! Can I ask you to try to disentangle the shame thing?
How much was pressure to be sure that your game actually worked? (And status related to that.)
How much was pressure to make sure that a game you knew worked would actually be accepted as such? (And status related to that.)
How much was pressure of some other kind?
55. On 2011-02-19, Will Hindmarch said:
Two different groups, and never the twain shall meet? I paid money for your game, I don't think you're an amateur. I think there's a spectrum of experience, in some ways, rather than separate groups. Maybe I'm wrong.
You also say there's no condemnation, but your joke about second-tier playtesting says otherwise, doesn't it? I read something, too, into the fact that you are a "designer" and I was a "worker," but that's probably just me being me.
Anyway, I don't think I was being disingenuous, but if you got that impression I'll take responsibility for it since I'm the author of my comments. I'm sorry I came across that way. I thought I had things to learn from your post and comment, so I applied it to my experiences as an independent designer and tried to understand why you chose the language you chose. If that's verboten, fair enough. It's your post. I guess I'm welcome to comment as long as I understand that my opinion is irrelevant? I'm a little confused there.
Rereading your comments, Ben, I guess you think it was lowly of me not to include my last name? Maybe? I left it off because I thought it was more casual and, really, didn't give it much thought. Sorry if that seemed disingenuous.
Again, I apologize for distracting from the topic at hand. I came back here to lurk because I'm genuinely interested in this discussion. I'll stop commenting when you stop talking about me.
56. On 2011-02-19, Elizabeth said:
Here is what I think. Specifically only about the original post/manifesto.
I think Ben used a lot of grumpy words and profanity, and sure, whatever, be offended if that offends you, you come by your offense honestly. I wouldn't talk that way publicly on the Internet, although I know a lot of people who would. I married one! (Not Ben.) It has lead to much eye-rolling on my part when things are couched in hyperbole, but it's not like that makes it completely unreadable.
I also think it's incredibly useful information that helped me find my way around a serious block playtesting wouldn't have been able to help me with.
Thanks, Ben!
57. On 2011-02-20, Will Hindmarch said:
(Last thing: That last statement's not fair of me. I've said my bit and this is your house. Say whatever you want about me. I'll be quiet.)
58. On 2011-02-19, Josh W said:
I'm not sure I'd call a social version of a game a barrier. Ok, yes it's true that you are intentionally delaying publication, and have one extra thing to write before you get to something to want to sell, but this idea is for those people who are already delaying publishing to do it honestly.
In addition, every time I've got the shivers about making something for public consumption, it has helped to make a draft version for a friend or a small group first. I've litterally writen essays, marketing tracts, and important emails as if I was writing them for a friend, maybe shown it to them, maybe not, and then seeing an actual finished version has allowed me to take the remaining steps to making one for the "general public" (ie an unknown recipient). You can reach clarity within a familiar setting, then replace the crutches you used with more general touchstones. The same works with trying to explain your game to friends with words.
On the broader issue, I can feel myself gearing up to discuss the different values of publishing for money on an open market vs private gift economies, but I'll probably save that for another time. I will say that I understand the bravery involved in putting your stuff out there for anyone to own, and that creating/legitimising an intermediary state can allow people to cop-out of something they'd otherwise really value doing.
But by labelling that mostly-existing intermediary state distinctly from playtesting, we can have that discussion seperately, without it causing negative side effects and pressure on new designers.
59. On 2011-02-20, Vincent said:
Hey Will, you're fine. This is my house, not Ben's or anyone else's; like anyone else, you're free to comment or lurk here just as you like.
60. On 2011-02-21, Joel said:
David,
I'd say it was some pressure to make sure the game worked, sure. Mainly due to really shallow game design experience—I don't yet have very developed instincts for knowing if my surface assessment of "well, it SEEMS like it should work..." is accurate.
More than that though, it was pressure to have the game ACCEPTED as working. With my Game Chef design, I wasn't sure before playtesting that it would be playable or fun. But my latest, now completed design? Yeah, I was pretty secure in that knowledge. When I playtested, it was mainly (as I noted in my original comment) to know how the game would FEEL to play, and to get myself comfortable with the play of it. But I felt pressure that if it wasn't "sufficiently" playtested, I'd be judged as a fraud.
As for pressure of another kind, I'm not sure. I can't think of any other categories of pressure. I guess pressure to be a hip Story-Games-posting type person in general, one way of which was to have a game, and that game SURE AS FUCK BETTER BE THOROUGHLY PLAYTESTED AND "FULLY BAKED."
I do remember where it all started. It was a conversation on the Forge, where Ron said that Joshua's first edition of Shock was a fun game, but a mess of a game text. Soon thereafter there was a whole rash of talk on "half=baked" vs. "fully-baked" game design on blogs and forums, and the lesson that I absorbed was that I had better not go public with a game unless it had been tested to a degree that was A) beyond my means) and B) incommensurate with the needs of the game.
In retrospect "stay in playtesting indefinitely" does sound like a shrewd workaround. But in my simple and direct way, I assumed the only solution was to get the game DONE, and let on as little as possible about how much it had been "playtested."
I did include a list of playtesters in the text, though, partly in the interest of honesty and partly in the interest of gratitude. By the time I had typed them all out, I found I was pretty happy with the list. It didn't look scant or anything, rather just right for the game.
Peace,
-Joel
61. On 2011-02-21, David Berg said:
Gotcha. Thanks for elaborating.
Man. Do you really think folks on the internet might belittle you over that? And that others would listen? That sucks.
Although, I guess needing to tell that story over and over would be a pain in the ass...
Maybe you should just post on the game's site "I was pretty sure this was done before playtesting, playtesting confirmed I was right; it didn't take a million sessions, so sue me; I wouldn't charge for it if I wasn't confident in it." And then link haters to that post. :)
62. On 2011-02-22, Ron Edwards said:
Don't lay this shit at my door. My criticisms about the first release of shock: were explicitly aimed at the writing. The system itself was and is brilliant. Joel, you say this yourself. So how come my critique, which Joshua agreed with and to his credit rectified, is being held responsible for the foofaraw about *playtesting*? The responsibility for that lies with the retarded and toxic context of discussion at various popular internet venues for being indie game designers.
To any reader: don't bother asking me where or who. If the shoe fits, wear it.
Best, Ron
63. On 2011-02-22, Ben Lehman said:
Can everyone see how an error of writing (which Ron is right, is what was wrong with Shock:) becomes an lack of playtesting, if you're laboring under the delusion that playtesting = editing, described above?
Yeah...
yrs—
—Ben
64. On 2011-02-22, Ben Lehman said:
ack forgot to close my italics.
65. On 2011-02-22, Vincent said:
Got it.
66. On 2011-02-22, jenskot said:
Text also benefits from playtesting.
There are many mistakes editors miss without the context of play. Rules may be solid but if the text doesn't communicate those rules clearly, it doesn't matter.
67. On 2011-02-22, Ron Edwards said:
But not from stupid playtesting, or playtesting with status priorities.
I remember when I posted about my design and publishing process for It Was a Mutual Decision - a game, John, that you have played and which as I recall, you value. I was a little puzzled at the smug, nasty shit I received in that thread about how little playtesting the design work included. I responded with what seemed obvious to me: each playtest, and there were indeed playtests, solved problems and helped me write. When I judged that no more problems persisted and that I was able to explain the thing, I moved on. In the case of that game, it simply didn't take many playtests, case closed.
Later, I understood what the internet message of those posts was, to a certain kind of audience and certain kind of mind. "Look, look, Mr. playtest-and-bake man, *he* doesn't even playtest his games!" Snide, evil little fuckers.
But more importantly, stupid. The point of playtesting is to help make one's game actually playable and write-able, with these goals subject all the while to the author's own judgment, just as Ben says. If it takes three sessions, or thirty, makes no difference. Sometimes it can be accomplished with only author playtesting, and other times external as well. I find that external playtesting is more about the writing, as typically a group gets off on the wrong foot and suffers three hours of monstrous nonsense, their careful notes about it aren't worth squat, and I learn more about how to write the introductory material. But again, the point is that the utility and extent of whatever sort of playtesting are specific to that particular design project, and always subject to the designer's process, not overwhelming it. Missing that point in favor of God knows *what* interpretation of advice to playtest your game is simply stupid.
My take is that we're really talking about the subcultural betrayal of the Ashcan Front. Paul and Matt were onto something with that. I don't know how many people actually understood that their target audience was not the on-line indie-enthusiast crowd, but instead the masses, unwashed and otherwise, wandering around GenCon and the gaming landscape in general, with more-than-half-finished games riding in their backpacks. The project was an explicit attempt to restore the outreach qualities of the early Forge. And instead, it was hijacked by insiders much like the Game Chef was partly hijacked, as a venue for their ... whatever it is. The best of the first batch by far was The Path of Journeys, by Ram Hull, a complete outsider in the scene and exactly the kind of person the Ashcan Front was for. No one saw it. They were too busy circle-jerking about what big designers and playtesters they were, with each other's games.
It takes a tough mind, a thick hide, and a truly subterranean sense of humor to operate in that crowd and still produce good games. Willow can do it. Some others, but not many. Sensitivity, active listening, receptiveness are all liabilities in that subculture, drawing people who should know better into the orbits around false sources of feedback, all of which turn out to be black holes.
I've had it with the stupidity, and I'm calling it out. This subornment of playtesting is merely one example. It's deeply associated with *economic* stupidity as well, and social stupidity in which the champion victim players become lauded as heroes when they are merely parasites. It's time for whoever can, to own up to this wasteland of bad dialogue, bad design, and bad business, and to get the hell out of it. We're past the point at which there really isn't any more excuse.
Best (to some of you), Ron
68. On 2011-02-22, jenskot said:
I enjoy It Was a Mutual Decision quite a lot.
I can't speak objectively to the usability of the text because I'm biased (learned to play first from the designer) but I've had no problem running it. That said, when I try to get people to play, I find that I'm successful 1 out of 3 times and the reason people often don't want to play seems to have very little to do with the game itself (people new to RPGs are all for it). I suspect (can't confirm) the same may be true for threads attacking the game's playtesting. It's possibly less about playtesting and more about hidden agendas with playtesting as a convenient avenue to attack someone to progress identity politics. People may not be actually arguing what they are arguing about. Which can get toxic pretty fast.
69. On 2011-02-22, jenskot said:
Speaking of bias, I've managed professional usability testing (playtesting) for websites, applications, printed materials, instruction booklets, kiosks, and more. That combined with me not being an indie RPG designer, puts me outside of the target audience here. I've also playtested D&D 3E, 4E and several other RPGs but that still is outside the scope at hand. I think many of the techniques I've honed overlap with RPG development, but I don't want to misrepresent myself.
I agree with Ben that playtesting is not a replacement for having goals. You need to know what your game does and for who for playtesting to be of strong value. Playtesting is also not a replacement for editing.
But my concern is that "Editing" may become the new "Playtesting." Editing is great if you know how to use it and its limitations. Specifically, an edited work may need to be playtested after it's been edited to confirm the editing is still in line with your goals. In my fields, and in larger RPGs, I've found this to be solidly true, but everyone has different needs.
Take D&D 4E, after several editing passes, the rules behind Skill Challenges became more obfuscated and no longer communicated the designer's intents. It needed a last playtest after the final edit to confirm everything was still inline with the designer's goals.
Playtesting and editing are both great tools if you know why you are using them and how. How much playtesting and editing you need depends 100% on your goals.
70. On 2011-02-22, Bret said:
Well I hope it's not me!
71. On 2011-02-22, Bret said:
Oh and thank you for elaborating Ben.
72. On 2011-02-22, Ben Lehman said:
Hey Bret. You're welcome.
Hey, John. I really think that talking about this outside of its intended audience (with corporate-developed games, which, as noted above, have different issues with playtesting; or with other random stuff that involves user testing like websites or video games) is distracting from the point.
Yes, if you take the essay outside of its intended context, and use it in some sort of mushy way to talk about totally different topics, it makes less sense. It might even make no sense at all, or be patently wrong! This is true of any essay, but it seems mostly to serve as a distraction and evasion rather than any sort of substantive engagement or critique.
Does "editing" become the new "playtesting?" I frankly hope that my writing has that kind of sub-cultural reach. And, you know what, if it does actually fall out that way, maybe I'll be doing another guest-post on anyway three years hence entitled "editing: stop." I'm willing to take that risk.
yrs—
—Ben
73. On 2011-02-23, Joel said:
Ron, I certainly wasn't trying to say this was your fault or anything. I'm sorry I implied anything of the kind.
I think we agree, in fact. I wasn't saying that the Forge Shock thread was the source of my absorbed shame message...it was the "rash of talk" that came after, which you also seem to be referring to in the most disparaging of terms. So I think we're on the same page on that, yeah?
It occurs to me that this kind of message decay is endemic to the whole internet, not just Forge-spawned concepts. Like John worrying that "editing" will become the new "playtesting"...yeah, it's always going to be something, right? Which sucks, because people coming into the scene (like me in 05-08) are going to have to navigate a mess of posturing, distortion and rhetoric. Like the Ashcan Front—I got so fucking confused about what the Ashcan front was about and for: why you should ashcan your game, whether your game counts as an ashcan, etc. etc.
Not sure what to do about that...proclaim the truth loudly? I won't discount the power of the right truth heard by the right person at the right time; take me and this thread for example. But it seems like the larger effect is to create a ripple of reaction that leads to a brand new wave of posturing.
How about creating organized repositories of knowledge and practical instruction, outside the context of blog and forum posts with battle lines drawn? A guide to playtesting, to editing, to physical publishing, etc. that laid out clearly A) what a given process is for, and B) the basics of how to go about it, would be solid gold. I'm not saying it would have to all be in one place—though the Indie RPG Publisher's Database would be a beautiful thing to behold. But any given person with a passion for a thing could make a website for THAT THING, and it would be a valuable resource for those who need it.
Of course, this is no guarantee. After all, the Ashcan Front has a website that explains their whole philosophy and look how that turned out! Maybe the lesson is, if all your information on a topic is gleaned from forums and blogs, be wary. If there were more repositories like what I'm talking about, perhaps the distortion effect would lessen.
Peace,
-Joel
74. On 2011-02-23, Joel said:
David,
The basis of that feeling (that people would belittle me over lack of "full" playtesting) was instinctual; nobody was breathing down my neck about how "that thar game better be playtested, son...FULLY."
What I DID see was the occasional taking to task by some person against some person for rushing their game or not playtesting or charging too much money or what have you. I think at some point this became the "new new honesty...?"
I don't have links or examples, and I don't have experience of people subjecting ME to such scrutiny directly. I just remember seeing other shit go down and thinking, "man, I better watch my fucking back or that could be me!"
Hmm, actually I did have one experience that wasn't...overtly a call-out, but seemed to me to have a weird undertone. So I'd completed my first version of my game, and was all like "yay, my game!" on Story-games. Someone asked, "Hey, could you tell me about your game some?" And I was like "Sure! Be happy to! It's like this!"
And the person said "Oh wait, I get it now! This is actually [previous, rejected name of game]. Now I know why it seemed to come out of nowhere; it actually didn't."
I came away from that conversation thinking "hmmm...was that some sort of test? Was this person checking me for proper game development cred, to ascertain by the number of forum posts my game appears in whether I've been designing the game for sufficiently long? And I narrowly averted criticism by demonstrating that the game was previously in development [=talked about on forums] under a different name?"
I have no idea if that's what the poster was getting at. But that's what it felt like.
Peace,
-Joel
75. On 2011-02-23, David Berg said:
Joel, gotcha. Again, thanks for elaborating! This is helping me understand things.
76. On 2011-02-23, Suna said:
Very insightful thoughts, Ben. I come late, so it took some time to read most of it, and yet I had no chance to read all of the participants' contributions. But I see many solid points being made.
I find these thoughts useful, and very, very badly expressed. Which makes them comfusing in some points.
A first reading feels like you're addressing all people engaged in designing games. The metaphor of the power drill is in my opinion misleading, and you seem like you're addressing the emotional side of readers rather than the rational side, while your point is part of rational, practical and reasonable game designing processes vs. an unhealthy tendency of only a part of the so called game designers (I loathe labeling people who create games. It's as if they were different because of that: being a game designer doesn't make you any better, more important, or different than anyone else).
I've read Nathan Paoletta's post. I find it more useful, clear and to the point in addressing the same matter. It also confirms my firm opinion that you don't need slapping people's faces to get their attention.
That is a major one, speaking of bad tendencies.
I've seen it too often for my tastes. It always makes me sad, especially when it comes from accomplished game designers.
77. On 2011-02-23, Vincent said:
Oh for god sake.
Please, no more complaints about the tone of Ben's piece. They're off-topic. If you really need to give Ben feedback on his writing, this is not the appropriate venue. Drop him a line, maybe.
If you have something to say about his points, that's still very welcome.
78. On 2011-02-23, Chris Chinn said:
A few years ago, I started developing The Emperor's Heart and had to learn everything Ben is talking about, the hard way.
The signal to noise ratio was also made much worse, because of the status-culture he's talking about. I had people jumping into playtest, who had never even heard of wuxia, and complaining that the game didn't fully immerse them in the genre because they couldn't be bothered to watch a few movies. It was game tourism, looking for the new hotness.
I even tried to promote more playtesting in the hopes of better feedback for a bit.
Luckily in a bad way, I got hit with health issues and had to put everything down. It gave me the chance to step back and really look at what I was doing and what I was getting from my efforts.
The easy trap a lot of folks have fallen into, is exchanging the core values of playing games or making games to be played, with impressing other people about how you're involved with games.
It's a lot more socially rewarding than sitting at home, thinking really hard about a game that doesn't exist yet, and may not be fun, yet.
And it basically comes down to self promotion, though a lot of folks keep trying to slide it under "community participation", "enthusiasm" or even "theory" when it's usually either a lot of empty attention questions or name dropping for status.
For example, how many folks even bothered to think about how Emily's Story Capital or Mo's Push & Pull is all up on Apocalypse World's Moves and Reward systems? Vincent has more name dropping value than either Mo or Emily, and so, it's really interesting to look at how ideas 3-4 years past are magically erased, even as things are built upon them.
Crowd-following is great as a fan and a gamer, and terrible as a designer - it will never seek roots of anything you're looking at, and, if anything, obscure them as people are more focused in exciting/new things, rather than why/how anything works.
Obviously, the line is thin- "every gamer is a designer" is not far off, yet, at the same point, just because you can do both doesn't mean it works well when you do both at the same time, or mix up one for the other.
79. On 2011-02-25, Marshall B said:
Pseudo-marginalia: what's Emily's Story Capital? I know about Push & Pull but the former is a new one on me. Can you link me?
80. On 2011-02-25, Suna said:
Vincent, a message's form, choice of words, examples provided (if any) and overall tone are a part of that message. If you leave a message in the form of a blog post that can be commented, then commenting any part of that message is not offtopic, including the form.
I had to discuss with friends this post by Lehman in order to understand whom it was aimed at and whom it is referring to. I'm quite far from being stupid, but I did not have the same problem with Nathan Paoletta's post. It was clear, reader-friendly.
And here I thought we were having a civil conversation. Whatever, I won't disturb further (also because I'm sure this will be pointed as offtopic).
There was no need to be so aggressively defensive, Vincent.
81. On 2011-02-25, Vincent said:
Marshall: I think it's probably this: Emily on Freeform Reward Mechanics.
82. On 2011-02-25, Gregor Hutton said:
Oh, for the love of goodness.
Ben, rock on.
Vincent, keep on posting this stuff.
83. On 2011-02-25, Brand Robins said:
I love social status reward mechanisms. They're awesome!
What I can't really fathom is how well we've managed to build a really terrible set of them. Perhaps we need more playtesting?
84. On 2011-02-25, Vincent said:
Ha ha ha!
We need to stop playtesting them and go back to design.
85. On 2011-02-25, Bret said:
That's our Brand!
86. On 2011-02-25, Joel said:
Actually, I'm quite interested in how to build functional, non-terrible social reward mechanism. Thoughts, Brand?
87. On 2011-02-27, Chris Chinn said:
Marshall: http://www.fairgame-rpgs.com/comment.php?entry=114
88. On 2011-02-28, Vincent said:
89. On 2011-02-28, Vincent said:
Joel: Functional, non-terrible social reward structures exist. That's what this is about. They exist, but they require discipline, clarity of purpose, and a perfect willingness to exclude people who aren't genuinely participating. Of course they do! You can't design a game for people who don't want to play it.
If you're in a forum without those things, you have to bring them with you and hold them for yourself. When you go about in public, you can't ban people, right, so you have to have the discipline and clarity to exclude them from your consideration. If their ideas and opinions aren't useful to you, no matter how socially rewarding, you have to be able to just disregard them. You can't expect any open, public forum to do it for you.
90. On 2011-02-28, Simon C said:
I've been thinking about real-life reward structures a bunch recently.
I was thinking about Knife Fight and what I loved about it and what I hated about it. You're right on, I think Vincent, about excluding people.
I think the myth is that you can have valuable and functional social reward structures as well as open entry to anyone, and equal access to the means of communication.
What I love about Joshua is what drives me nuts in Graham. I can't stop Graham from driving me nuts without also stopping Joshua from saying hilarious shit that cracks me up. Just like you don't invite just anyone to your party, or to your game session, don't invite everyone to your forum, if you want to foster a particular kind of community.
Of course, if you care more about numbers of people participating than the quality of their participation, in all cases, go for open invite.
91. On 2011-02-28, Joel said:
Thanks for the link, Vincent! I've been away for awhile and trying to catch up on all the discussion. For the record, I wasn't expressing skepticism that functional non-terrible social reward could exist, or anything. Just wanting to hear people's thoughts teasing apart WHAT makes it work. So this stuff is just what I need! :)
Great point about excluding, and a great qualifier: "from your consideration." Reminds me of how when I first started learning about boundary-setting (upon discovering that I HAD no functional boundaries), I was all "I HAVE A BOUNDARY PEOPLE! YOU HAVE CROSSED THE BOUNDARY I SET AND NOW I MUST PROTEST LOUDLY!" all over everyone.
Whereas simply, quietly, directing my focus away from folks (online and off) who cross my boundaries is far more functional.
Peace,
-Joel
92. On 2011-03-03, Frank T said:
One merit of playtesting I'd like to point out is that it generates people you can substantially talk to about your design. Sure some people can dig into a manuscript and provide good feedback based on just the text, but if they've played it once, they're just much more concious of the thing as a whole.
93. On 2011-05-26, isildur said:
(Context: I get paid to design computer games, and design RPGs as a way to think about games outside of the aggravation of my day job. Take from that what you will; I have no real ambitions to publish what I write because I have no real motivation to do so.)
I was running Mongoose's Traveller, and I didn't like the ship combat system. It seemed to be a system that gave one or two players a lot of interesting stuff to do, and everyone else nothing to do.
I rewrote the whole thing from scratch into a resource management game, in which players were able to spend currency to affect the action, or trade currency with each other according to some constrained pathways.
I used this for a major ship battle, then debriefed people on the new system. I took their experiences, added them to my observations about how well the system accomplished its overall goal (to give every player a stake in the scene and the ability to influence it), and also how well the specific resource-flow pathways functioned (did everyone have some resources at all times? were people given incentive to move resources to each other as well as spending them? did resources back up in a glut anywhere?).
The system featured problems in all these areas, so I revised the specific mechanics that seemed to cause the problems, and used the revised system in the next major ship combat, a week or so later. Part of that revision was proposing my changes to each sub-component to the person who'd been using that sub-component, and getting their reaction. This was usually in the form of 'Do you think it would have been more interesting to do X?' and then 'Would you be willing to try X next week to see if it works better?'
I eventually got the system to a point where I felt like it was sufficient to continue using for play, although it almost certainly had a few more problems that could have been rooted out.
This is a playtest.
Step one: build a game or game component, until you believe it is done and working properly.
Step two: use it yourself, with a group of players who are amenable to being the front line of a potentially broken system.
Step three: revise based on your observations in step two, and those of your players.
It's possible that there's some other kind of 'playtest' of which I am not aware, but I can't imagine what would happen in that sort of playtest.
As an aside, this:
"The right thing to do about rules holes is to take your rules set, once it's finished being written and modified and so on, and critically evaluate every possible rules interaction, looking for holes. Use a pencil and paper, if you're like me, or a blank text file, or even just your own head and the shower..."
made me laugh and laugh and laugh. I shared it with my design team, and they laughed too. In the MMO world, we assume that even a few thousand players become a genetic algorithm dedicated to 'solving' your game. MMO players will, as a mass, find something as tiny as a two percent variance in power between character A and character B. The idea that a pencil and paper could possibly keep ahead of them is absurd.
94. On 2011-05-26, Ben Lehman said:
Note to those following along at home: there's a couple of month gap here.
When I write an essay targetted at MMO designers, you'll know. Presumably it will be when I know anything at all about MMO design which would be after I've written an MMO.
yrs—
—Ben
95. On 2011-05-26, isildur said:
huh... I had a whole reply typed up, and realized that your response was actually a dismissal.
http://www.penny-arcade.com/comic/2004/3/24/
Enjoy your niche.
96. On 2011-05-26, Elizabeth said:
For what it's worth, in the stock photography world we assume that numbers are a good starting point, but that there's nothing very interesting about a mathematically perfect composition? it's about the artistic eye.
I mean, as long as we're all talking about how this essay doesn't jive with our day jobs.
97. On 2011-05-26, Elizabeth said:
Also, playtesting photographs is ridiculous and absurd and I'm not even sure how you would do it. So I agree with Ben.
98. On 2011-05-27, Emily said:
Hmm. I'm having all sorts of savory and unsavory thoughts about playtesting photographs. Shadowplays anyone?
Isildur's write-up puts into words the best experiences I've had (or hoped to) with playtest.
99. On 2012-03-24, Brian said:
Ben,
I cited this article in my answer to: A question on the RPG stack exchange about what elements are necessary for a playtest. I'd love to get your feedback on my answer (as I hope I captured your meaning well.)
100. On 2012-03-24, Ben Lehman said:
Brian: I think that partially quoting is probably doing you a dis-service. You're right in that it's absolutely not the right time for that guy to playtest, though.
RSS feed: new comments to this thread
This thread is closed to new comments.
home: anyway.
newer thread: 2011-02-18 : 2010 at lumpley games
older thread: 2011-02-16 : OH MY GOD OH MY GOD
