thread: 2011-02-17 : Ben Lehman: Playtesting: Stop
On 2011-02-19, jenskot wrote:
"Now that we know the bad way... what's the good way to playtest?" was asked on Story-games in reaction to this thread. My response...
My playtesting (Usability Testing) experience is with interface design (websites, applications) and information design (instruction booklets, posters) but this is how I would playtest RPGs...
Answer these questions...
1. What is your game? What problem does it solve? What makes it unique?
2. Who is your game for?
Without the above, you don't know what you're playtesting. You don't know who should playtest your game. Additionally, you won't know which playtest information to take seriously and what to ignore. And finally, you won't know if you even need to playtest.
Why are these answers important?
- If you don't know what your game does and doesn't do, how do you know what feedback to listen to? You can't succeed, if you don't have goals. You won't know what direction to walk unless you know where you're headed. You won't know when to stop unless you know what your final destination looks like.
- If your game is just for your personal gaming group, why bother with outside playtesting?
- If your game is just for you, why does it matter what anyone else thinks?
- If strategy and tactics are important to your game, why send a playtest copy to freeform gamers?
- If you don't know what problem your games solves, how do you know what rules to write?
You can't please everyone!
Ultimately, no matter what you do, thousands maybe millions of people will disagree with you. You can't please everyone. Fortunately, if you know who you are trying to please... you don't have to care what everyone thinks. If you reached 1% of people currently playing D&D... that's 60,000 people. A lot of design, not just game design, is about focus and eliminating anything that distracts from that focus. With focus, you know who and who not to listen to.
Once you identify what problem your game solves, you can nail down whose problem you're solving. This is your audience. This is who you want playtesting your game.
I can't find playtesters!
If you can't find anyone to playtest your game, that might be a warning sign to go back and revise what problem your game solves and for who. If people don't want to playtest, maybe it's because your game doesn't actually solve a problem they have. Or you've misidentified your audience and are talking to the wrong people. Or your audience is very busy and needs fair compensation to make playtesting worth their time. If you can't compensate them, maybe the project is too small to warrant all the time, energy, and resources dedicated to it as a commercial project and would be better off released non-commercially.
Every roadblock you hit is a sign that you may need to rethink your strategy. It's a gift. It will force you to step back, sharpen your focus, and ultimately learn from your experience to make you even stronger.
Even if you find playtesters, be realistic about your expectations...
Sometimes people volunteer to playtest and never come through. If 1 person out of 3 who volunteer actually deliver... I would consider that a success! If you need 6 playtesters, find 18.
Create your playtest...
Decide what you need playtested. Think small. Start with the absolute bare minimum rules your game needs to achieve its goals. Identify the top 1-3 things you need tested. Ignore everything else. Create scenarios that will allow playtesters to focus and test these top priorities.
Design your scenarios so what you're testing isn't obvious to the playtesters. Your scenario might be, "create a character" but what you're specifically testing is "how long does it take", "is stat allocation frustrating", "does character creation give the GM enough information to design an adventure." Don't tell your playtesters what you are actually playtesting.
Take caution that your scenario doesn't influence your playtesters actions. Don't ask leading questions or make leading statements. If you want to test "how long does this take", in your scenario, don't say "character creation is super fast". Don't influence!
Run through your scenario before you give it to playtesters. Make sure it's clear and error free, otherwise you tests may fail for reasons unrelated to your game. Check your grammar, make sure any jargon is clearly defined, give clarifying examples, and cut out anything that isn't related to your test that could distract from your goals.
Running playtests...
Make sure your audience is uninitiated. Don't test the same scenarios with the same people. Ideally, don't run multiple tests with the same people at all. And run each test with 3 separate groups of testers.
Record all your tests. It may not be possible for you to be present for all your tests. And even if you are, in the moment you may miss something.
Ideally your scenarios are prepared so playtesters can run through them without your involvement. Your goal is to observe and not interfere. If your rules are in the pre-alpha state (more ideas than solid procedures), feel free to guide the playtesters through your scenarios but recruit a third party to observe the test or absolutely make sure to record the test so you can make observations later. Don't multitask.
Train your playtesters to think out loud. If you're recording audio, you'll miss all their revealing facial expressions. Talk to them, make them feel comfortable, run through practice scenarios first (not your actual real scenarios) so people get used to thinking out loud. If people are quiet, ask them what they are thinking. If people talk out loud, nod along and give them encouragement. Reassure them that the playtest is a safe space, they can say anything, just be honest. Make sure they know that they aren't being judged. I know this will sound funny... but it's probably best not to tell them you're the game designer! That alone will bias the test.
If you're taking notes while people playtest, do so in a way that people can't see you.
Don't ask playtesters for their opinions. Especially after the test is over. People are unreliable self-reporters. Plus their job is to playtest, not observe themselves playtesting or trying to fix your problems. Your goal is to make observations by studying what people do, not what they say they did. In playtesting, actions speak louder than words!
Creating reports, identifying problems, solutions, and more tests!
Review your recordings and notes to create a report identifying all the problems (even if you don't think they are actual problems). Take special note of any problems or unusual behavior that happen more than once.
If you can, get a second person to create a report by reviewing the recordings and identifying the problems independently of your observations. Compare the two reports.
Compare these problems to your goals.
Decide which problems actually need to be solved.
Fix these problems.
Create new scenarios to confirm your fixes work.
Run additional playtests with new people.
Playtesting your text is more important than playtesting your rules!
When someone buys your game... all your rules become guidelines to be followed or discarded at the whim of your audience!
And even if they decide to follow your guidelines, the rules don't matter unless they understand what the rules are, how to use them, why they are important, when to use them, what will happen if you don't use them... and then actually remember all this when it is time to play!
This is where your game text comes in and why playtesting text is more important than playtesting rules. Your rules have no power at anyone's gaming table. You must persuade players to use them (this is true even in boardgames). Your text is your only tool of persuasion. Use it wisely!
Your text needs to do more than teach. Your text needs to turn its advocates into teachers to spread its message. Your text has to arm its readers with the tools to teach other people who will never bother to read your text! This is where handouts, character sheets, and reference sheets all come in. They not only help people who haven't read the rules to learn... but also remind everyone what the rules are, when, and how to use them.
I don't know how to write my rules...
But you probably know how to verbally teach others how to play them. So do that and record yourself. Then listen to the recording and create an outline. What order did you explain things? What questions did people ask? What rules needed extra explanation? What examples really helped people understand how to play? What confused people?
And then do it again but this time, as scary as it might sound, video tape yourself doing it! Why? Because when you're teaching people how to play, you're probably using body language and visual cues (pointing to specific rule pages, charts, character sheets, rolling dice) to help people understand what you're talking about. And your text can do the same thing with the help of images.
And hire an editor!
Editing isn't just fixing grammar and spelling mistakes!
Editing is also about content. What's the best way to express an idea? What order is the clearest and most effective way to express your ideas? Which ideas need to be reinforced with examples? Which ideas aren't important and need to be cut out?
Editing is not enough!
You need to playtest your editing just like you would playtest any rule changes. Grab someone and watch them read your rules out loud. If its' painful, don't turn away... find out why it hurts and fix it.
If I do all this, will my game sell?
Does a contract protect your rights? Not absolutely. My friend Chris who's an IP lawyer once described contracts as a +2 circumstance bonus in a d20 law roll. Nothing is guaranteed but just like in D&D when you're facing a red dragon, you're going to want every bonus you can get! You might attack unprepared and get lucky rolling a natural 20! Or watch in despair when that die reveals that cruel critical miss... a natural 1! But the game isn't about just 1 roll. It's about trying again and again. Learning from mistakes. And arming yourself as best you can.
I hope that helps!
It all sounds harder than it actually is. And even if you can only do 25% of the above... you're ahead of most game designers! Hell, if you just clearly know what problem your game solves and who your audience is... that alone puts you way ahead. Focus on doing what you can. Be honest with yourself. Remember that getting started is the hardest part. And doing the first 10% is the first step in doing the next 10%.
Rock,
John
