Now that we’ve provided some pretty detailed guidance on how to exploit flawed assessments, we’d like to take that same knowledge and provide some guidance on designing better assessments; ones that don’t have easily-exploitable flaws that savvy test-takers can use to distort the assessment results.
Avoid Obvious Structural Flaws
If your assessment includes multiple-choice questions, you’ll want to avoid grammatical inconsistencies in how the choices are written. As you’ve seen from earlier installments of this series, savvy test takers can exploit patterns in the grammar of your questions to help identify wrong answers.
Best Practice: After writing the assessment, look at the grammar of multiple-choice answers. Verify that there are no grammar patterns that correspond with correct or incorrect answers.
Example:
Question: What is the correct order of steps for starting your car’s engine?
A. Insert the key, secure your seat belt, verify that the driver’s window is cracked open, and then start the engine.
B. Insert the key, check the rear-view mirror, start the engine.
C. Secure your seat belt, check the rear-view mirror, insert the key, start the engine.
Note that answer A ends the comma clause with “and then,” while B and C use a different structure. This inconsistency in parallel structure could be enough to tip a savvy test taker that A is either the right answer or a poorly written distractor.
When writing questions, structure the questions as consistently as possible. By doing this, you focus on assessing knowledge or skill rather than forcing students to waste time compensating for variations in the question structure or wondering whether variations in question structure are meaningful or not.
Best Practice: After writing the assessment, read all the questions. Verify that they are reasonable consistent in their structure. Example structure: Describe a scenario, provide additional important details, then ask how the learner would solve a particular problem related to that situation.
Example:
Question 1: You are in a dark alley. You hear footsteps behind you. What should you do?
Question 2: You are in a foreign country. A stranger asks you to deliver an unmarked package for them. What should you do?
Question 3: What should you do if you get lost in a strange city and someone asks you for money?
Question 3 uses a completely different structure than the other two. This can be confusing to the person taking the assessment.
In some contexts, patently wrong or silly answers can help identify random guessing attempts, but on assessments designed for an adult audience, silly answers have no place.
Best Practice: Avoid silliness when writing wrong answers. A wrong answer that is purposefully silly is insulting to your audience and reduces the quality of your assessment.
Design and Implement an Appropriate Mix of Conceptual Knowledge
It’s easy to create an assessment that targets knowledge of trivia. If it’s software knowledge you’re assessing, it’s very easy to determine whether the learner knows where to click to accomplish a desired outcome. If it’s customer service telephone skills you’re assessing, it’s very easy to determine whether the learner understands that it’s better to refer to a caller as Sir/Madam, their first name, or Dude.
What’s more difficult is to assess conceptual knowledge. But before you attempt to create an assessment that tests for conceptual understanding, make sure that’s audience-appropriate! In other words, you need to circle back around to your training objectives. You did design your training around defined objectives, didn’t you? Your assessment(s) should support the same objectives your training was also designed to support. In fact, this is what assessments do: verify the success if your training.
Best Practice: Map assessment design to training objectives.
Avoid the Other Common Assessment Flaws: Excessive Trivia and Edge Case Interest
Again, assessing conceptual knowledge is more difficult than assessing understanding of trivia. This can result in assessments that have an excessive percentage of questions that are designed to test:
- Trivia: obscure details of the subject you are assessing that have little or no relevance to your training objectives.
- Excessive interest in edge cases: Unusual situations that pertain to the subject you are assessing. Example: You are an astronaut. How long can you survive the vacuum of space without a space suit?
Best Practice: Read through the finished assessment. Classify each question based on whether it is a skill, knowledge, detail, trivia, or edge case question. Obviously, this is a subjective call. Determine whether the breakdown of question type is appropriate to training objectives and audience. Redesign or rewrite questions as necessary.
Finally, Assess Your Assessment
Despite your best efforts to write a good assessment, you may end up with some questions that are clunkers, poorly written, or are otherwise invalid tests of knowledge or skill. The best way to deal with this is to beta test your assessment. Administer your assessment to a relatively small but representative group of people; perhaps 5% or so of the expected audience. Look for patterns in the results:
- Outliers: questions that almost everybody or nobody gets correct. These are probably flawed in some way and need to be redesigned or rewritten.
- Result distribution: What is the distribution of assessment outcomes? Are you trying to fit the outcome to a predetermined distribution, or meet some other criteria?
Conclusion
Good assessments are an important part of any training effort. By paying attention to consistent grammar, consistent question structure, and by mapping your assessment design to your learning objectives, you can create assessments that provide maximally useful feedback for you and your learners.
