Two years after starting Word Lions with Philip, I now empathize with that parent. Talking to me about anything other than content creation, instructional design, or running a small business is akin to trying to have a conversation with that new parent. Their baby always makes its way into the conversation—sometimes without a thread of relevance.

I have become a broken record, and I love it. I am so deeply thankful for my life and my work:

• I am thankful that my work is to provide valuable, meaningful content for people who need it.
• I am thankful that I can write for a living.
• I am forever thankful that I can partner with the inestimable Philip Morgan.
• I am thankful that together we are growing a company designed around our authentic selves. That when I am working I am able to be myself.
• I am thankful that we prove every day work can integrate with life.

Growing Word Lions is not easy, but it is constantly interesting and meaningful.

When you talk to a new parent, and ask them how they are doing, they seem to always simply say

“I am tired. All the time.”

It seems unbearably difficult. Now, raising a business, I think I understand why new parents are smiling when they say that.

1. We’ve created a lot of training content that was ultimately distributed either online–via a Learning Management System (LMS) or web page–and via optical media like CD-ROM or DVD.
2. We’ve had to learn a lot about how to do optical media content distribution through difficult on-the-job lessons.

Shiny Disks or Web Pages

You’ll want to decide whether you are going to distribute your content on physical media (shiny optical disks) or electronically (using a LMS or web page) early on in the content development process. Why? Because the Quality Assurance (QA) process is very different depending on which distribution method you go with, and getting the QA right will make or break the customer perception of your training content.

You decide whether to use CD-ROM or HTTP by thinking through the constraints of your training environment and other relevant considerations, summarized in the table below:

Optical Media Best Practices

If optical media is the best distribution method for your training content, keep the following best practices in mind:

• We recommend that customers plan for the same kind of QA procedure you would have if you were distributing software on optical media. In general, this means having at least:
  • One QA pass on the finished content before you create a master CD-ROM/DVD image.
  • One QA pass on the finished content after you use the content to create a master CD-ROM/DVD image.
  • Testing the content end-to-end on one or more computers configured exactly like those your customers will use.
  • Allow adequate time for CD or DVD duplocation and shipping.
  • Allow some time (and budget) for the unexpected!

LMS/HTTP Best Practices

if an LMS or web page is the best distribution method for your case, keep these best practices in mind:

• While we don’t encourage skimping on QA if your are using an electronic distribution method, you certainly can fix issues that your QA process uncovers more easily than with optical media. For this reason, you may want to structure a beta release of your training content–via web or LMS–to a limited audience as a way of testing your content before it’s wider release.
• While there are lots of nice LMS packages that you can self-host, give serious consideration to hosted solutions. Hosted LMS companies can save you a lot of setup, configuration, backup, and other operational headaches.
• Even though an LMS makes lots of things easier, there will be support calls. Create good instructions on using the LMS for your customers, and a good FAQ, to reduce the number of support calls.

In both cases, I ended up asking the clerks a simple question. “What is the benefit to me of being in your system?” In both cases, the clerks had difficulty articulating how I would benefit from taking additional time to complete the sale by giving them my name and address so I could be added to their customer database.

While I’m not surprised that neither clerk could articulate the value or benefit of being in their customer database, it did remind me that this sort of thing happens all the time in the technology industry.

Some companies assume that it is enough to do a good job of describing the features of a product. This is equivalent to the clerks who–in the form of a question–said, “One of the features of our store is a customer database. Would you like to use this feature? It will only cost you two minutes.”

This approach puts the burden of translating the feature into a benefit or value on the customer, who is often ill-prepared to make this translation. Why didn’t these clerks instead frame the feature in terms of its benefits? And not its benefit to the business; i.e. the ability to send me advertising, but in terms of its benefit to the customer. Something like, “If you’re in our system, we can process a merchandise return twice as fast. Are you in our system?”

Remember that while it’s good to describe features, it’s better to lead with the value or the benefit of those features. Save for the feature descriptions for after you’ve hooked a prospect with the value or benefits of your product or service.

If you want to discuss ideas for using content marketing to get the word out about your product, drop us a line at TheLions@wordlions.com.

A few months ago, I ranted about my affection for mechanical keyswitch keyboards. As a writer, having an accurate, responsive keyboard is important, and I was able to meet that need with the relatively inexpensive keyboard Solidtek ASK-6600.

Well, the urge to keep exploring high-end keyboards struck again, and I got a Realforce 86U. This keyboard uses the highly-regarded Topre rubber dome keyswitch. Rubber dome keyswitches are used in some truly awful keyboards, but as it turns out, that does not mean that rubber dome keyswitches are themselves awful.

In fact, they can be a real delight. I love my Realforce 86U! It makes less noise than my Solidtek ASK-6600, it has an ergonomic design that works very well for fast, accurate touch typing, and a springy, precise feel to the keyswitches.

It’s a spendy little keyboard, but I can see myself being very happy with this keyboard for a long time!

By the way, “The good feeling of oneness with cup rubber” is the hilariously garbled english tag line printed on the Realforce product box.

Avoid Obvious Structural Flaws

If your assessment includes multiple-choice questions, you’ll want to avoid grammatical inconsistencies in how the choices are written. As you’ve seen from earlier installments of this series, savvy test takers can exploit patterns in the grammar of your questions to help identify wrong answers.

Best Practice: After writing the assessment, look at the grammar of multiple-choice answers. Verify that there are no grammar patterns that correspond with correct or incorrect answers.

Example: 

Question: What is the correct order of steps for starting your car’s engine?

A. Insert the key, secure your seat belt, verify that the driver’s window is cracked open, and then start the engine.

B. Insert the key, check the rear-view mirror, start the engine.

C. Secure your seat belt, check the rear-view mirror, insert the key, start the engine.

Note that answer A ends the comma clause with “and then,” while B and C use a different structure. This inconsistency in parallel structure could be enough to tip a savvy test taker that A is either the right answer or a poorly written distractor.

When writing questions, structure the questions as consistently as possible. By doing this, you focus on assessing knowledge or skill rather than forcing students to waste time compensating for variations in the question structure or wondering whether variations in question structure are meaningful or not.

Best Practice: After writing the assessment, read all the questions. Verify that they are reasonable consistent in their structure. Example structure: Describe a scenario, provide additional important details, then ask how the learner would solve a particular problem related to that situation.

Example: 

Question 1: You are in a dark alley. You hear footsteps behind you. What should you do?

Question 2: You are in a foreign country. A stranger asks you to deliver an unmarked package for them. What should you do?

Question 3: What should you do if you get lost in a strange city and someone asks you for money?

Question 3 uses a completely different structure than the other two. This can be confusing to the person taking the assessment.

In some contexts, patently wrong or silly answers can help identify random guessing attempts, but on assessments designed for an adult audience, silly answers have no place.

Best Practice: Avoid silliness when writing wrong answers. A wrong answer that is purposefully silly is insulting to your audience and reduces the quality of your assessment.

Design and Implement an Appropriate Mix of Conceptual Knowledge

It’s easy to create an assessment that targets knowledge of trivia. If it’s software knowledge you’re assessing, it’s very easy to determine whether the learner knows where to click to accomplish a desired outcome. If it’s customer service telephone skills you’re assessing, it’s very easy to determine whether the learner understands that it’s better to refer to a caller as Sir/Madam, their first name, or Dude.

What’s more difficult is to assess conceptual knowledge. But before you attempt to create an assessment that tests for conceptual understanding, make sure that’s audience-appropriate! In other words, you need to circle back around to your training objectives. You did design your training around defined objectives, didn’t you? Your assessment(s) should support the same objectives your training was also designed to support. In fact, this is what assessments do: verify the success if your training.

Best Practice: Map assessment design to training objectives. 

Avoid the Other Common Assessment Flaws: Excessive Trivia and Edge Case Interest

Again, assessing conceptual knowledge is more difficult than assessing understanding of trivia. This can result in assessments that have an excessive percentage of questions that are designed to test:

• Trivia: obscure details of the subject you are assessing that have little or no relevance to your training objectives.
• Excessive interest in edge cases: Unusual situations that pertain to the subject you are assessing. Example: You are an astronaut. How long can you survive the vacuum of space without a space suit?

• Outliers: questions that almost everybody or nobody gets correct. These are probably flawed in some way and need to be redesigned or rewritten.
• Result distribution: What is the distribution of assessment outcomes? Are you trying to fit the outcome to a predetermined distribution, or meet some other criteria?

Conclusion

Good assessments are an important part of any training effort. By paying attention to consistent grammar, consistent question structure, and by mapping your assessment design to your learning objectives, you can create assessments that provide maximally useful feedback for you and your learners.

Putting the Pieces Together

Here’s the exam-taking approach I’m suggesting, from beginning to end:

1. Get a conceptual understanding of how the software works
2. Learn what features and capabilities are new to the current version of the software
3. Learn best practices for using the software
4. Learn the limits of the software
5. Learn any other details you can about the software
6. Get plenty of rest before the exam. Make sure you are not hungry, thirsty, or unwell during the exam.
7. Read questions backwards. Identify the problem to be solved, then evaluate potential answers and the scenario in light of the problem to be solved.
8. Use the techniques above to identify Distractors and wrong answers.
9. From the remaining answers, choose a best answer using your domain knowledge.

In the next installment of this series, we’ll start discussing how to mitigate common assessment design flaws in order to create more effective assessments.

Targeting Your Domain Knowledge Study

Just like you can intelligently analyze test question structure during the exam, you can intelligently target your domain knowledge study while preparing for the exam. Here are my tips for maximizing the effectiveness of your exam preparation:

Tip 1: Focus on Concepts First, Trivia Last

Think of your mind like a closet. Details are like the items in the closet. Concepts are like the hooks and hangars that you hang the details on. Without a conceptual understanding of how a piece of software works, you will lack hooks to support the details about that software. For example, how can you make sense of the details about how a network packet is handled if you don’t understand the big picture (the concept) of how packets flow between computers on a network? How can you make sense of the differences between two VPN encrption algorithms if you don’t understand the overall VPN setup and data transfer process on a high level?

So make sure you understand the concepts of how a piece of software works first before you focus on details you might be tested on in the certification exam.

Tip 2: Identify Software Deltas

Find out what has changed (the deltas) between the current and previous version of the software you are studying. Exam writers want to make sure that you have current domain knowledge, so they tend to focus a significant percentage (maybe 20% on some exams) of exam questions on new features, procedures, and functionality for the current version of the software. So identify the deltas to the most recent version of the software, and expand your domain knowledge of how to use these new features, procedures, and capabilities.

Tip 3: Identify Software Best Practices

Many test questions test your understanding of software best practices, rather than details about the software itself. Knowing software best practices will help you with questions that have several correct answers, but only one best answer. The best answer will be identified because it adheres to software best practices.

Tip 4: Learn Interfaces and Procedures

Make sure you have seen and used every interface the software offers, and you have configured, used, and re-configured every function it can perform. Make sure you can identify and understand:

• Every feature described on the software’s website
• Every menu option
• Every configuration setting

Make sure you get plenty hands-on usage of the software. Setting the software up in a Virtual Machine is often a great way to do this.

Tip 5: Identify Software Limits

Sometimes exam writers run out of good questions to ask, so they resort to trivial questions about the limits of the software in question. “How many widgets can this software support?” “If you install parts A, B, and D of the software, will it make coffee for you or not?” These types of questions focus on:

• What the software can not do
• Built-in limits to what the software can do

As you are preparing for an exam, pay particular attention to details of this nature.

Tip 3: Look for Wrong Answers First

Identifying wrong answers using the techniques I’ve outlined above is more mechanistic (uses the pattern-matching capabilities of your brain) than choosing the best answer from among 2 or 3 correct answers (which taps the critical-thinking , associative capabilities of your brain). Using critical thought to analyze 4 or 5 answers is more taxing than using critical thought to analyze 2 or 3 answers. So rule out the wrong answers (the Distractors) first, and free up your mental bandwidth for the possible answers that you can’t mechanisticaly rule out.

Tip 4: Stay organized.

When you start reverse engineering test questions rather than answering them strictly based on your domain knowledge, you run the risk of losing track of which answers have been ruled out. So take notes if it helps and if note-taking is allowed in the exam format. Often you will be provided with a dry-erase tablet for use during certification exams, and this can be a place to take notes.

Tip 5: Mark Iffy Questions

If the exam format allows you to, mark questions you are uncertain about and come back to review your answer later. It is possible that a later question in the exam will jog your memory and help you answer a previous question you marked. Again, use any note-taking capabilities you are allowed to during the exam to stay organized by jotting down a note about the previous question so that when you review your marked questions, you can utilize that note to revise your previous answer to the marked question.

It’s a good idea to allocate about 10% of the total exam time limit to question review.

Tip 2: Eliminate Distractors

Distractors are wrong answers. Distractors can range from “close but no cigar” to patently wrong. Over the course of my personal experience with many certification exams, I have noticed some patterns that are common to many Distractors. Remember, your goal is to use your domain knowledge to choose the correct/best answer to every question on a certification exam, but the more quickly you can weed out wrong answers, the more time available to you to choose the best answer from among the correct answers. Developing your ability to spot distractors is very useful.

Syntax Clues

Good technical writing uses parallel structure when creating lists of items. This technical writing norm often appears in certification exams as well, with the list of potential answers for a question using parallel structure. Look for any exceptions to parallel structure when reading the list of potential answers. An answer that deviates from parallel structure is likely one of the following:

• A poorly written distractor.
• A correct/best answer.

Wait, really? You can identify the correct answer just by its grammar? Based on my experience, I would say that 15% of the time it is possible to use only your ability to spot exceptions to parallel list structure to identify a correct answer even if you have little or no domain knowledge that is relevant to that question. Remember, that some exam questions have several correct answers, and after you have weeded out the distractors you must choose the best answer from among the correct answers, so weeding out the distractors is only part of the challenge.

I’m going into speculation mode here, so treat this as a sidebar, but I suspect that Distractors are often written by someone who lacks the domain knowledge of the person/team who wrote the correct answer. This could cause a small, but perciptible, difference in the grammar or sentence structure of the Distractor answer(s).

It’s very difficult to write good Distractors. And sometimes, as a test-taker, you just get lucky when the exam authors overshoot and put in a Distractor that is just patently wrong. Often you can use your domain knowledge to determine which Distractors are patently wrong or infeasible.

Factual Clues

Look closely for references to non-existent interface components or wrong procedures. The ability to spot Distractors based on factual clues is dependent on your level of domain knowledge, but remember that Distractors often use the following format:

Answer = correct procedure + fictitious software interface name

OR

Answer = incorrect procedure + correct software interface name

Distractors that follow either of the above two formats fall into the “close but no cigar” category and can be ruled out immediately.

Remember that many software companies use fairly strict style guides when naming interface components, and they try to keep interface names consistent. Distractors that reference fictitious interface components or procedures that don’t exist in the actual product sometimes clearly and obviously deviate from the usual style guide norms for interface components. This can be a subtle clue that a potential answer is actually a Distractor.

Logical Stretches

Distractors will often outline a procedure that deviates from best practices or something that is just not possible in the software you are being tested on. This is why, when you are working on your domain knowledge in preparation for the exam, you want to focus on best practices and limits to what the software can do. These two areas easily lend themselves to the challenge of writing test questions, and so they are obvious candidates for your domain knowledge preparation. More on this below.

Our understanding of how to create effective assessments is based on a lot of experience taking other people’s assessments. In the course of taking dozens of assessments, we’ve identified common design flaws and actually learned to exploit these flaws. In other words, we’re “good test takers.”

In this blog series, we’d like to first present the results of our experience exploiting design flaws in assessments, then we’ll make recommendations for how to avoidthese same flaws in assessments.

Our Credentials

Philip has attempted and passed 17 Microsoft software certification exams and earned 6 certifications. In the process he has learned a few things about software certification exams, and we think they might apply to other multiple-choice software certification exams:

• Domain knowledge (factual knowledge of the exam subject) is important but by itself is often not enough to make you successful on a multiple-choice exam.
• Understanding the exam format and analyzing the structure of each question can help you eliminate wrong answers (thereby raising the odds of a correct guess), even if your domain knowledge is insufficient to point out the correct answer for a given question.
• Good domain knowledge combined with an understanding of a few exam-taking techniques described below can raise your success rate on certification exams.

Reverse Engineering the Exam Format

Here are my tips for “reverse engineering” the exam format. The goal of all these tips is to help you quickly weed out wrong answers so that you can have more time to use your domain knowledge to choose the correct or best answer. These tips apply almost exclusively to multiple-choice text-based questions, not questions that use a simulated software environment. More quickly choosing the correct or best answer for each question does two beneficial things:

• Reduces the chance the exam time limit will expire before you have completed the exam and reviewed any questions you are not sure about (if the exam format allows question marking for review).
• Reduces your fatigue, raising your chances of correctly answering questions that occur later in the exam.

By the way, make sure you know the difference between a correct and a best answer! Many software certification exams will present two or three answers that correctly address the question, but one of those answers will entail less effort or time and that makes it the “best” answer. Exam instructions will tell you whether you need to choose correct answers or best answers.

Tip 1: Read the Question Backwards

Many certification exam questions are lengthy, and provide multiple supporting details or a scenario that is related to the question. The typical question structure looks like this:

1. Scenario
2. Description of the Problem you have to solve
3. List of Answers, each of which is one potential way to solve the problem

Because the Scenario section can often be lengthy, by the time you reach the second section that describes the Problem you must solve, you may have lost track of important details in the scenario. This forces you to go back and read the scenario a second time, costing you time on that question. Why not read the question in the following order:

1. Description of the Problem you have to solve (often the last sentence or paragraph in the question)
2. List of Answers, one of which may “jump out” at you as the correct answer.
3. Lastly, read the Scenario to make sure you haven’t missed anything.

By approaching the question in this order, you might save a minute or two on some questions, and on timed exams this can be crucial. If your domain knowledge is particularly good, you can scan or entirely skip reading the Scenario section if one of the answers is obviously correct, further saving time and reducing fatigue.

If you do need to read the Scenario, you will do so after reading the description of the Problem you must solve and after reading the potential answers. As you read the Scenario, you will be prepared to mentally filter out parts of the Scenario that have nothing or little to do with the Problem. This will help focus your thinking as you evaluate the answers to the question.

The next installment of this blog post series will have more test-taking tips.