WordLions.com

This post continues our series on the common flaws we’ve found in assessment tests (and how to exploit those as a test-taker). After we finish describing the common flaws, we’ll talk about how to avoid those flaws through better assessment design.

Putting the Pieces Together

Here’s the exam-taking approach I’m suggesting, from beginning to end:

1. Get a conceptual understanding of how the software works
2. Learn what features and capabilities are new to the current version of the software
3. Learn best practices for using the software
4. Learn the limits of the software
5. Learn any other details you can about the software
6. Get plenty of rest before the exam. Make sure you are not hungry, thirsty, or unwell during the exam.
7. Read questions backwards. Identify the problem to be solved, then evaluate potential answers and the scenario in light of the problem to be solved.
8. Use the techniques above to identify Distractors and wrong answers.
9. From the remaining answers, choose a best answer using your domain knowledge.

In the next installment of this series, we’ll start discussing how to mitigate common assessment design flaws in order to create more effective assessments.

This post continues our series on the common flaws we’ve found in assessment tests (and how to exploit those as a test-taker). After we finish describing the common flaws, we’ll talk about how to avoid those flaws through better assessment design.

Targeting Your Domain Knowledge Study

Just like you can intelligently analyze test question structure during the exam, you can intelligently target your domain knowledge study while preparing for the exam. Here are my tips for maximizing the effectiveness of your exam preparation:

Tip 1: Focus on Concepts First, Trivia Last

Think of your mind like a closet. Details are like the items in the closet. Concepts are like the hooks and hangars that you hang the details on. Without a conceptual understanding of how a piece of software works, you will lack hooks to support the details about that software. For example, how can you make sense of the details about how a network packet is handled if you don’t understand the big picture (the concept) of how packets flow between computers on a network? How can you make sense of the differences between two VPN encrption algorithms if you don’t understand the overall VPN setup and data transfer process on a high level?

So make sure you understand the concepts of how a piece of software works first before you focus on details you might be tested on in the certification exam.

Tip 2: Identify Software Deltas

Find out what has changed (the deltas) between the current and previous version of the software you are studying. Exam writers want to make sure that you have current domain knowledge, so they tend to focus a significant percentage (maybe 20% on some exams) of exam questions on new features, procedures, and functionality for the current version of the software. So identify the deltas to the most recent version of the software, and expand your domain knowledge of how to use these new features, procedures, and capabilities.

Tip 3: Identify Software Best Practices

Many test questions test your understanding of software best practices, rather than details about the software itself. Knowing software best practices will help you with questions that have several correct answers, but only one best answer. The best answer will be identified because it adheres to software best practices.

Tip 4: Learn Interfaces and Procedures

Make sure you have seen and used every interface the software offers, and you have configured, used, and re-configured every function it can perform. Make sure you can identify and understand:

• Every feature described on the software’s website
• Every menu option
• Every configuration setting

Make sure you get plenty hands-on usage of the software. Setting the software up in a Virtual Machine is often a great way to do this.

Tip 5: Identify Software Limits

Sometimes exam writers run out of good questions to ask, so they resort to trivial questions about the limits of the software in question. “How many widgets can this software support?” “If you install parts A, B, and D of the software, will it make coffee for you or not?” These types of questions focus on:

• What the software can not do
• Built-in limits to what the software can do

As you are preparing for an exam, pay particular attention to details of this nature.

This post continues our series on the common flaws we’ve found in assessment tests (and how to exploit those as a test-taker). After we finish describing the common flaws, we’ll talk about how to avoid those flaws through better assessment design.

Tip 3: Look for Wrong Answers First

Identifying wrong answers using the techniques I’ve outlined above is more mechanistic (uses the pattern-matching capabilities of your brain) than choosing the best answer from among 2 or 3 correct answers (which taps the critical-thinking , associative capabilities of your brain). Using critical thought to analyze 4 or 5 answers is more taxing than using critical thought to analyze 2 or 3 answers. So rule out the wrong answers (the Distractors) first, and free up your mental bandwidth for the possible answers that you can’t mechanisticaly rule out.

Tip 4: Stay organized.

When you start reverse engineering test questions rather than answering them strictly based on your domain knowledge, you run the risk of losing track of which answers have been ruled out. So take notes if it helps and if note-taking is allowed in the exam format. Often you will be provided with a dry-erase tablet for use during certification exams, and this can be a place to take notes.

Tip 5: Mark Iffy Questions

If the exam format allows you to, mark questions you are uncertain about and come back to review your answer later. It is possible that a later question in the exam will jog your memory and help you answer a previous question you marked. Again, use any note-taking capabilities you are allowed to during the exam to stay organized by jotting down a note about the previous question so that when you review your marked questions, you can utilize that note to revise your previous answer to the marked question.

It’s a good idea to allocate about 10% of the total exam time limit to question review.

This post continues our series on the common flaws we’ve found in assessment tests (and how to exploit those as a test-taker). After we finish describing the common flaws, we’ll talk about how to avoid those flaws through better assessment design.

Tip 2: Eliminate Distractors

Distractors are wrong answers. Distractors can range from “close but no cigar” to patently wrong. Over the course of my personal experience with many certification exams, I have noticed some patterns that are common to many Distractors. Remember, your goal is to use your domain knowledge to choose the correct/best answer to every question on a certification exam, but the more quickly you can weed out wrong answers, the more time available to you to choose the best answer from among the correct answers. Developing your ability to spot distractors is very useful.

Syntax Clues

Good technical writing uses parallel structure when creating lists of items. This technical writing norm often appears in certification exams as well, with the list of potential answers for a question using parallel structure. Look for any exceptions to parallel structure when reading the list of potential answers. An answer that deviates from parallel structure is likely one of the following:

• A poorly written distractor.
• A correct/best answer.

Wait, really? You can identify the correct answer just by its grammar? Based on my experience, I would say that 15% of the time it is possible to use only your ability to spot exceptions to parallel list structure to identify a correct answer even if you have little or no domain knowledge that is relevant to that question. Remember, that some exam questions have several correct answers, and after you have weeded out the distractors you must choose the best answer from among the correct answers, so weeding out the distractors is only part of the challenge.

I’m going into speculation mode here, so treat this as a sidebar, but I suspect that Distractors are often written by someone who lacks the domain knowledge of the person/team who wrote the correct answer. This could cause a small, but perciptible, difference in the grammar or sentence structure of the Distractor answer(s).

It’s very difficult to write good Distractors. And sometimes, as a test-taker, you just get lucky when the exam authors overshoot and put in a Distractor that is just patently wrong. Often you can use your domain knowledge to determine which Distractors are patently wrong or infeasible.

Factual Clues

Look closely for references to non-existent interface components or wrong procedures. The ability to spot Distractors based on factual clues is dependent on your level of domain knowledge, but remember that Distractors often use the following format:

Answer = correct procedure + fictitious software interface name

OR

Answer = incorrect procedure + correct software interface name

Distractors that follow either of the above two formats fall into the “close but no cigar” category and can be ruled out immediately.

Remember that many software companies use fairly strict style guides when naming interface components, and they try to keep interface names consistent. Distractors that reference fictitious interface components or procedures that don’t exist in the actual product sometimes clearly and obviously deviate from the usual style guide norms for interface components. This can be a subtle clue that a potential answer is actually a Distractor.

Logical Stretches

Distractors will often outline a procedure that deviates from best practices or something that is just not possible in the software you are being tested on. This is why, when you are working on your domain knowledge in preparation for the exam, you want to focus on best practices and limits to what the software can do. These two areas easily lend themselves to the challenge of writing test questions, and so they are obvious candidates for your domain knowledge preparation. More on this below.

One of the things Word Lions does is design and author software training. From time to time, this entails creating quizzes or tests to assess whether learners have retained and integrated new concepts and details.

Our understanding of how to create effective assessments is based on a lot of experience taking other people’s assessments. In the course of taking dozens of assessments, we’ve identified common design flaws and actually learned to exploit these flaws. In other words, we’re “good test takers.”

In this blog series, we’d like to first present the results of our experience exploiting design flaws in assessments, then we’ll make recommendations for how to avoidthese same flaws in assessments.

Our Credentials

Philip has attempted and passed 17 Microsoft software certification exams and earned 6 certifications. In the process he has learned a few things about software certification exams, and we think they might apply to other multiple-choice software certification exams:

• Domain knowledge (factual knowledge of the exam subject) is important but by itself is often not enough to make you successful on a multiple-choice exam.
• Understanding the exam format and analyzing the structure of each question can help you eliminate wrong answers (thereby raising the odds of a correct guess), even if your domain knowledge is insufficient to point out the correct answer for a given question.
• Good domain knowledge combined with an understanding of a few exam-taking techniques described below can raise your success rate on certification exams.

Reverse Engineering the Exam Format

Here are my tips for “reverse engineering” the exam format. The goal of all these tips is to help you quickly weed out wrong answers so that you can have more time to use your domain knowledge to choose the correct or best answer. These tips apply almost exclusively to multiple-choice text-based questions, not questions that use a simulated software environment. More quickly choosing the correct or best answer for each question does two beneficial things:

• Reduces the chance the exam time limit will expire before you have completed the exam and reviewed any questions you are not sure about (if the exam format allows question marking for review).
• Reduces your fatigue, raising your chances of correctly answering questions that occur later in the exam.

By the way, make sure you know the difference between a correct and a best answer! Many software certification exams will present two or three answers that correctly address the question, but one of those answers will entail less effort or time and that makes it the “best” answer. Exam instructions will tell you whether you need to choose correct answers or best answers.

Tip 1: Read the Question Backwards

Many certification exam questions are lengthy, and provide multiple supporting details or a scenario that is related to the question. The typical question structure looks like this:

1. Scenario
2. Description of the Problem you have to solve
3. List of Answers, each of which is one potential way to solve the problem

Because the Scenario section can often be lengthy, by the time you reach the second section that describes the Problem you must solve, you may have lost track of important details in the scenario. This forces you to go back and read the scenario a second time, costing you time on that question. Why not read the question in the following order:

1. Description of the Problem you have to solve (often the last sentence or paragraph in the question)
2. List of Answers, one of which may “jump out” at you as the correct answer.
3. Lastly, read the Scenario to make sure you haven’t missed anything.

By approaching the question in this order, you might save a minute or two on some questions, and on timed exams this can be crucial. If your domain knowledge is particularly good, you can scan or entirely skip reading the Scenario section if one of the answers is obviously correct, further saving time and reducing fatigue.

If you do need to read the Scenario, you will do so after reading the description of the Problem you must solve and after reading the potential answers. As you read the Scenario, you will be prepared to mentally filter out parts of the Scenario that have nothing or little to do with the Problem. This will help focus your thinking as you evaluate the answers to the question.

The next installment of this blog post series will have more test-taking tips.