0

Especially where there are numerous possible answers to each question, e.g. 5.

If one doesn’t know the answer to a question but puts forth a (somewhat) educated guess, at what point is the chance of getting >0 for this question secondary to lowering the value of ‘definitely’ correct answers?

In other words, by how much does ‘indiscriminate’ guessing devalue points accrued from confident answers?

Perhaps this is more of a statistical question, but hopefully there are some general opinions on this matter.

Mad Banners
  • 141
  • 4
  • 18
    Depends on the the scoring/weighting system. – Daniel R. Collins Jun 05 '22 at 06:29
  • 1
    As @DanielR.Collins said. And in addition it also depends on what an "educated guess" means in statistical terms. Moreover, whether guessing is really advantageous for you also depends on your personal preference on how to weigh higher expected outcome against higher standard deviation of the outcome. – Jochen Glueck Jun 05 '22 at 07:00
  • 1
    To the close voters - which "community-specific guideline" does this question violate? – xLeitix Jun 05 '22 at 07:37
  • 12
    @xLeitix: I voted to close because it's essentially a question about probability theory, not about academia. – Jochen Glueck Jun 05 '22 at 09:02
  • 1
    I'm with Jochen, et al. In statistical terms, the Question is so trivial as to be almost irrelevant. Other than in statistical terms, how could the Question have a useful meaning? – Robbie Goodwin Jun 05 '22 at 23:12
  • 1
    @JochenGlueck Yes, it's about probability, but very much applied to a context that many students and some instructors care deeply about. I don't think a question is out of scope because it also happens to involve probability. – xLeitix Jun 06 '22 at 07:47
  • 1
    @RobbieGoodwin I would say that there are three answers which get debated quite hotly is a good indicator that the real-life interpretation isn't so straight-forward. I really do believe that question should not be closed. – xLeitix Jun 06 '22 at 08:04
  • I was afraid that this would become a discussion about statistics, but too late. In addition to the two issues that I pointed out in my comments below, all three answers so far contain the following statistical error (which I thus point out here instead of writing a separate comment for each answer): they apparently assume (explicitly or implicitly) that if, for each question, the expected value of guessing is 0 points, then guessing the answer to any number of questions will not change the expected value of your overall points. [...] – Jochen Glueck Jun 06 '22 at 08:17
  • [...] For most exams this is not correct since you typically cannot get a negative total number of points (for the entire exam, and probably also for the multiple choice part of the exam only). [...] – Jochen Glueck Jun 06 '22 at 08:17
  • [...] Example: Say the multiple choice part of the exam consists of 10 questions, each with just two possible answers, with precisely one of the two being correct. Assume you get 1 point for a correct answer, and -1 for a wrong answer, but at the lower end the score for the multiple choice part is cut off at 0. If you know the answer to 3 questions for sure and have no glue about the other 7 questions, guessing at all remaining seven answers will increase your expected points to a number strictly larger than 3. – Jochen Glueck Jun 06 '22 at 08:18
  • @RobbieGoodwin: Hmm, admittedly, I would not consider the statistics to be completely trivial; please see my previous comments. – Jochen Glueck Jun 06 '22 at 08:32
  • @xLeitix: Your previous two comments convinced me that I was wrong to vote to close. I thus voted to reopen the question. – Jochen Glueck Jun 06 '22 at 08:33
  • 2
    I would add that the answers all assume that every wrong answer is given the same number of negative points. I have set quizzes where one wrong answer is more reasonable and gets fewer negative points assigned to it. I see no reason a quiz cannot result in negative overall points (see the show QI), other than it would be cruel and unusual. – Terry Loring Jun 06 '22 at 20:46
  • @JochenGlueck I would be quite interested in your answer (not ironically, I would really like to read your analysis and learn something). – xLeitix Jun 07 '22 at 09:12
  • @xLeitix: In case that the question gets reopened, I'll try to write an answer (but the conclusion will probably be very boring, along the lines of "any serious optimization is not feasible in practice"). – Jochen Glueck Jun 07 '22 at 11:30

3 Answers3

9

Before answering a question, calculate your expected value. If it's positive then you should guess, otherwise you should not.

Expected value is how much you expect to get from the guess. It's calculated by multiplying p(correct) * points for correct + p(incorrect) * points for incorrect. So for example:

  1. If you are 100% sure an answer is correct, then p(correct) = 1 and p(incorrect) = 0. You should always guess.
  2. If you are 25% sure an answer is correct (as is typical for a 4-option MCQ for which you have no idea), then you need the points for getting the right answer to be at least 3x that of the penalty of getting the incorrect answer. If there's no drawback for guessing wrong then the penalty is 0, and you should always guess.
  3. Similarly, if you are able to eliminate two of the options then p(correct) becomes 0.5. You need the points for getting the right answer to be at least equal that of the penalty of getting the incorrect answer.

I once took an exam which awarded 8 points for getting the right answer and deducted 3 points for an incorrect guess. With these numbers, a 50-50 question is still worth a guess. You'll have to do the calculations for your specific case (and also estimate p(correct)).

Allure
  • 127,528
  • 50
  • 325
  • 493
  • 7
    I'm sorry for repeating myself, but as in the other answer the claim that one should guess if the expected value of the points that one gets from guessing is positive, ignores two important aspects: (1) the target function that one wants to optimise does typically not depend linearly on the number of points, and (2) one has to take the standard deviation into account in addition to the expected value. E.g., in financial applications of decision theory, it is typically assumed that higher standard deviation is only acceptable if it is rewarded by a sufficient increase of the expected value. – Jochen Glueck Jun 05 '22 at 16:07
  • @JochenGlueck My suspicion is that we envision different types of MC questions, or different circumstances. In the exams I had in mind a typical student would in fact have a fairly linear target function - if they get it right they get a slightly better grade, if they get it wrong a slightly worse one. Of course extreme cases are imaginable, like if you think you have just enough points to pass without answering, then you should probably just skip guessing. But for a student somewhere in the middle of the scale I still feel EV is a fairly good heuristic. – xLeitix Jun 06 '22 at 08:00
  • @xLeitix: Yes, the strongest non-linearity certainly occurs close to the pass mark of the exam. But there might but other important non-linearities: When I was I student, it often happened that you got the best possible grade with, say, only 85% of the points (due to a version of "grading-on the curve", combined with low average scores in the course). So between, say, 80% and 100% of the points the grade depended very non-linearly on the number of points. (But admittedly that's all quite theoretical, because in practice you typically have insufficient information for any serious optimisation). – Jochen Glueck Jun 06 '22 at 08:08
  • @xLeitix: Oh, for the sake of completeness I should add that my previous comment does, of course, not contradict your claim that "for a student somewhere in the middle of the scale I still feel EV is a fairly good heuristic". – Jochen Glueck Jun 06 '22 at 08:37
6

As Daniel correctly observes, whether or not guessing is a good idea depends on the scoring system, more precisely whether you lose points for wrong answers or not. By doing a (very simple) assessment of your expected (point) value we can distinguish the following cases:

Case 1 - No point loss for wrong answers This is the simple (and probably most common) case - a wrong answer is 0 points, same as not answering the question. It should be fairly obvious that in this exam design there is no reason not to guess. Quite frankly, if there is no punishment for guessing, you should always guess, independently of what else is going and or how much of an idea you have what the answer might be. You can only win.

Case 2 - Wrong answers lose some points An alternative exam design is one where correct answers give a point, and wrong answers lead to some amount of point loss (either a full point or a fraction thereof). Not answering of course neither gains nor loses a point. Now it depends on how sure you are and how much you will lose if you are wrong:

Case 2.1 - Full point loss for wrong answers In the most extreme (realistic) case, a right answer gives as many points as a wrong answer loses. Statistically speaking, it's ideal to guess here if and only if the probability to be right is >50%. So if you are quite, but not 100%, sure you should still take the answer. If you can narrow the options down to 2 but you don't know which it is, it's statistically speaking a wash if you pick one or skip the question. If you think three or more options could be right it's ideal not to pick any of them and move on.

Case 2.2 - Fractional point loss for wrong answers If a wrong answer loses a fraction of the points that a right answer wins, you need to generalise a bit from Case 2.1. It's easiest to analyse this case by considering how many of the possible options you can exclude, and then assess your expected value of randomly guessing between the remaining options. For example, if there are four options, and you can exclude two of them, you really only have two plausible options left. When a wrong answer loses less points than a correct answer gives you, you should still guess. If you cannot exclude anything, you should probably not guess (under reasonable assumption about the test design).

Some further considerations:

If multiple answers are possible: If multiple answers can be correct (e.g., answers A and C are correct, but not B and D) the same principles from above still apply, but the space of possible choices explodes to the product of all possible answers. If there is no point loss for wrong answers it's still always ideal to take a guess, but if a wrong answer loses points it becomes much more unlikely that guessing is ever really the right choice unless you are pretty sure you know what the right answer(s) are.

If "none of the above" is a possible option: Statistically speaking "none of the above" is just one more option to consider in your analysis. That said, in practice this is a bit tricky since you will never be able to exclude this option unless you already know the correct answer for sure (and then you don't need this analysis). So if "none of the above" is a possible answer and wrong answers lose full points, you should only guess if you are pretty sure that one specific answer is right.

As a note for educators looking at this - you may read this answer and (correctly) conclude that the best way to disincentivize guessing is to subtract a full point for wrong answers, combined with having "none of the above" as a possible choice. While this is undoubtedly true, consider what the impact on students is who are not guessing, but who simply made an honest mistake. Essentially, this leads to the awkward situation where a partial but slightly incorrect solution is worse than knowing nothing and skipping a question entirely. If you intend your exam to be a measure of student knowledge, that's not what you want.

For exams with calculations, I found "none of the above" to be a particularly nefarious exam design - this prevents students who know in principle how to do the calculation from sanity-checking for simple calculation errors (did I arrive at one of the possible solutions?). In my opinion, the only reason to pick this design is if you want as many people as possible to do poorly, independently of how much they actually know.

xLeitix
  • 135,037
  • 46
  • 333
  • 493
  • 1
    I have two objections concerning your analysis in case 2.1: the conclusion about the optimal strategy is only correct if (1) the function that you would like to optimize depends linearly on the number of points you reach, and (2) you are indifferent with respect to risk. Assumption (1) is not correct in many exams, for instance since there is often a cut-off of points below which you fail the exam. Assumption (2) is only correct for students who are not risk averse, i.e. who are willing to accept higher standard deviation of the outcome even if it does not result in higher expected outcome. – Jochen Glueck Jun 05 '22 at 09:24
  • IIRC many standardized exams used in admissions use the 2.2 method: partial deduction for wrong answers to account for pure guessing, but not for "educated" guessing. – Buffy Jun 05 '22 at 13:22
  • 1
    I find this interesting in an even wider context - if, say, your exam is 50% of the grade for your course. A student going in with a failing grade may be incentivised to, say, pick more answers at random than a student trying to preserve a good grade - the failing student benefits from a larger standard deviation, if the dice come up in their favour! – lupe Jun 05 '22 at 17:40
  • 1
    @lupe: For related reasons, both of these existing answers are wrong. The objective is not to maximize the expected score but to minimize the likelihood of a bad grade. So if I know I have answered exactly enough to get an A, but I am uncertain about the answer to one other question, why on earth would I answer it just to reduce my likelihood of keeping an A?!?! – user21820 Jun 05 '22 at 18:34
  • 2
    @user21820 In my experience, it's vanishingly rare to have that level of precision when estimating your own performance during an exam. Unless you are a perfect student, there are always questions you're not sure about. Exams also often have short answer or essay portions, about which you basically never have better than an educated guess. In my academic career, this made it very rare to be able to strategize about individual questions beyond naively maximizing expected value of points received. – A_S00 Jun 05 '22 at 21:41
  • 1
    @A_S00: Firstly, what I said is true regardless of whether a student can or cannot achieve the objective. Secondly, if you are a perfect student, then you do not need to guess. Thirdly, when I was a student I frequently got close or equal to 100% and it was the case that I could sometimes know all the answers except to a few questions. Fourthly, it doesn't matter whether you have an open-ended portion, if you are a risk-averse person and want to keep likelihood of bad outcomes below some threshold. (I didn't mention this fourth point earlier because some people simply can't grasp it.) – user21820 Jun 06 '22 at 07:29
2

If you are guessing completely at random from n choices, then you have 1/n chance of guessing correctly (if completely at random). That means with probability 1/n you get 1 point, and with probability (n-1)/n you get 0 points. Your expected value therefore is 1/n.

Typically, it's desired that a student with 0 knowledge should get a 0% on an exam, rather than 25% or 20% from guessing. This can be done by penalizing incorrect answers by 1/(n-1): Then your expected penalized value becomes 1*1/n - 1/(n-1) * (n-1)/n = 1/n - 1/n = 0 which is what we want. Another way to think of it is for say 5 choices, on average 1 out 5 guesses will make you gain a point, the other 4 will collectively make you lose a point to balance it out, therefore the penalty is 0.25 per wrong guess.

As soon as you can eliminate even one of the incorrect choices, the expected value of course goes above 0. For example, in the same 5 choice test, if you manage to narrow every question down to 4 possible answers and guess them all, you will end up guessing 1 correcly for every 3 wrong - you get 1 point for the correct, -0.75 for the incorrect, and come out with 0.25 points. The more you narrow it down the better it gets. But then again, if you were able to eliminate even one choice, then it cannot be said that you came in with 0 knowledge. You clearly knew at least enough to eliminate the one choice.

Jessica
  • 759
  • 1
  • 5