12

What are some common methods for assessing how well a exam works, once you have the results back? This is something of a generalization of How to measure entropy of exam results.

I've seen professors regress scores against some potentially interesting covariates (majors vs. non-majors, gender, etc.) and look at which questions were predictive of your overall grade, but I'd be interested to see if there's a particular battery of tests one should look at to see if an exam is "well written".

Community
  • 1
Fomite
  • 51,973
  • 5
  • 115
  • 229
  • https://en.wikipedia.org/wiki/Item_response_theory – vadim123 May 11 '14 at 03:19
  • Short answer or multiple choice? Are you interested in discrimination and validity (objectively "well written"), or whether the exam matches your course learning goals (philosophically "well-written")? – Adrienne May 11 '14 at 23:07
  • @Adrienne A mix of short answer and multiple choice, and discrimination and validity. – Fomite May 12 '14 at 15:27
  • Point biserial can be what you are looking for, see here: http://www.measuredprogress.org/learning-tools-statistical-analysis-the-point-biserial – Viktor Takacs Oct 27 '14 at 15:19

2 Answers2

5

The most common analyses I've seen are generated automatically by scantron grading software, and are thus only used for multiple choice questions. But a motivated person could perform the analyses themselves.

I've read a few articles on test analysis over the years, and this summary of a particular scantron software's results seems like a good overview of test analysis in general. A quick summary to improve searchability:

Item statistics:

Item difficulty - percentage of students answering correctly. Desirable = above chance.

Item discrimination - how much item correlates with test as a whole (students who did well on the exam get this right more than low-level students). Uses stat like Pearson Product Moment correlation. A "good" question has a score over 0.2.

Test statistics:

Reliability coefficient - A general measure of test length, breadth and its intercorrelations. Scores above 0.8 are considered excellent for a classroom test.

Some other interesting measurements are used more in standardized testing, rather than classroom testing. These definitions are taken from this overview.

Construct validity - does the exam actually measure the subject, or some other variable like reading skill. Generally uses a panel of "experts" or feedback by students.

Split-half reliability - measures whether different test items that purport to test the same concept produce similar results within a single exam.

Criterion-related validity - measures how well the new test correlates with a known exam, like an ETS field test or GRE subject test.

If you don't get a basic output from scantron software, is there a department on your campus that will analyze your exam for you?

Adrienne
  • 4,003
  • 2
  • 15
  • 19
  • Thanks very much for your answer - this is a hypothetical question, rather than something immediately relevant. I should be able to do all those analyses myself. – Fomite May 12 '14 at 18:16
  • "Item difficulty - percentage of students answering correctly. Desirable = above chance." Is this really true? Hard items should only be answered by good students, and might only have 5% success. Item response theory is designed to answer these sorts of questions. – Behacad May 14 '14 at 08:35
3

For multiple-choice exams, split-halves reliability is (for better or worse) a common measure of internal consistency. The basic idea is to divide the questions into halves (e.g., odd vs. even) and look correlate students' scores on the two halves.

Another common measure is the discrimination index. These indices giveyou information about how well an exam item "discriminates" between high performing and low performing students. This I find most useful when I've mis-coded the key. Students who otherwise do well, do particularly poorly on a question.

Both of these methods are most easily applied to exams with lots of questions (e.g., multiple choice), but they could probably be adapted for essay or short answer exams as well.

kmm
  • 1,716
  • 17
  • 20