1

As an example, imagine you're running the faculty of computer science and decide which books to use for the next semester.

  • You pick book A for all students -> no IRB review needed
  • You pick book B for all students -> no IRB needed
  • You pick book A for half the classes and book B for the other half (due to professor preference, for example) -> no IRB needed
  • You decide to run an experiment where you want to see whether book A works better than book B -> IRB needed
  • You retroactively decide to analyze whether book A was better than book B -> simplified version of IRB is needed for retrospective data analysis

What is so special about something being an experiment?

JonathanReez
  • 1,550
  • 1
  • 10
  • 24
  • 2
    I believe your prem8se is wrong, at least in the US. It's not research unless you plan to disseminate the results. – Scott Seidman Jan 03 '24 at 17:01
  • 4
    I suspect what triggers the need for an IRB is not the fact that it's an experiment, but the fact that it's human subjects research, with your students being the subjects. – Daniel Hatton Jan 03 '24 at 17:08
  • 1
    Non-experiment research is also regulated. "Experiment" is not the relevant feature, research is – Bryan Krause Jan 03 '24 at 17:10
  • @DanielHatton sure but scenario 3 and scenario 4 involve the exact same set of actions. The only difference is whether or not you label the actions as an "experiment". – JonathanReez Jan 03 '24 at 17:20
  • @JonathanReez Scenario 3 seems a bit off to me. If you're acknowledging two suitable books, why wouldn't you just tell the students the bibliographic details of both and let individual students choose? – Daniel Hatton Jan 03 '24 at 17:52
  • @DanielHatton you might have 4 different profs teaching undergrad Algebra, 2 profs prefer Book A, 2 profs prefer book B, so you split the student groups into two. – JonathanReez Jan 03 '24 at 17:58
  • 5
    @JonathanReez - and now you have a useless "experiment" since you can't separate out the profs from the books. – Jon Custer Jan 03 '24 at 18:05
  • @JonathanReez I repeat: why wouldn't you just tell the students the bibliographic details of both and let individual students choose? We want students to learn, so as long as both books cover the course material comprehensively and accurately, what matters is which book's presentation makes most sense to the student, not which book's presentation makes most sense to the prof. – Daniel Hatton Jan 03 '24 at 18:05
  • 1
    Existing methods exist for Textbook evaluation. Doing legwork and a quick search to help inform this question would help to improve the question. For example, I discovered http://www.project2061.org/publications/textbook/articles/approach.htm for the US that covers the example in the question: How to evaluate textbooks. Also, many papers exist on using teaching evaluations for research. Lastly, the OP could improve their post by citing sources for their definitions or "research" and "experiment" – Richard Erickson Jan 03 '24 at 21:11

5 Answers5

8

Research creates a conflict of interest between the researcher and the research subjects: researchers benefit personally in their profession for performing and disseminating research. Research protections exist because people abused their research subjects for these personal gains (money, fame, etc), they weren't invented to create paperwork.

Seemingly innocuous research is still regulated or at least subject to institutional review because it's not sensible to have the researchers who benefit from their research make their own decisions about what they think the risks are or what paperwork they want to do.

Many things are regulated in one context but not another; research is not particularly special in that sense (for example: if friends come to your house, you can make them dinner and serve them beer without any sort of permit; if you want to collect money for making dinner and serving beer you typically need a permit for each and are subject to many additional regulations). For this to work, it's necessary to define what is and isn't regulated somehow. The standard I'm most familiar with is that research is intended to contribute to generalizable knowledge. It would not be practical or reasonable to regulate an instructor's ordinary choice of textbook as if it's research any more than an ordinary medical procedure.

Bryan Krause
  • 114,149
  • 27
  • 331
  • 420
  • +1 but note that this also applies to medical procedures all the time. Hospital A decides to use procedure KB-1 for treating condition XYZ. Hospital B decides to use KB-2. If there's no "research", no one bats an eye assuming KB-1 and KB-2 are seen as equal options in the medical community. But as soon as you start doing "research", you get a ton of extra work on top. – JonathanReez Jan 03 '24 at 20:18
  • 3
    @JonathanReez Yes that's exactly what I'm saying. We don't regulate all decisions as research, we regulate research. – Bryan Krause Jan 03 '24 at 20:25
  • 1
    @JonathanReez We don't merely “assume” KB-1 and KB-2 are equivalent. Patients expect and physicians have a duty not to carelessly chose a treatment that's inferior in every way. If the physician choses a treatment for any other reason, this expectation breaks down. In fact, as an experimenter, it's very tempting to chose treatments that you expect to be different to maximise statistical power and make it easier to publish. That's one reason why intent matters (and disingenuously omitting to “label” it as an experiment – as you put it in your question – doesn't really change anything). – Relaxed Jan 03 '24 at 20:29
  • @Relaxed right but my question is why "intent" is supposed to matter. Either KB-1 and KB-2 are safe and accepted within the medical community or they're not. Why you do KB-1 vs KB-2 shouldn't matter, assuming all else is equal. – JonathanReez Jan 03 '24 at 20:35
  • 2
    @JonathanReez But not all else is equal, which I discuss in this answer. – Bryan Krause Jan 03 '24 at 20:40
  • @JonathanReez How could intent not matter? The intent of the physician is precisely what the expectation I was referring to is about. Either you chose KB-1 because you believe it's the absolute best option and you're not even considering any other factor or you chose it because you're looking for an interesting condition in an experimental design. It cannot be both. – Relaxed Jan 03 '24 at 22:40
  • Incidentally, that's a very contrived example, it makes no sense to run an experiment if you already know KB-1 and KB-2 to be perfectly equivalent. That would be unethical, if only because it's wasting resources. Which brings us back to the point I was alluding to earlier: If your intent is to research something, your mindset is completely different, you have a very strong incentive to include at least one treatment you believe to be inferior in some way and you have a conflict of interest when deciding whether both are safe and accepted. – Relaxed Jan 03 '24 at 22:45
  • @Relaxed often times KB-1 and KB-2 efficacy rates have large error bars, especially once you try to understand how they work for a particular subset of patients. I.e. for ADHD one might use Ritalin, Adderall, Dexedrine, Evekeo, Desoxyn, Vyvanse, and a few other more exotic treatments. There's no consensus on which option is the "best" so different psychiatrists will use different options in a different order, even if you only look at the best ones from top schools. Lots of medical conditions are in the same boat. – JonathanReez Jan 03 '24 at 23:05
  • 1
    @JonathanReez You're describing things for an IRB to consider when evaluating a research proposal. Having researchers decide their own work is low risk would be like having the police investigate themselves for possible wrongdoing in policing. – Bryan Krause Jan 04 '24 at 01:36
  • 1
    @JonathanReez You're the one who contrived an example with two equivalent treatments but ultimately that's completely irrelevant, the point was about how you approach the treatment decision for an individual patient, whether you are only trying to do what's best for them in their specific situation or if other considerations figure in your choice. – Relaxed Jan 04 '24 at 10:42
  • You wrote “either KB-1 and KB-2 are safe and accepted within the medical community or they're not“ but as you just pointed out, it's usually not that easy, especially for any treatment you would care to research so starting from this assumption isn't very useful. – Relaxed Jan 04 '24 at 10:48
3

To be specific to your scenarios:

In the US, the controlling department is Health and Human Services, and their requirements are easily found at HHS.gov. From "§46.104 Exempt research." subpart (d) we find

(d) Except as described in paragraph (a) of this section, the following categories of human subjects research are exempt from this policy:

(1) Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students’ opportunity to learn required educational content or the assessment of educators who provide instruction. This includes most research on regular and special education instructional strategies, and research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods.

So, in the US at least, no IRB review is required in any of your scenarios above.

Jon Custer
  • 12,775
  • 7
  • 43
  • 54
  • Good answer. As an aside, I've worked with academics who wanted to use course evaluations for research and their specific universities still required IRB approval. Likely specific universities erring on the side of being risk adverse given your references. – Richard Erickson Jan 03 '24 at 21:05
  • 1
    @RichardErickson - that is a grey area, because you are now (potentially) collecting information from individuals ("interaction" or "intervention" in the definitions section). I would bet the IRB would go very smoothly, particularly if the course evaluation questions were the standard set used by the institution already. Asking specific, course-related questions could be seen as different. – Jon Custer Jan 03 '24 at 22:25
  • 1
    The professor I worked with wanted to evaluate her teaching methods using research and you're correct. She got IRB approval prior to teaching and had no problems. Sometime real life human interactions go smoother than hypothetical internet questions :) – Richard Erickson Jan 04 '24 at 01:43
1

What is so special about something being an experiment?

The answer is location specific for exact policy, but largely depends upon what you want to do with the results. For example, Loyola notes

Do student course evaluations of faculty need to be reviewed by the IRB?

Student course evaluations of faculty do not need to be reviewed by the IRB, unless the evaluation results will be used for research purposes or will be published or disseminated.

Basically, if you want to do (including the important step of publishing) research, you need IRB approval. This example would directly apply to your example as well.

Richard Erickson
  • 15,076
  • 3
  • 43
  • 70
  • But how could something be perfectly acceptable if done on its own but then not acceptable without IRB if tagged as "research"? – JonathanReez Jan 03 '24 at 17:21
  • 2
    What makes you think it's not acceptable. It is perfectly acceptable, it just undergoes one layer of review to certify it is compliant. – R1NaNo Jan 03 '24 at 17:33
  • 3
    @JonathanReez -- This is because the IRB exists -- in the US, by Federal Mandate -- to review the treatment of human subjects research, not teaching. Students, just by being students and participating in coursework, are not "research subjects". There are certainly ways to abuse your students while teaching, but they are beyond the purview if the IRB (though many of them are covered by different laws, federal or otherwise) – Scott Seidman Jan 03 '24 at 17:47
  • 3
    A necessary element of research is 'generalizable knowledge.' If your purpose is to develop a conclusion that can be widely generalized to the pedagogic process then you are seeking generalizable knowledge and you must have IRB review. If you just want to choose a better book for yourself, you don't. – David Smith Jan 03 '24 at 18:06
  • 1
    Anecdotally, some psychologist at my school was doing a study surrounding course evaluations, and a different evaluation was used for a subset of courses. The questions weren't shown to faculty. Some of the students were not comfortable with the questions. Students and faculty in the involved courses got an apology from the Dean and Provost, stating that the IRB had retroactively determined that IRB review was required to roll this out (I can't remember if they also said informed consent was required). I know this, as I was teaching one of the courses involved, and got the apology letter. – Scott Seidman Jan 03 '24 at 18:14
  • 1
    @ScottSeidman yep - but if the same questions were shown without the intent of doing "research", its likely that no one would take the complaints seriously or apologize for anything. – JonathanReez Jan 03 '24 at 20:22
1

While I don't believe you posit research here, research subject are protected, often with IRB review being a part of that process, because horrible injustices have been done to subjects in the guise of research.

When dealing with students, in particular, some of the main risks have to do with coercion and voluntariness. Also, in the US, subjects have a right to NOT PARTICIPATE in research, and the IRB would look to make sure you have a fair path around the research for student who opt not to participate.

Also, students who are research subjects also need to have their privacy protected, and they need to be protected from social wrongs.

But again, I disagree that the activity in your description, in and of itself, constitutes research under US PHS definitions.

Scott Seidman
  • 31,120
  • 4
  • 52
  • 121
  • If I publish a paper comparing student performance with Book A vs performance with Book B, can I do so without IRB approval? – JonathanReez Jan 03 '24 at 17:22
  • 1
    @JonathanReez No, you cannot -- but it's the publishing (or plan to publish, anyway) that makes the difference, not the assessing textbook efficacy for your own teaching purposes. – Scott Seidman Jan 03 '24 at 17:36
1

If you are conducting/disseminating research related to human subjects, you will need IRB approval. This is a safe-guard. Just switching books and keeping note internally is part of departmental pedagogy and something those students signed up for, being the subject of research is not (unless you allowed them to sign a disclaimer, which itself would require IRB approval).

It's not the experiment that's the trigger, it's the human subjects.

In a case like this your IRB approval will just sail through. Someone looking at efficacy of pedagogical material is not high on the list of the type of ethical violations universities/agencies are trying to curtail.

EDIT: I see from your other comments that you are looking at the requirement for IRB approval as meaning that the work or topic is not acceptable. It is perfectly acceptable. It just requires one layer of overview/review to ensure compliance. That's all.

Not the best analogy, but on the fly I see it somewhat similar to when you order something from a foreign vendor, your credit card may need an authentication code to make sure there's no fraud. It doesn't mean you can't buy it, just one layer of protection for everyone involved.

R1NaNo
  • 3,977
  • 1
  • 6
  • 20
  • I disagree with your characterization, at least in the US. It's not just the Human Subjects, it's the subjects and the RESEARCH, and research is "a systematic investigation designed to develop or contribute to generalizable knowledge relating broadly to public health, including behavioral and social-sciences research." (from https://www.ncbi.nlm.nih.gov/books/NBK22941/) -- no desire to contribute to the generalizable knowledge (i.e., no publication, or plan to do so) == not research. – Scott Seidman Jan 03 '24 at 17:42
  • @ScottSeidman my response was implicit that it was research related to Human Subjects which triggers an institutional review. That seemed to be the crux of the question from the OP, ie what is triggering IRB, not necessarily all of the nuances of IRBs. Perhaps I misunderstood. – R1NaNo Jan 03 '24 at 17:46
  • @ScottSeidman That definition, though, would suggest that participant observation/action research/qualitative-methodology research projects, where generalizability is not among the goals, would not require IRB approval, but I bet they do. – Daniel Hatton Jan 03 '24 at 17:55
  • 2
    @DanielHatton I suppose different IRBs can be more stringent, but the definition comes right from the Code for Federal Regulations. I can tell you that I've gone to my IRB about 20 times with questions like this, and every time, the question I get back is "do you intend to publish?", and if the answer is "no" their response is "you don't need IRB approval because it's not research". – Scott Seidman Jan 03 '24 at 18:02
  • 1
    I don't think this study would necessarily "sail through" an IRB. There is no described way for a student to opt out of the study. There are classes of studies that don't require this, but in this case, there is conceivable harm to subjects (e.g., if students with textbook A do great in the course, but students with textbook B all fail, then the latter group has been harmed). – Scott Seidman Jan 03 '24 at 18:09
  • @ScottSeidman yep, IMO the answers so far downplay the experiment / research aspect. If Book B sucks but there's no experiment, the students will just be told "its a you problem" if they complain. But if Book B sucks and its an experiment, then all of a sudden it becomes a problem. – JonathanReez Jan 03 '24 at 20:15
  • @JonathanReez -- no, they're not necessarily told it's a "you problem" if book B sucks. It doesn't always reach a level of law. Sometimes it does. If different classes of students are being given different educational materials, it may be a Title 9 or some other discrimination issue. If privacy in education is compromised, it may be a FERPA issue. If it's a research thing, then it may be an IRB issue. – Scott Seidman Jan 03 '24 at 20:55
  • @ScottSeidman in my college the leadership would just shrug and say that Book A (along with any other book on the planet) was widely accessible to everyone on the course and that would be the end of it. – JonathanReez Jan 03 '24 at 20:57