Within-subject vs. between-subject design
- Harmony Jankowski
Within-subject design vs. between-subject design
Terracotta supports both within-subject and between-subject experiment designs. In a between-subject design, half the participants get treatment “A” and half get treatment “B.” In a within-subject design, participants get multiple conditions at different times during the experiment - it might have half the participants get treatments "A-then-B," and the other half get "B-then-A".
There are a couple big advantages to within-subject designs (also called crossover designs):
When each participant receives all treatments, there is less concern about the research causing inequities. For example, imagine that version B turns out to be better than version A for student learning. In a between-subject design, this might cause students who received version A to have worse outcomes, but in a within-subject design, all students received the same treatments, just staggered in time.
Because each participant in a within-subject design receives both treatments, each participant is almost like their own mini-experiment. In effect, the research study has more statistical power to infer differences between treatments A and B.
Balancing the experiment
In order to have a balanced design, a research study needs to include as many experimental treatments as there are experimental conditions. Terracotta eliminates some of the complexities of organizing these sorts of experiments by organizing things around the idea of an “exposure set.”
Imagine that you’re making an experiment with four conditions: 1, 2, 3, and 4. Because there are four conditions, there will be four exposure sets. Each one has a different arrangement of students to conditions. Note in the screenshots below that you can navigate between exposure sets using the tabs, and that the Design summary at the bottom of the page shows which experimental condition is assigned to each group.
Notice in the screenshot for Exposure Set 2 that the group assignments have changed, so each experimental group receives a new condition.
Within any exposure set, the researcher can add multiple assignments, and Terracotta will handle the hard work of ensuring that the right students see the right versions of each assignment (according to the student's treatment condition in that exposure set).
Finally, when mapping outcomes for a within-subject design, the researcher only needs to specify one outcome score for each exposure set, for each student (although it’s possible to add more). For more on Outcomes, please see this article: https://terracotta-education.atlassian.net/wiki/spaces/TC/pages/221806593 .