Experimental Design, Repeated Measures and Order Effects
Note: It appears you have not yet completed our interactive activity, The Envelope and the Vintage Sedan
. If you work through this - it'll take about 5 minutes - before reading the analysis below, things will make a lot more sense. At the end of the activity, you'll find a link that will return you to this page.
Between-subjects and Within-subjects
Let's suppose you're interested in the effects of head trauma on memory recall,
and you devise an experiment that compares the performance of people across two
different conditions: the first, a standard test of recall; the second, a standard
test of recall taken after having been hit over the head with a large mallet. (Don't
try this at home.)
You then have a choice to make: do you get the participants in your experiment -
your experimental subjects - to do the test in both conditions (allowing you to
compare the results against each other); or do you split them into two groups, and
get one group to do the test in the frst condition, and the second, to do the test
in the second condition (allowing you to compare the results of the two groups)?
If you opt for the first choice, then your experiment is a within-subjects
or repeated measures design; if you opt for the second, it's a between-subjects
Order (Sometimes Called Carryover) Effects
A within-subjects design tests each subject under all conditions. This has
one big disadvantage, namely, that participation in one condition may affect performance
in another. So, for example, somebody might do better in the second memory test,
despite having been hit over the head with a mallet, simply because they'd had
the chance to practice the first time around. This is the problem of what are known
as "order effects" or "carryover effects". It really is a big
deal, as the data below will show (well, hopefully - if it doesn't - oops!)
Order Effects and the Envelope
In the first part of The Envelope and the Vintage Sedan,
people are asked to make a moral judgment about three scenarios - the Envelope,
the Sedan and the Thesis. Two of these scenarios - Sedan and Thesis - are designed
to elicit the response that there is bad behavior on display; the other scenario
- Envelope - is designed to elicit the response that there isn't.
Everybody who completes The Envelope and the Vintage Sedan is required
to make a judgment about all three scenarios, which makes it a within-subjects design
(where the dependent variable is the moral judgment). This means it is susceptible
to order effects. To put it simply, it seems likely that the judgments people make
will be affected by the order in which the scenarios are presented.
In order to control for order effects, the experiment randomizes the order in which
the three scenarios appear, and then assigns a code to the responses. This makes
it possible to determine whether order effects are in fact operating. The chart
above shows the percentage of people who responded "I behaved badly" to the Envelope
condition - where the expectation is that most people won't make this judgement
- depending on whether they saw the Envelope scenario first, second or third.
If there are no order effects, then the percentage will be more or less the
same in each case. However, if there are order effects, then one would expect the
"Behaved Badly" percentage to increase with each case. This is because
both the Sedan and the Thesis scenarios will tend to prime people to respond
that there is bad behavior in the Envelope case (and also because people
might be aiming for consistency). In particular, one would expect a significant
increase in the Behaved Badly score where the Envelope scenario is the last to appear
(i.e., in third place).
As of writing, there is not enough data to know whether this will turn out to be
the case. However, if it does, then not only will it show the impact of order effects,
it will also raise questions about the nature of moral judgments. If moral judgments
are susceptible to order effects, then clearly, sometimes at least, they rest on
shaky grounds, even if moral agents themselves are not aware of it.