Monday, July 06, 2015

Bigger Is Not Always Better: Why Psychology Needs Smaller Studies

Psychology has been embroiled in a professional crisis as of late, and deservedly so. The research methods commonly used by psychologists, especially the statistical analyses used to analyze experimental data, have come under scrutiny—again, deservedly so. Although it is encouraging that so many people are becoming aware of the many problems evident in mainstream psychology research, one fundamental problem has received almost no attention. Namely, the focus on studying large groups of people has gone unquestioned. However, focusing on between-group comparisons is, in my estimation, THE problem, especially because those designs are exactly the kind that lead to the very statistical analyses at the center of psychology's professional crisis.

Smaller within-subject studies are much more appropriate for the kinds of questions most psychologists are asking. Smaller studies also tend to produce data that can be understood without complicated statistics. Moreover, and contrary to popular belief, within-subject studies actually tell us more about each subject studied and, therefore, provide us with more information about when and where the findings are likely to be useful.

Generally speaking, within-subject research allows each subject to be studied very intensively and over a prolonged period of time. Also, because there are fewer subjects, within-subject research often can be conducted under very controlled conditions, unlike studies of large groups of people, which have to rely on one or a few measures of each subject. Not to mention that those few measures typically are measures of what the subjects say the will do, rather than what they actually do. (Unfortunately, what we say we will do rarely matches what we actually do.)

Moreover, because large group designs focus on the average performance of a large group of subjects, they don't tell us about any real effects on any real person. The "average" effect doesn't exist, and an individual subject almost never responds like the mythical "average." This makes it very difficult to translate research findings into practice, because we will never meet the average person, we will only meet a real person.

What we get from studies of large groups of people typically amounts to very little information about any actual person or persons, few and possibly invalid measures of subject performance, and findings that might be "statistically significant" but have no practical implications in the real world.