(Original posted July 31, 2023)
As an early graduate student, most (perhaps all) of my research involved mapping some type of in-lab behavior (responding to cognitive probes, movement behavior, etc.) to self-report questionnaires of individual differences. I did studies on the Big Five, emotion regulation, aggression, sexuality… you name it. And for all of these domains, I used “off-the-shelf” questionnaires that had been published by other researchers. I thought “hey, if they’re peer-reviewed, well-validated, and everyone else uses them… they must be pretty good measurement tools!”
Over time, I grew as a researcher, and I came to really understand just how freaking awful a lot of published questionnaires are. It’s really quite amazing: many of the self-report instruments that are widely used within and outside of academic research are, ultimately, deeply flawed. Often, they do an incomplete job of measuring a construct, or they don’t measure what they claim they are measuring and… in the worst cases… they’re simply a reflection of the authors’ pet theory. And when the theory turns out to be wrong… well, the assessment tools made to go with that theory are problematic as well.
People — even in psychology — largely fail to appreciate the sheer difficulty of making a really, really good self-report questionnaire. Just like NLP and other complex/sophisticated methods, there are a lot of people out there who do it poorly, and don’t really understand that their resulting “product” is deeply flawed or problematic. As such, I’d strongly encourage everyone to read the attached paper, “Constructing Validity” by Clark & Watson (of PANAS fame). It’s a very accessible read, and even if you’re well-educated on questionnaires and personality assessment, there are a lot of insights here that are likely to provide some refined ideas on the massive challenges that have to be overcome to make a good self-report instrument. It’ll also, hopefully, help you to become an even more discerning scholar when it comes time to choose (or create your own) questionnaires for your own research down the road.
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319. https://doi.org/10.1037/1040-3590.7.3.309