Last week, my lab members and I had our first ‘scrum’ session. It was intended for us to share the challenges and failures we have encountered over the past week and to provide both emotional and practical support to each other. Vicky started by sharing her frustration over the inability to obtain statistically significant results for her study, despite her advisor’s assurance that the paradigm will definitely work.
The claim that an idea will “definitely work” gave me some unease. If one is certain what the results of an experiment will be, then why bother conducting the experiment? The point of experiments is to provide evidence for or against some hypothesis whose truth value is uncertain. There is no point for gathering information for hypotheses held with complete certainty, since one will not update their belief whatsoever, as Bayes tells us.
But testing highly certain ideas seems to have become the trending default in psychological research (at least the research I have been part of). Before an idea conjured up by a student is put to empirical examination, it has to pass an initial “quality check” by the student’s advisor, lest it is either too banal or too wild. The problem is, an advisor often evaluates a student’s idea based on its intuitive appeal, rather than objective standards.
What is it that makes an idea a good one, exactly? Our intuitions have been shown to be context-dependent and potentially unreliable. It seems unlikely to me that experts are completely free from these biases. So, why do we rely so much on our intuitions during the idea generation stage of research?
Here is one possibility. Intuitions come from life experiences, and one of the major aims of psychological research is to understand human life experiences at the level of individual minds. Thus, intuitive ideas are worth testing to the extent that they pertain to the basic human conditions. Good research shows the mechanisms underlying human experiences, whether these experiences are shared across the globe (like the Stroop effect, or fear conditioning), or observed only among tribes in the Pacific islands (be it some exotic moral codes or unique manners of emotion expression).
In this sense, all intuitions call for some degree of empirical examination, and ideas that are more intuitive require close attention all the more. Intuition is a first step that leads to fuller understanding of the phenomenon of interest. Whenever I presented psychological research ideas to my friends, someone would reply by saying “of course this is the case.” I believe they are missing the point. Indeed, many psychological findings seem so self-evident and trivial that they are regarded merely as commonsense knowledge in the disguise of fancy scientific jargons. We probably don’t need psychologists to tell us that experts know their field better than lay people do, or that people like their own group members better than outsiders, but psychologists does more than merely characterizing the statistical patterns of the world and confirming our intuitions about the human mind (though these efforts are indispensable, too). Psychologists aim to understand how the human mind works and why we think the way we do.
Mahzarin Banaji made this point elegantly in her colloquium talk earlier this week, quoting a poet, that we as humans are unfinished—we are yet to fully understand our own mind, however clearly it seems to have manifested itself already.
Thus, for a psychologist, even the most intuitively appealing idea should not be taken for granted. Every well-designed experiment, be it a seemingly banal replication of an old finding in the ‘80s, or a revolutionary piece that appears on Nature, is a step towards humans’ understanding of themselves. Nor should the failure to obtain significance discourage us. The insignificance should debunk our illusion of omnipotence and set us onto a journey towards more refined scientific knowledge. Science should be the goal in itself rather than a means for expressing deeply held convictions (though the former may very reasonably shape the latter).
Yi's Blog 