Everyone of us is a Data Point - Part I
The important thing is knowing what to do about it
If you tend to drift towards the non-fiction books section of the bookstore of your preference, you will likely know that a fairly common technique used by authors is what follows.
The Technique
Author states a point they want to make.
Author mentions a study (usually a psychological study) that supports the point they’re making.
Author piles up anecdotes on top of the study.
Author moves on to another point.
The Problems
That sounds like a fairly reasonable approach to take, doesn’t it? Well, it surely does, but it presents at least a couple of problems for the reader in the way it’s usually carried out. I’ll call them the cherry picking problem and the Dark Side of the Moon problem.
The Cherry Picking Problem
In order to explain this problem I cannot think of a better way to do it than with an example. In his book Influence, Robert Cialdini uses the study by Ellen Langer titled The Mindlessness of Ostensibly Thoughtful Action: The Role of “Placebic” Information in Interpersonal Interaction usually called “the copy machine study” for reasons that will become evident, to explain how we have deeply ingrained automatic responses we cannot really control nor are usually aware of. In the book, Cialdini mentions Langer demonstrates his thesis, which he then explains: “A well-known principle of human behavior says that when we ask someone to do us a favor we will be more successful if we provide a reason. People simply like to have reasons for what they do.” He then proceeds to describe the experiment.
The experiment consisted in having a researcher approach a person waiting in line for a library copy machine with the goal of using it first. To do so the researcher asked one of three questions:
Excuse me, I have five pages. May I use the Xerox machine? [No reason]
Excuse me, I have five pages. May I use the Xerox machine because I’m in a rush? [Real reason]
Excuse me, I have five pages. May I use the Xerox machine because I have to make copies? [Useless reason - “placebic” information]
People complied with the request 60% of the time in version 1, the version without a reason. People complied with the request 94% of the time in version 2, with an actual reason. The kicker is that, in version 3, people complied 93% of the time.
The conclusion that Cialdini arrives at is “…so, too, did the word ‘because’ trigger an automatic compliance response from Langer’s subjects, even when they were given no subsequent reason to comply. Click, whirr!”
On the next paragraph, Cialdini says that while some of Langer’s additional findings show that in many situations people’s behavior does not work in such a mechanical way, it’s astonishing how often it does. And then follows with anecdotes and stories about several automatic responses in the human and animal realms.
What Cialdini condenses in two lines when he says that Langer’s study show that humans in many situations do not show automatic responses is totally forgotten by the time you are reading about how the killer firefly of the genus Photuris mimics the mating behavior of the fireflies of genus Photinus to feast on them.
But if we paid attention and go read the actual full study by Langer we see that, if we change the requested favor from 5 pages to 20 pages, the compliance rates tell a totally different story. The “no reason” and “placebic information” versions of the request have the same exact compliance rate at 24% and the real reason almost doubles the compliance rate to 42%.
This is the Cherry Picking problem. Authors tend to pick only the most favorable part of studies to push forward their point. And this is totally reasonable, why would someone with something to say select the part that doesn’t confirm what you want to say over the part that does? But we, as readers, might be missing important information if we don’t verify!
The Dark Side of the Moon problem
Now, for the second problem. If I hadn’t found the actual Langer’s study on James Clear’s website, I wouldn’t have ever noticed the second part of it. And like many people I would have kept parroting away the 93% of compliance rate you get when using the word because, with no strings attached.
On this specific case we are lucky because the Langer’s study actually explores a bit more in depth the issue in question compared to what’s usually cited in popular media. And so just reading the study makes us see a more nuanced perspective than “say because,” and then wonder why people are not complying. Ever tried “Give me a thousand dollars because I want them."?
But there are two harder versions of this problem. The first one is actually getting access to the studies, many books don’t even have references, or if they do, the papers are behind a paywall (RIP, Aaron Swartz), or they require lots of effort to just understand and validate the claims. Getting the time to fact check and analyze each study is ultimately impractical.
And the second version of the problem is that many psychological studies are said to not be reproducible, and they don’t (and likely cannot) control for many variables, and so one could argue their applicability in the general case is fairly low even though many books and articles cite them as gospel.
Let’s take Langer’s study once more. That study was performed over 120 adults that tried to use the copy machine at the Graduate Center of the City University of New York. Is it reasonable to think that 120 graduate students living in New York in the 1970s completely model the global behavior of people in queues? Is the behavior of people in queues applicable to just about any favor? Is the behavior of people in queues constant across cultures? And what about the behavior regarding favors, is it constant? If not, does it vary significantly, and how? What about the word - how can we translate it? Can we use a synonym in English? What about the researchers, were they attractive? Some were smelly, perhaps? Did anyone have bad breath? Or did they have an exceptionally magnetic personality? I certainly don’t know the answer to any single one of these questions, but I’m fairly sure at least some of those variables could potentially have an impact in the outcomes you would get out of a replication of Langer’s study.
Conclusion
So, what are we left with? Just with the certainty that you should take upon yourself the responsibility of recognizing and understanding the technique, and then follow up on the parts that are of interest to you to see if they hold water for your particular circumstances. Also, experiment! Things that may not hold water scientifically might actually work for you.
Do me a favor: next time you read something of interest to you that cites studies and backs them up with anecdotes, would you please verify the claims because I need to make copies?
There is a third problem I’ve dubbed the Where’s Wally? problem I plan to talk about soon and will reveal the title’s meaning. Stay tuned!