Am I fooling myself?

Want a real-life example of how difficult it is to make inference and why science and the scientific method are essential if we value the advancement of knowledge? (Borrowed from an Instagram post last week.)

For the sake of illustration, I’ll use an example from my recent archery practice, though this point is not about archery, but about inference and the process of hypothesis generation, and testing.

I closed my archery practice with the following set:

• 10 rounds of 6 shots at 50 yards onto a 65 cm target.
• Scoring is as follows: 5 points in the white, 4 in the inner black ring, and 3 in the outer black ring. If you hit the “x” (i.e., the inner white ring) you get the 5 points, but also an “x” to break ties.

Take a look at the result of my first round. My green scored 15, 3x and my pink scored 13, 0x:

I thought, “Man, there must be something wrong with the pink fletched arrows!”

Point #1—we always look for patterns

Pattern-seeking behavior help animals, and us humans, survive. (It also helps us go through the day without having to make predictions about every moment of every day.) If it seems like a subtle movement detected in the bushes sometimes translates to a predator behind those bushes, our first thought is not to run an experiment and see if we get eaten.

Then I said, “Wait, that’s too small a sample, let me do an ‘experiment’ and see how it shakes out after all 10 rounds.” (A fair effort, but really the only thing I could do that was experimental was randomly select the arrows from my quiver, which I did, to be sure it wasn’t an artifact of something like fatigue or shot sequence.)

After a few rounds, it really seemed like the green fletched arrows were flying better, but in my mind, I said, “No way…I know they are the same!”

Point #2—still biased

Here, I’m having an internal argument, and it actually may be somewhat competitive. That is, my bias is that there’s no difference between the arrows. There may be a Rosenthal effect at play here: I may subconsciously behave in a way that somehow tips the scales in favor of my preconceptions.

“I fletched them myself (i.e., I was the one who attached the green and pink fletchings to the shafts of the arrows) and there is no way color alone could account for this difference.”

Point #3—this is a generally accepted and good idea in science, to maintain a strong null hypothesis, but if it impacts your ability to be objective (see below) it’s a problem

Then I found myself ‘trying harder’ with the pink arrows to narrow the gap between them!

Point #4—without blinding, it is damn near impossible to eliminate experimenter bias

This is a variant of the Hawthorne effect. It’s also sometimes referred to as the Avis effect, after the rental car company’s slogan, “We try harder.” The aforementioned Rosenthal effect is named after a researcher’s study that supported the hypothesis that reality can be positively or negatively influenced by the expectations of others. Rosenthal argued that biased expectancies could affect reality and create self-fulfilling prophecies.

In this case, I’m the observer and the subject in the experiment. That is a dangerous combination. I’m trying harder in the direction of confirming my preconceptions. I should be asking: does my narrative fit the evidence? However, I’m the one producing the evidence, and I may consciously or subconsciously manipulate the evidence. I may be creating a self-fulfilling prophecy.

In the end, the results were  (far from stellar, by the way, even at this distance):

• Green shot 138, 8x
• Pink shot 133, 6x

So, all things considered, I have no idea if there is a difference between the green and pink fletching. But that’s not the point of the story.

We’re emotional creatures that look for signals in a sea of mostly noise. We like to see things as we wish them to be and we sometimes consciously or unconsciously act in a manner to coax those things to our wishes. Without a Ulysses pact, fooling ourselves appears to be the default state of the human condition. Without a framework, in this case, the scientific method, we’re far too likely to see what we want to see rather than the alternative.

This story serves to remind me that we are not wired to think scientifically. Full stop. It is the quintessential human flaw. But scientific thinking is a skill to be practiced and improved upon. I’m better at it today than I was 5 years ago, and I’m confident 5 years from now I’ll be saying the same things. So can you.

– Peter