Bonferroni's Principle

In a nutshell: if you look harder than the quantity of data supports, you will find a pattern that "fits".

I have just heard of Bonferroni's Principle, under that label, but this is a key issue and well deserving of a name. The meaningfulness of an answer depends on the evidence that stands behind it. Bonferroni's Principle, that you will "find" something if you look hard enough, underscores conspiracy theories, much of social and economic empirical findings, and data-mining in general. We have all done this ourselves, and have seen others do this. Our brains are pattern matching and story telling machines - we will find a pattern and a story to fit what we see (barring complete lack of imagination). The question is if the pattern and story is legitimate or not: is it overly fragile and "overfit" to the data? Did we use a "training set" to come up with our story, and a "test set" to test it out? Conspiracy theories are notoriously brittle - they strain credulity, even in the case in question, and if you take the theory and look at any other piece of life it falls apart: the theory over fit some specific event, often with liberal doses of biased assumptions included, and makes no sense once you expose it to new, fresh data.

In large part science is about Bonferroni's Principle - we want to learn how to pick out legitimate patterns and stories from what we observe, for several reasons (we love stories and patterns, we find our lives improve if we have accurate and interesting ones, we can get social recognition from a group of likeminded searchers in our quest, etc.). The scientific method is all about good story telling - by asking (good) questions and (honestly) listening to the answers we get a good, and more true than not, story.

Our brains are amazing pattern finding and story weaving organs, and our job is to consciously test the stories and patterns that are being spun: we want to help our brains in its "brain storming" ways, by adding even more possibilities, and we need to help with critical analysis of the conclusions we jump to.

Examples:

A related sub-effect is the Barnum effect, which is Bonferroni's Principle applied to personality categories: people will agree with categories they are placed on, and believe that the categories are illuminating, for example astrological signs. Again, until recently, I did not know there was a name for this effect - but have seen it is strong in many people. I had a roommate who was convinced that horoscopes where accurate in describing people, as a test I read two signs and had him pick which one was his (e.g. his sign, and as random as one I could pick as another option, read in random order) and every week for a month I had him select his using a weekly 'scope he thought was good. Surprisingly he picked wrong every time (less than 10% odds, if his selection was random).

Townhall meeting: otherwise known as sample bias. Only those in the tails of the distribution of people effected by some decision will bother to show up. Most political decisions are structured so a small subset gain significant advantage, with a cost borne by everyone else. This small-many ratio ensures a small cost (so most likely not worth showing up) and a huge gain (so if you stand gain you will show up). So the tail of those to gain will be in the house, and perhaps some "anti" wackos. Result: "widespread support" that is underscored by only an embittered small scattering of crazies who oppose. The choice is clear...

Health science. Are supplements good for you? Who knows. We do know that those who take supplements tend to care deeply about health, so if we just compare these people against the average (i.e. unhealthy subjects) are we comparing difference in taking supplements, or differences in: exercise, positive attitude, smoking, drinking, etc. We are testing them all. So unless supplements have huge negative effects we will see a positive effect. Sadly, many studies have poor protocols such as using different sample groups - the study will find a difference between groups, but what causes this? In general, the scientific scam known as "significance" is at play here. Passing a p-test deems a hypothesis "significant", but Bonferroni's Principle tells us if we look hard enough we can always pass a p-test. Further, the use of "significant" hijacks our brains by misusing our filters. This is unfortunate, as the field of health is important and people make choices based on weak "science".

Lowdown:
- Quality must have quantity as support.
- Use simple tests to check your data (representative?) and story (legitimate?).
- Much of "science", particularly in social and health fields where biases and feelings and complexity runs high, falls prey to Bonferroni's Principle and is a scam. Cavet emporor.
- Having a label for something is nice - drop Bonferroni's Principle on people. Like an idiom a label for a developed idea allows rapid, deep, incisive discussion. In learning about science and human thinking Bonferroni's Principle should be one of the key points discussed.

The scientific method: seeing the forest, not the trees

Philosophers of science and social critics will sometimes talk about the scientific method, and argue about whether there actually is a scientific method or not. The definitions and arguments often lose sight of the big picture and instead focus on details: is the scientific method positive or negative, is it this or that, blah or gha? Often the debate boils down to whether specific tactics used by scientists is a strategy or not - the debate is stylistically like this: "Is a forest a pine tree?" with the yes side pointing out pine trees in a forest - "yes!", and the no side pointing out you can have a forest without a pine tree - "no!". Others chime in with, "No, a forest is - in fact - an oak tree". The debate confuses the basics. Of course a pine tree is not a forest. You can have a forest full of pine trees, or with no pine trees, and even if you have a pine forest you have a lot of other things in it that make up the forest. A forest is a forest. A tree is a tree. Specific trees are often a significant part of a forest. A forest is not a tree. "Proving" that this is so does not prove there is no forest, but simply that you have wasted your time and are confused.

The scientific method is simple: ask questions of reality, and honestly listen to the answer. We often couch this in overly clinical and un-illuminating terms - e.g. "hypothesis" (question) - which makes it sound like the method is limited to scientists, instead of being the fundamental approach to life one should take. How can you argue against asking questions coupled with honestly listening? This is simple, but not easy - it is hard to craft good questions (you have to come in with a lot of background information often and keep a fresh perspective), to realize when you accidentally ask the wrong question (good scientists pick up on this, and thus find new aspects of reality, poor scientists will minimize how the answer does not fit within their current understanding behind what they thought they asked - if you have ever programmed a computer you know that one often mean one thing but say another, if you ever had a real discussion with someone you find the same), and to honestly take in what you learn: it is hard to admit you were wrong, you wasted your time, your precious idea is not interesting or significant, etc.

The scientific method is a strategy, much of the bickering about the scientific method is merely linguistic confusion and confuses the tactical level with strategy. The tactics are important, but are not "the" scientific method, and should not - and are not - used in every situation. But the overriding strategy is both simple and consistent: ask questions. listen. honestly.

The scientific method is a general approach to life, one that is difficult yet is simple. You can learn it, and it will help you. In your particular field of work or hobbies or interests you can learn more specific tactics that are optimized for that aspect of reality, and gain a lot from your efforts in learning the tactics. Like many things that are simple the scientific method is subtle and allows you to hone your skills while confronting life. The scientific method is a lifestyle, a code of life to force you to grow and engage reality. The samurai had their code, scientists have theirs. Our master is reality, Truth - with a capital T. In the short term this code brings a lot of difficulty and pain - in the long run it brings great joy and beauty and improves you vastly.

The particular tactics are often related to objective reality, but the strategy is equally powerful when applied to subjective and social reality, and in fact if you do not apply it there you are short changing yourself and not living the scientific way.

A true scientist is simply someone in awe of the beauty of reality, and is so taken by the beauty that they ask questions and listen to learn more and more subtle, amazing, and mind blowing truths. Anyone can do this, and everyone who does will gain, enormously. You can either simply believe what you believe now, e.g. assume your limited experiences and biases are accurate reflections of the world and stay stuck in your limited current situation, or you can go out and explore and engage the crazy amazing thing we call life and get blown away every single day out just how insanely beautiful things are. Which approach do you think will work out better?