It can be shockingly hard just to understand three variables

April 19, 2021

In science (and when developing hypotheses more generally), it is very common to come across situations where a variable of interest (let’s call this the dependent variable, “Y”) is strongly correlated with at least two other variables (let’s call them “A” and “B”). Here are some examples: If you’re a psychology researcher investigating possible causes of depression (Y), you may have trouble disentangling the effects of poor sleep quality (A) and anxiety (B), both of which tend to be corre...

Testing Too Many Hypotheses

October 10, 2011

For each dataset, there is a limit to what we can use that dataset to test. Using the standard p-value based methods of science, the more hypotheses we check against the data, the more likely it will be that some of these checks give inaccurate conclusions. And this presents a big problem for the way science is practiced. Let's take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includ...