How to spot real expertise

April 23, 2024

Thanks go to Travis (from the Clearer Thinking team) for coauthoring this with me. This is a cross-post from Clearer Thinking. How can you tell who is a valid expert, and who is full of B.S.? On almost any topic of importance you can find a mix of valid experts (who are giving you reliable information) and false but confident-seeming "experts" (who are giving you misinformation). To make matters even more confusing, sometimes the fake experts even have very impressive credentials, and ev...

Did That Treatment Actually Help You?

April 15, 2023

A mistake we all make sometimes is attributing an improvement to whatever we've tried recently. For instance, we may get medicine from a doctor (or go to an acupuncturist) and feel better, so we conclude it worked. But did it actually work, or was it just chance? Here's a trick to help you decide: What matters (evidence-wise) is how likely that level of improvement would have been in that time period if the treatment works relative to how likely that improvement would have been if the treatm...

Demystifying p-values

December 31, 2022

There is a tremendous amount of confusion around what a p-value actually is, despite their widespread use in science. Here is my attempt to explain the concept of p-values concisely and clearly (including why they are useful and what often goes wrong with them). — What's a p-value? — If you run a study, then (all else equal, aside from rare edge cases) the lower the p-value, the lower the chance that your results are due to random chance or luck. More precisely: a p-value is the probab...

Importance Hacking: a major (yet rarely-discussed) problem in science

December 19, 2022

I first published this post on the Clearer Thinking blog on December 19, 2022, and first cross-posted it to this site on January 21, 2023. You have probably heard the phrase "replication crisis." It refers to the grim fact that, in a number of fields of science, when researchers attempt to replicate previously published studies, they fairly often don't get the same results. The magnitude of the problem depends on the field, but in psychology, it seems that something like 40% of studies i...

The many ways to make inferences

October 7, 2018

There are a LOT of ways to make inferences. Many more, I think, than is generally realized. And they all have their weaknesses. You can make inferences using… (1) Deduction: As a consequence of the definition of X and Y, if X then Y. X applies to this case. Therefore Y. “Plato is a man, and all men are mortal; therefore Plato is mortal.” “For any number that is an integer, there exists another integer greater than that number. 1,000,000 is an integer. So there exists an in...

A thought experiment about what you’d be truly capable of doing (if you had no choice)

April 26, 2018

Think of something you value that:A. multiple other people you know are capable of achieving, but thatB. you assume you would not be capable of achieving, even thoughC. you have never actually tried to do this thing well before. Now suppose, for a moment, that you have no choice but to do the thing. That is, everything you care about in the world will be destroyed if you do not achieve it in X months. Here, X could be 1 if it's a very small thing, or X could be 100 if it's a much larger thin...

Which Risks of Dying Are Worth Taking?

October 16, 2013

First, click here to figure out your chance of dying tomorrow. Is it worth taking a 1 in 100,000 chance of dying , in order to experience the novel thrill of sky diving? Is a 1 in 500,000 chance of death worth it to go bungee jumping? It's hard to know whether these risks are reasonable, because numbers like 100,000 or 500,000 feel so abstract to us. To think more clearly about these numbers, it helps to get our intuitions engaged. We can start by figuring out the daily risk of dying that...

Testing Too Many Hypotheses

October 10, 2011

For each dataset, there is a limit to what we can use that dataset to test. Using the standard p-value based methods of science, the more hypotheses we check against the data, the more likely it will be that some of these checks give inaccurate conclusions. And this presents a big problem for the way science is practiced. Let's take an example to illustrate the principle. Suppose that you have information about 1000 people selected at random from the U.S. adult population. Your dataset includ...

Surprised? Update your model.

September 24, 2011

In order to make predictions, your brain must have a model of reality. This model is necessarily much simpler than reality itself. To see why, imagine that you are about to drop a baseball from waist height. Your brain can't possibly know enough about the atoms composing that baseball and the air around it to simulate what will happen at the atomic level. And even if your brain did have accurate knowledge about the atoms, using information at such a fine a level of detail would be extremely comp...

(Almost) Everything is Uncertain

August 23, 2011

If you try to enumerate all of the things that you know with absolute, 100% certainty, you will find that the list is very small. You know that “something” exists. If you have mental experiences, then you know that “you” exist (though coming up with a reasonable definition for what “you” means can be remarkably tricky). If your mental experiences are varied, then you know that whatever exists creates varied mental experiences. With some cleverness, you may be able to add to this list ...