Yes, X Causes Y

Holden wants to know: Does X cause Y? Whenever we try to figure it out, there’s an observational study, followed by maybe a couple more interesting studies that still don’t replicate, or have other horrendous drawbacks. To summarize the grand total of human knowledge of the subject: “it’s super unclear”.

It’s satire, but he’s also serious. Claiming inspiration from (among other sources), Scott Alexander’s “Much More Than You Wanted to Know” series, Holden concludes “the bottom line usually has the kind of frustrating ambiguity seen in this conclusion.”

This is too pessimistic.

First, on the meta level, “does X cause Y” debates tend to occur right on the boundary line of epistemic confidence. So by nature the discussions are designed to be highly contentious and uncertain. We no longer ask “does X cause Y” questions about matters superseded by modern physics (“Does phlogiston cause combustion? No.”), nor about matters settled by empirical data (“Do germs cause disease? Yes.”). But this progress goes underappreciated.

For people living on the cutting edge of science, the important questions will always have ambiguous answers.

Second, object level, Holden is overstating the ambiguity of Scott’s evidence reviews.

  • In Melatonin, Scott pretty definitively concludes that Melatonin is an effective hypnotic, and the right dosage is 0.3mg.
  • In Autism, Scott concludes with ~100% confidence that “genes that increase risk of autism are disproportionately also genes that increase intelligence”.
  • In College Admissions, Scott concludes that “There is strong evidence for more competition for places at top colleges now than 10, 50, or 100 years ago.”

(I’m cherry picking a bit, but the point stands. In a good chunk of cases, we can be reasonably confident that X causes Y. I’m taking Scott’s confidence at face value here, but I think that’s reasonable, at least with regards to responding to Holden. In another post, Holden agrees that Scott’s predictions have a good track record.)

Third, most of the time, we care more about making the right decisions than about having the right epistemics. By which I mean, it’s okay to not be certain, as long as the expected value works out. For example, in Face Masks, Scott remains uncertain about the efficacy, but points out that for high-risk situations, the annoyance of wearing a mask is probably a price worth paying for even a small likelihood of reducing Covid risk.

Finally, it’s working taking a look at some really quick high level heuristics.

So look. Of course it sounds bad that 2/3rd of psychology studies don’t replicate. But that means that 1/3rd of them do!

It would be nice to know which third is which, but still, that’s not nothing.

Admittedly, it would be very bad if:

  • 1/3rd of studies replicate
  • 1/3rd show no effect
  • 1/3rd demonstrate the opposite effect

Though there is a bit of this in nutrition (“does X cause or prevent cancer? Maybe!”), it’s not universal. For example: I’m not 100% sure how well face masks prevent Covid, but I am pretty darn confident they don’t cause Covid.

This asymmetry matters a lot. And it matters for Holden’s more serious questions too. GiveWell isn’t totally sure if the parasite-killing drugs improve test scores, but they probably don’t make them worse.

So start with a reasonable prior (50% of all studies are wrong, or 66%, or 40% or whatever), update based on some object-level evidence, and multiply out to get an expected value estimate. As far as I can tell, this is immensely simplified, but basically what GiveWell does.

I’m not being critical of Holden. He knows all of this far far better than I do.

I’m just saying as a reader, when you encounter stuff like this, don’t throw your hands up in the air and conclude that since nutrition science is bad, we should all just eat pizza and whiskey for every meal. Don’t make the mistake of falling into epistemic nihilism.

To conclude:

  • Our discussions will tend to revolve around contentious issues, creating the illusion of an epistemic crisis.
  • Even on contentious issues, literature reviews are not useless.
  • The point is not to achieve certainty, the point is good decision making.
  • It might be possible to know some things some of the time.

Footnotes
[1] He clarifies this does not mean up to 40% of doctor’s opinions are wrong. Doctors are not relying on a single study. And again, I would point out that the bulk of doctor’s opinions are totally mundane and non-controversial.