Today’s lecture introduced a selection of data scales — refinements of the qualitative/quantitative distinction — and discussed in more detail issues of the application and misapplication of statistics.
Hypothesis testing can be a tremendously sensitive and powerful tool for discovering new science and identifying the connections between events. However, when used poorly it becomes misleading and unhelpful. The lecture covered a range of concerns about these risks: confusing correlation with causation; what p-values can tell us and what they can’t; and ways in which the arrival of big data and massive computation can amplify the challenges. There is also hope and success, though: in the discovery of robust results through meta-analysis; the active discussions around reproducibility and predictive power in scientific research; and the many projects to record trials, replicate results, and improve publication.
There is no lecture on Friday. Next week is the final teaching week, with one lecture reviewing course content and another working through some past exam questions.
I’m not setting specific reading, but below are a lot of references that I think interesting. Pick one of the topics that interests you and follow some of the links. And definitely read the comics.
Too long? Skip to the comics at the bottom.
- The Copernican Principle
- Correlation Does Not Imply Causation
- The Bad News About Significance Testing
- Working to Make Things Better
The Copernican Principle
- Wikipedia on the Copernican Principle in its appropriately cosmological sense.
- A Grim Reckoning. Article by J. Richard Gott III on his application of this to everything, up to and including the end of civilization. (May require EASE login; this is an article from the University’s paid subscription to New Scientist.)
- How to Predict Everything. Timothy Ferris, New Yorker, 12 July 1999. This is the article about Gott in which he discusses the performance of plays on Broadway. This is an online preview; the full article is only available to subscribers.
Correlation Does Not Imply Causation
- Tobacco Use and Academic Achievement. CDC: Centers for Disease Control.
- Health-Risk Behaviors and Academic Achievement. CDC with statistics relating academic achievement to just everything: physical exercise, alcohol, sex, watching television, and carrying a weapon. Who knew?
- John Aldrich. A Guide to R. A. Fisher. “I occasionally meet geneticists who ask me whether it is true that the great geneticist R. A. Fisher was also an important statistician.”
- Smoking Gun, Jean Marston, March 2008.
Letter to New Scientist recalling Fisher’s sceptical response to Doll and Bradford Hill’s work connecting smoking and lung cancer.
- Correlation and Causality: Polio. Short video describing the presumed link between polio and ice-cream.
- Kill or Cure? Documenting the Daily Mail’s achievements in ontological oncology.
- Spurious Correlations. Data dredging at its best.
The Bad News About Significance Testing
John P. A. Ioannidis. Why Most Published Research Findings Are False. PLoS Medicine 2005 2(8):e124.
David Trafimowa and Michael Marks. Editorial.
Basic and Applied Social Psychology 37(1)1–2, 2015.
This is the announcement that BASP will not accept papers using p-values, significance testing, or confidence intervals.
Allen Downey. Statistical Inference is only Mostly Wrong.
Probably Overthinking It, March 2015.
Blog article responding to BASP ban.
Ella Rhodes. Liberating, or Locking Away our Best Tools?
The Psychologist, 2015.
Commentary on the BASP ban, reporting reactions from a range of parties.
- Many scientific studies can’t be replicated. That’s a problem. Washington Post, August 2015.
- Over half of psychology studies fail reproducibility test Nature News, August 2015.
- Open Science Collaboration. Estimating the Reproducibility of Psychological Science. Science, 349(6521), 2015.
- Reproducibility: A tragedy of errors. Nature News, February 2016.
- Psychology’s reproducibility problem is exaggerated – say psychologists. Nature News, March 2016.
- Taking on Chemistry’s Reproducibility Problem. Chemistry World, March 2017.
Working to Make Things Better
- The Cochrane Collaboration. Global project systematically reviewing evidence to improve health decisions.
- Ben Goldacre. Bad Pharma: How Medicine is Broken, and How We Can Fix It. Fourth Estate, 2013.
- AllTrials. Campaign for all clinical trials, past and present, to be registered and their results reported.
- EBM DataLab. University of Oxford research group building tools to improve medicine with evidence and data.