Tuesday, May 30, 2017

There is a lot of media attention these days to a supposed crisis in the replication of scientific studies. However, I don't agree that there is such a replication "crisis". Biological experimentation involves a very high degree of complexity, with many variables that are extremely difficult to control. For example, even the same inbred mouse strains, or tissue culture cell lines, can actually be quite divergent between different laboratories. It is very difficult for one lab to assess the number of statistical degrees of freedom in its own work. Thus it is to be expected that some, many even many, results which are reported reliably on experiments performed in good conscience, will fail to replicate among labs. That is part of the cost of doing truly novel research. Just based on an anecdotal sense of the kinds of work being described as "non-replicable", much of it is either in the social sciences, where controlling variables is even harder, or in clinical trials on human cohorts where again it is challenging to control for environmental factors. In contrast, in my own field of human medical genetics, results tend to be highly replicable, as we are usually looking at the effects of severe high penetrance mutations. Different mutations in the same gene typically generate very similar, though not identical phenotypes even in very different populations. A quick survey of the Human Gene Mutation Database corroborates this observation.

That said, it is true that scientists are under way too much pressure to generate positive results, especially for very expensive programs like clinical trials or large scale population genetic studies. Moreover, the hypercompetitive state of grant funding these days encourages hype and excessive optimism over sober analysis. This is not the fault of the NIH, this is the result of decades of government underfunding of science measured as a proportion of national GDP.

No comments:

Post a Comment