> At a deeper level, the criticism fails to appreciate how people use citations as a measure for academic promotion. In most cases tenure committees care about aggregate statistics like total citations, h-index or i10-index. If a researcher publishes a work that receives hundreds citations for ten years and then fails to replicate, then it basically doesn't matter if the work stops receiving future citations. A retraction might matter. Reports of the failed replication might matter. But nobody is going to lose out on a promotion specifically because some random paper receives 8,000 citations in the first ten years and then zero citations after the failed replication.
This is his point, though. Not only are authors still getting tenure after failed replication, they're still getting citations! Citations that don't even mention the failure to replicate!
The fact that citations are used as a metric to get tenure is the problem. There are two solutions to that problem: Change the culture around citing things, or change the metrics people use. That is the whole point of the post.
> This is his point, though. Not only are authors still getting tenure after failed replication, they're still getting citations!
And his point is irrelevant. For two reasons that I've already explained multiple times now!
(1) As I've tried to point out (I've written three posts now to explain this, why are we still debating this!), there are plenty of valid reasons why non-replicating works might get cited. To prevent these works from being cited, you would need to fundamentally change citation practices in a manner that would harm researchers' ability to use citations for their intended purpose.
(2) As I also explained above: even if you somehow managed to force all other researchers to alter their citation practices, it almost certainly wouldn't matter for the purposes of promotion decisions anyway. For the purposes of promotion, the influence of each incremental citation drops off exponentially. After a few years of a work receiving citations, later incremental citations have at most a negligible influence on a researcher's record.
Unless replication failures happen extremely quickly, it doesn't really matter whether future researchers do or do not cite the failed work. The early citations will still exist and will vastly dominate later ones in promotion decisions. The only situation where citation-bans would help is one where citing authors could somehow intuit that a work would fail to replicate, prior to seeing an actual failed replication. (TFA claims this is easy. I think TFA is not credible.)
TL;DR: forcing the entire field to change the way they use citations is (1) harmful to researchers, and (2) despite that is unlikely to have any major benefit anyway, since later citations are not heavily weighted when promoting researchers, and replication attempts generally occur later in a work's citation lifetime.
This is his point, though. Not only are authors still getting tenure after failed replication, they're still getting citations! Citations that don't even mention the failure to replicate!
The fact that citations are used as a metric to get tenure is the problem. There are two solutions to that problem: Change the culture around citing things, or change the metrics people use. That is the whole point of the post.