Bsrhrki

Nature article on reproducibility in science.

According to that article, a (surprisingly) large number of experiments aren't reproducible, or at least there have failed attempted reproductions. In one of the figures, it's said that 70% of scientists in physics & engineering have failed to reproduce someone else's results, and 50% have failed to reproduce their own.

Clearly, if something cannot be reproduced, its veracity is called into question. Also clearly, because there's only one particle accelerator with the power of the LHC in the world, we aren't able to independently reproduce LHC results; in fact naively one might expect there's a 50% chance that if we built another LHC, it will not reach the same results. How, then, do we know that the LHC results (such as the discovery of the Higgs boson) are robust? Or do we not know the LHC results are robust, and are effectively proceeding on faith that they are?

That's a really great question. The 'replication crisis' is that many effects in social sciences couldn't be reproduced. There are many factors leading to this phenomenon, including

• Weak standards of evidence, e.g., $2sigma$ evidence required to demonstrate an effect


• Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.


• Poor training in statistical methods.

I'm not entirely sure about the exact efforts that the LHC experiments are making to ensure that they don't suffer the same problems. But let me say some things that should at least put your mind at ease:

• Particle physics typically requires a high-standard of evidence for discoveries ($5sigma$)


• The results at the LHC are already replicated!
  
  
  • There are several detectors placed around the LHC ring. Two them, called ATLAS and CMS, are general purpose detectors for Standard Model and Beyond the Standard Model physics. Both of them found compelling evidence for the Higgs boson. They are in principle completely independent (though in practice staff switch experiments, experimentalists from each experiment presumably talk and socialize with each other etc, so possibly a very small dependence in analysis choices etc).
  
  
  • The Tevatron, a similar collider experiment in the USA operating at lower-energies, found direct evidence for the Higgs boson.
  
  
  • The Higgs boson was observed several datasets collected at the LHC
  
  


• The LHC (typically) publishes findings regardless of their statistical significance, i.e., significant results are not selectively reported.


• The LHC teams are guided by statistical committees, hopefully ensuring good practice


• The LHC is in principle committed to open data, which means a lot of the data should at some point become public. This is one recommendation for helping the crisis in social sciences.


• Typical training for experimentalists at the LHC includes basic statistics (although in my experience LHC experimentalits are still subject to the same traps and misinterpretations as everyone else).


• All members (thousands) of the experimental teams are authors on the papers. The incentive for bad practices such as $p$-hacking is presumably slightly lowered, as you cannot 'discover' a new effect and publish it only under your own name, and have improved job/grant prospects. This incentive might be a factor in the replication crisis in social sciences.


• All papers are subject to internal review (which I understand to be quite rigorous) as well as external review by a journal


• LHC analyses are often (I'm not sure who plans or decides this) blinded. This means that the experimentalists cannot tweak the analyses depending on the result. They are 'blind' to the result, make their choices, then unblind it only at the end. This should help prevent $p$-hacking
  


• LHC analysis typically (though not always) report a global $p$-value, which has beeen corrected for multiple comparisons (the look-elsewhere effect).

If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.

In addition to innisfree's excellent list, there's another fundamental difference between modern physics experiments and human-based experiments: While the latter tend to be exploratory, physics experiments these days are primarily confirmatory.

In particular, we have theories (sometimes competing theories) that model our idea of how physics works. These theories make specific predictions about the kinds of results we ought to see, and physics experiments are generally then built to discriminate between the various predictions, which are typically either of the form "this effect happens or doesn't" (jet quenching, dispersion in the speed of light due to quantized space), or "this variable has some value" (the mass of the Higgs boson). We use computer simulations to produce pictures of what the results would look like in the different cases and then match the experimental data with those models; nearly always, what we get matches one or the other of the suspected cases. In this way, experimental results in physics are rarely shocking.

Occasionally, however, what we see is something really unexpected, such as the time OPERA seemed to have observed faster-than-light motion—or, for that matter, Rutherford's gold-foil experiment. In these cases, priority tends to go toward reproducing the effect if possible and explaining what's going on (which usually tends to be an error of some sort, such as the miswired cable in OPERA, but does sometimes reveal something totally new, which then tends to become the subject of intense research itself until the new effect is understood well enough to start making models of it again).

The paper seems to be a statistical analysis of opinions, and in no way is rigorous enough to raise a question about the LHC. It is statistics about undisclosed statistics.

Here is a simpler example for statistics of failures: Take an Olympics athlete. How many failures before breaking the record? Is the record not broken because there may have been a thousand failures before breaking it?

What about the hundreds of athletes who try to reproduce and get a better record? Should they not try?

The statistics of failed experiments is similar: There is a goal (actually thousands of goals depending on the physics discipline), and a number of trials to reach the goal, though the olympics record analogy should not be taken too far, only to point out the difficulty of combining statistics from a large number of sets. In physics there may be wrong assumptions, blind alleys, logical errors... that may contribute to the failure of reproducibility. The confidence level from statistical and systematic errors are used to define the robustness of a measurement.

We know the LHC results are robust because there are two major and many smaller experiments trying for the same goals. The reason there are two experiments is so that systematic errors in one will not give spurious results. We trust that the measurement statistics that give the end results are correct, as we trust for the record breaking run that the measured times and distances are correct.

(And LHC is not an experiment. It is where experiments can be carried out depending on the efforts and ingenuity of researchers, it is the field where the Olympics takes place.)

The robustness of scientific results depends on the specific experimental measurements, not on integrating over all disparate experiments ever made. Bad use of statistics. For statistics of statistics, i.e. the confidence level of the "failed experiments" have to be done rigorously and the paper is not doing that.

Another way to look at it: If there were no failures , would the experiments mean anything? They would be predictable by pen and paper.