A new study published in Nature Human Behaviour suggests that a tool similar to a stock market could help scientists identify research findings that are likely to hold up to scrutiny. The study, conducted by a group of social scientists from around the world, tested the ability of a "decision market" to predict which results from previously published social science experiments would be successfully replicated.
The decision market focused on 41 experiments that used participants recruited through Amazon Mechanical Turk (MTurk), an online platform that connects researchers with people willing to complete tasks for pay. All of the experiments were originally published in the Proceedings of the National Academy of Sciences (PNAS) between 2015 and 2018.
In the decision market, 162 social scientists bought and sold shares representing the likelihood that a given study's findings would replicate. The market was set up so that the studies whose shares ended up with the 12 highest and 12 lowest prices would be selected for replication. Two additional studies were selected randomly.
The results suggest that the decision market was able to successfully distinguish between replicable and non-replicable findings. The percentage of studies that produced statistically significant results in the same direction as the original study was 83% for the studies in the "top 12" price category. In contrast, only 33% of the studies with the lowest decision market prices successfully replicated. "Our study provides a 'proof of concept' that decision markets can be a useful tool for identifying likely true and likely false research findings," says Felix Holzmeister, author on the paper.
Overall, replications were successful for 54% of the 26 studies selected for replication. The average effect size observed in the replication studies was 45% of the average original effect size. This overall replication rate is comparable to those found in previous large-scale efforts to replicate findings from social science experiments.
The study's findings have important implications for the way that science is conducted. “Decision markets, or similar tools that rely on scientists' collective wisdom, could offer a principled mechanism to prioritize research efforts and resources, ultimately increasing the efficiency and reliability of scientific progress,” says Holzmeister.
