Meta-research in environmental science and economics

Abstract

Selective reporting, the practice of publishing favorable results while omitting unfavorable ones, poses a significant challenge to the reliability of scientific research. Studies with statistically significant or positive outcomes are more likely to be published than those with null or negative results. Such bias distorts the overall body of scientific literature, potentially leading to a false understanding of phenomena. This can have far‑reaching implications in the fields of environmental science and economics since policies crafted without a comprehensive understanding of the available evidence may fail to address societal issues effectively or may even exacerbate them. In Chapter 2 and 3, this dissertation introduces a new method used to detect and quantify the extent of selective reporting. To illustrate the practicality and relevance of the new approach, two large data sets collected from published and unpublished meta‑analyses in the fields of economics and environmental science are presented. Moreover, we use a negative binomial regression model to explore potential driving factors associated with selective reporting. In economics, we based our analysis on 70,399 effect sizes collectedfrom 192 meta‑analyses. We empirically estimate the extent of selective reporting by comparing the distribution of observed p‑values with the counterfactual distribution of p‑values generated under the assumption of no biases. Our study shows that about 58–72% fewer significant p‑values should have been published in the absence of any biases. Subsample analysis suggests that the extent of selective reporting is reduced in research fields that use experimental designs, analyze microeconomics research questions, and have at least some adequately powered studies. A possible explanation for this might be researchers are less likely to feel pressure to engage in questionable practices like p‑hacking to achieve statistical significance when a study is adequately powered to detect an effect. To quantify the extent of selective reporting in environmental science, we analyzed over 60,000 statistical tests gathered from 705 meta‑analyses. By employing the new method, we find that 30–53% fewer significant p‑values should have been published in the absence of any biases in the research and publication process. Moreover, our study indicates that the median statistical power in environmental science is only 8–13%, and only 9% of tests are adequately powered at the conventional threshold of 80% or more. This raises concern over the reliability of published findings in the field. Exploratory regressions suggest that the extent of selective reporting decreases with increasing statistical power. The greater the power of the study, the lower the likelihood of Type II errors (false‑negatives) and, therefore, the greater the chance of finding true effects. As a result, researchers will be less inclined to p‑hack to obtain significant results. We also employ Bayesian model‑averaging to identify the presence or absence of selective reporting and obtain the bias‑corrected average to evaluate its impact in four scientific disciplines, namely, medicine, psychology, economics, and environmental science. By analyzing a large data set from more than 68,000 meta‑analyses, we find that economics meta‑analyses are most affected by selective reporting, followed by environmental science and psychology. In contrast, meta‑analyses in medicine are the least affected. Finally, five tests used to detect selective reporting due to the omission of control variables are evaluated (caliper test, discontinuity test, monotonicity test (CS1), joint test of monotonicity and bounds (CS2B), and concavity test (LCM)). The tests are compared on the basis of their statistical power and ability to control Type I errors using exact, rounded, and derounded data sets simulated through the Monte Carlo procedure. We find that the discontinuity test applied to the z‑value distribution seems to perform well, followed by the CS2B test, regardless of the data type. Overall, this dissertation highlights the importance of addressing selective reporting in evidence synthesis, mainly by considering the statistical power of studies. Underpowered studies are less reliable and could jeopardize scientific progress and knowledge accumulation, which further undermines evidence‑based decision and policy making. In addition, this work may encourage further meta‑research studies and suggests fostering the culture of open science practices could help mitigate selective reporting in the synthesis of research across scientific disciplines.