Flexibility in the selection of patent counts: Implications for p-hacking and evidence-based policymaking

Abstract

This study analyzes how researchers' degrees of freedom in selecting patent counts influence econometrically estimated policy effects. Using the evaluation of solar energy policies as an example, we identify 51 strategies to select solar patents from the literature resulting in 306 different solar patent counts, considering six common quality levels of patents. We replicate two leading studies in this literature and re-estimate their econometric models using all of these patent counts. Our results demonstrate severe uncertainty regarding sizes and even signs of key policy effects, opening up the potential for p-hacking and posing a fundamental challenge for evidence-based policymaking. We recommend that more emphasis should be devoted to patent selection procedures, including careful sensitivity analysis regarding key assumptions, such as search strategy and patent quality level. More research is needed to develop common quality standards in working with patent data.