Update on Editorial Policies at the Journal of Research in Psychology
Richard Lucas
Michigan State University
Last year, in response to concerns about the crisis in confidence in psychological research, the editors of the Journal of Research in Personality instituted new policies designed to improve the quality of research we publish. First, we announced a new emphasis on power and precision, noting that authors would be required to discuss the rationale for their chosen sample size and that severely under-powered studies would be likely to be rejected without review. Second, we asked authors to explicitly commit to sharing their data with those who wished to verify their findings (though we do not require authors to actually post their data when their articles are accepted, and we do allow for exceptions where data sharing is simply not possible). Finally, we instituted a new policy whereby attempts to replicate articles published in JRP within the past five years would be subject to an abbreviated review process focused solely on whether the methods of the new replication study were adequate.
Although one year is a relatively short period of time in which to evaluate the impact of these policies, there is some evidence that they are making a difference. For instance, just this past month, Chris Fraley and Simine Vazire published a paper proposing a new index of journal quality, the N-Pact factor (Fraley & Vazire, 2014, PLOSOne). This index assesses the median sample size of articles published in different journals, with the goal of determining whether journals vary in this important indicator of research quality. Although in personality psychology, we like to think that we typically publish high-powered studies, their index showed otherwise, at least for the years 2006 to 2010. Personality journals did publish studies with larger sample sizes than journals in social psychology, but even the Ns for personality journals were quite low (for instance, the five-year N-Pact for JRP, the second highest journal, was 129).
In light of the changes in policies that we instituted, I decided to calculate the N-Pact Factor for JRP in the year 2014 (one of the nice things about Fraley and Vazire’s index is that it is quite easy to compute). I was happy to see that the typical sample size has increased pretty dramatically, to a respectable 303. It is important to note that this increase did not come through greater reliance on large student samples participating in simple survey studies. Indeed, these types of studies are almost always rejected without review at JRP. Instead, we are publishing quite diverse research that uses large internet samples, pre-existing data, smart-phone apps that can collect intensive data from large samples, and large-scale collaborations across multiple research labs (in addition to the more typical student-sample study). In other words, many of the fears that have been expressed about increasing requirements for sample size in psychological literature are not supported by the trends in articles published at JRP over the past years. I think that this is an encouraging sign and hopefully other journals in personality and social psychology will follow our lead.
It is also possible to look back at our decision to consider papers that report direct replications of studies previously published in JRP. One concern that I often heard when discussing this policy change was that the journal would be overrun with uninteresting replications. The concern is that these replications would squeeze out more exciting investigations of novel phenomena. This has certainly not occurred, at least in the first year in which this policy was in place. JRP received exactly two qualifying submissions under the replication-report policy. To be fair, it may take some time for researchers to react to the call and to plan their replication studies; a year may simply not be enough time to evaluate its effects. However, our experience with this policy can help guide expectations for other journal editors who are considering allowing and encouraging direct replications. In addition, I encourage more researchers to consider conducting high-quality replications of the research studies we publish.
Beyond these changes, JRP continues to pursue novel ways to improve the quality, transparency, and reproducibility of the research we publish. For instance, in early 2015, we will publish a special issue on using R for personality research. The goal of this special issue is not to introduce readers to R or even to provide guidance on how to use the program. Rather, it is designed to show off new on-line functionality that allows authors to upload their R code and data to the Journal’s website. This code can then be presented in a special viewer that appears alongside the relevant portions of the article; and depending on the nature of analyses, the code can even be run using data stored on the journal’s website. This innovative feature will increase the transparency of the analyses that are reported in the journal.
So far, the steps we have taken towards improving the quality of research published in the journal have, I think, been quite a success. However, more work is left to be done, and the other editors and I are open to more suggestions about further improvements that we could make or additional innovative initiatives that could address current concerns. In short, if you have ideas, please send them to me and we will certainly consider them.
Finally, I wanted to note that these policy changes could not be implemented without buy-in from the editors and reviewers who evaluate the work that is submitted to JRP. So also wanted to an extend my thanks both to the great team of editors we have, and the wonderful reviewers who donate their time to provide thoughtful comments on the articles we receive. And on that note, I did want to acknowledge both Phebe Cramer and Colin DeYoung, who are winding up their terms as Associate Editor for JRP this year. They have both contributed considerable time and effort to the journal over the years, and I wanted to thank them both for their excellent service.