Commentary: Personality Psychology Has a Serious Problem (and So Do Many Other Areas of Psychology)

In the previous issue of P, we published a feature article by Brent Roberts, entitled “Personality Psychology Has a Serious Problem (and So Do Many Other Areas of Psychology),” which addressed the issue of scientific integrity in personality research. We invited reader responses, two of which are reprinted below.

PsychFileDrawer.org

Alex Holcombe and Hal Pashler

We agree wholeheartedly with your diagnosis of a major problem in publication practices in psychology. As you explain, any solution has to include a reduction in the systematic bias against publishing non-replications that now exists. Such a bias seems to be present in the editorial practices of all of the major psychology journals.  In addition, discussions with colleagues lead us to believe that investigators themselves tend to lose interest in a phenomenon when they fail to replicate a result, partly because they know that publishing negative findings is likely to be difficult and writing the manuscript time-consuming.  Given these biases, it seems inevitable that our literature and even our textbooks are filling with fascinating "findings" that lack validity.

To help address this problem, together with colleagues we have created a website that allows psychology researchers to post brief notices of replication attempts (whether successful or unsuccessful). In designing the website, PsychFileDrawer.org, we put a premium on making the submission process quick and easy, in recognition of the fact that the incentives for posting are modest. As of 5 January 2013, there are 22 postings reporting the results of replication attempts, and we hope readers of P will post studies from their file drawers and provide us with feedback regarding the site.

Introducing “Rigorous Papers”

John Rauthmann

I wholeheartedly agree with Brent Roberts’ suggestions in “Personality Psychology Has a Serious Problem (and So Do Many Other Areas of Psychology),” where he proposed that journals should actively seek work that likely reports reproducible, robust, and generalizable findings. Replication of findings is the cornerstone of “hard science” that is aimed at accumulating a solid knowledge base. Also fundamental is a priori and exactly planning how a study will be conducted, thus avoiding some of the 10 problematic practices in current psychological research outlined by Brent Roberts. I thus see another path that journals could easily implement. Each journal issue could have a special section with so-called “Rigorous Papers” (RPs; maybe 1-3 papers per issue) that have undergone a, well, rigorous procedure prior to publishing. RPs in each issue would undergo six steps, from submitting a detailed research proposal to conducting a study exactly as specified in the proposal to publishing an RP paper based on a consequent path of research practice that was concretized a priori.

Step 1: Research Proposal

Authors prepare a detailed research proposal (including, but not limited to, theoretical background, significance of research, questions and hypotheses, time plan and/or tree/flow chart of processes within the study, estimates of power and effect size , sample considerations, measures and instruments, planned data analytical procedures, anticipated findings, significance and merits of research, limitations and possible problems during conduct, alternative paths during conduct of the study that may need to be taken, ethical issues, literature references, and any material to be used in the study1) of usually no more than 25 pages in length (excluding cover letter, title pages, tables, and supplemental material or any other appendixes). The authors send the research proposal to the journal editor, seeking approval to conduct a carefully designed study (or set of related studies) for a planned future project that will be published in the journal. If the proposal is rejected, the research may still be conducted but cannot be accepted as an RP.

Step 2: Review of the Research Proposal

If the proposal is evaluated positively by the editor, then it goes out to 2-3 peer-reviewers who will also evaluate it on several criteria (including, but not limited to, scientific merit, internal and external validity, soundness of methodology, likelihood of being conducted in the way presented, etc.). A positive evaluation of the proposal means an “interim acceptance,” that is, the research will very likely be published regardless of its results—under the conditions that (a) the study is conducted exactly as described in the proposal2 and (b) the resultant paper is written in a style that conforms with APA and journal guidelines. The authors may need to adjust or modify their proposal according to the editor’s and reviewers’ suggestions. In this case, the decision is a “revise and resubmit,” and the revised proposal may go out for review again (if the editor decides so). It is paramount that the research proposal is maximally sound.

Step 3: Conduct the Research EXACTLY as Planned

If the editorial decision is positive, the authors may conduct their proposed research exactly as outlined. Modifications are not encouraged, but if they (must) occur, they have to be justified.

Step 4: Paper Write-Up

The authors write their paper on the study regardless of the (significance or catchiness of) findings, as delineated in the research proposal, and submit it to the journal.

Step 5: Final Decision on the Paper

Usually, the editor would now be able to make an editorial decision. If they feel that they cannot do so, the paper may be sent out for review once more. This will most likely depend on the quality of the write-up (e.g., discussion and interpretation) and whether or not modifications to the original research plan were made.

Step 6: Publication

Once the editor has reached a final, positive decision on the manuscript, it can be published. The online supplemental material will have to include (a) the original research proposal (i.e., the revised version accepted by the editor), (b) all material used in the study, (c) all syntax for data analyses (if not already included in the original research proposal), and (d) the raw data matrix.

Benefits of “Rigorous Papers”

  1. The research presented in RPs would conform more closely to scientific principles than research presented in most other papers because everything has been specified a priori.3
  2. Despite at times posing more time, work, and effort, RPs would likely become more prestigious. RPs would be limited only to a certain amount per journal issue and their findings could be deemed more solid than other findings (as they are not based on any “bad science habits”). Authors, journals, and publishers (as well as, ultimately, all other people not in the field of academic psychology) would benefit.
  3. Aiming for an RP can hone one’s scientific skills and also be motivating. After all: The research is almost accepted for publication once the research proposal has been positively evaluated. This also puts off the pressure of “having to find significant/catchy results” because it is not a requirement for publication. The RP system actually rewards and values sound psychological science and good habits in conducting research.
  4. Many of the problematic practices in psychological science outlined by Brent Roberts can be avoided, such as data churning, peeking, HARKing, data topiary, and betting against the house. The essence of an RP makes all of these practices obsolete. To add even more: Should the editor and reviewers wish it, the proposed research must also include one or even more replication studies in order to be publishable.
  5. The RP system could be easily implemented into personality journals and would add to their prestige.

All of this is just a keen suggestion. But we can (and should) start somewhere. I can see how my proposal may be radical and even more costly at some points (especially for editors). However, sound and exact research should be worth it. The approaches outlined by Brent Roberts and me could guarantee more (if not full) disclosure and transparency in what we do in our research— which is just what we need to reinstate our standing as psychological scientists.

1 I would even go as far as to include a variable list of all variables to be sampled (names and labels, explanations, rationale for and concrete use in the study, etc.) and also syntax for all data analyses proposed at the end. This maximum disclosure and transparency gives editors and reviewers much more ground for approving or suggesting changes/amendments of a given research proposal.

2 There may be unexpected issues and problems during the conduct of the research which make it necessary to depart from the original research plan. In this case, any modifications must be carefully protocolled, explained/justified in detail, and clearly denoted as such in the future manuscript. The resultant paper may be rejected at the discretion of the editor if it departs too far from the original research proposal. Alternatively, the editor may send the paper out again for review.

3 Of course, there are some things that can hardly or not at all be specified beforehand, and sometimes it will be necessary to pursue different approaches within the research process. Research may be flexible and dynamic, but the flow of one’s research can very well be planned with painstaking detail, exact timing and cut-off criteria, and mapped out on tree-like decision charts (with exact criteria for pursuing each branch). Not all research might be equally well suited for RPs, but a majority of papers would be.