Author Archives: Sanjay Srivastava

Recently I posted a comment on a PLOS ONE article for the first time. As someone who had a decent chunk of his career before post-publication peer review came along — and has an even larger chunk of his career left with it around — it was an interesting experience.

It started when a colleague posted an article to his Facebook wall. I followed the link out of curiosity about the subject matter, but what immediately jumped out at me was that it was a 4-study sequence with pretty small samples. (See Uli Schimmack’s excellent article The ironic effect of significant results on the credibility of multiple-study articles [pdf] for why that’s noteworthy.) That got me curious about effect sizes and power, so I looked a little bit more closely and noticed some odd things. Like that different N’s were reported in the abstract and the method section. And when I calculated effect sizes from the reported means and SDs, some of them were enormous. Like Cohen’s d > 3.0 level of enormous. (If all this sounds a little hazy, it’s because my goal in this post is to talk about my experience of engaging in post-publication review — not to rehash the details. You can follow the links to the article and comments for those.)

In the old days of publishing, it wouldn’t have been clear what to do next. Continue reading →

Big news today is that Perspectives on Psychological Science is going to start publishing pre-registered replication reports. The inaugural editors will be Daniel Simons and Alex Holcombe, who have done the serious legwork to make this happen. See the official announcement and blog posts by Ed Yong and Melanie Tannenbaum. (Note: this isn’t the same as the earlier plan I wrote about for Psychological Science to publish replications, but it appears to be related.)

The gist of the plan is that after getting pre-approval from the editors (mainly to filter for important but as-yet unreplicated studies), proposers will create a detailed protocol. The original authors (and maybe other reviewers?) will have a chance to review the protocol. Once it has been approved, the proposer and other interested labs will run the study. Publication will be contingent on carrying out the protocol but not on the results. Collections of replications from multiple labs will be published together as final reports.

I think this is great news. In my ideal world published replications would be more routine, and wouldn’t require all the hoopla of prior review by original authors, multiple independent replications packaged together, etc. etc. Continue reading →

Happy holidays, readers! Today, of course, is the day to gather around the aluminum pole with friends and family and air your grievances. And here at The Hardest Science I am adding a holiday tradition of my own to help that process along. So sometime after the fifth “it must be nice not to have to work over your long break” but before someone pins the head of household so you can all go home, gather together all your non-academic loved ones and read this to them aloud:

What the heck is research anyway?

by Brent Roberts

Recently, I was asked for the 17^th time by a family member, “So, what are you going to do this summer?”  As usual, I answered, “research.”  And, as usual, I was met with that quizzical look that says, “What the heck is research anyway?”

It struck me in retrospect that I’ve done a pretty poor job of describing what research is to my family and friends.  So, I thought it might be a good idea to write an open letter that tries explaining research a little better.  You deserve an explanation.  So do other people, like parents of students and the general public. Continue reading →

Melissa Ferguson and I are the program co-chairs for the upcoming SPSP conference in New Orleans, January 17-19. That means we are in charge of the scientific content of the program. (Cindy Pickett is the convention chair, meaning she’s in charge of pretty much everything else, which I have discovered is a heck of a lot more than 99% of the world knows. If you see Cindy at the conference, please buy her a drink.) The conference is going to be awesome. You should go.

One issue that I’m particularly attuned to is the representation of personality psychology on the program. During my work as program co-chair, I heard from some people who are from a more centrally personality-psych background that they’re worried that the conference is tilted too heavily toward social psych, and therefore there won’t be enough interesting stuff to go to.

I am writing here to dispel that notion. If you are a personality psychologist and you’re wavering about going, trust me: there’ll be lots of exciting stuff for you.

SPSP has a long-standing commitment to ensuring that both of its parent disciplines are well represented at the conference. That means, first of all, that the 2 program co-chairs are picked to make sure there is broad representation at the top. Continue reading →

There is a great profile of Uri Simonsohn’s fraud-detection work in the Atlantic Monthly, written by Chris Shea (via Andrew Gelman). This paragraph popped out at me:

So what, then, is driving Simonsohn? His fraud-busting has an almost existential flavor. “I couldn’t tolerate knowing something was fake and not doing something about it,” he told me. “Everything loses meaning. What’s the point of writing a paper, fighting very hard to get it published, going to conferences?”

 It reminded me of a story involving my colleague (and grand-advisor) Lew Goldberg. Lew was at a conference once when someone presented a result that he was certain could not be correct. After the talk, Lew stood up and publicly challenged the speaker to a bet that she’d made a coding error in the data. (The bet offer is officially part of the published scientific record. According to people who were there, it was for a case of whiskey.)

The research got published anyway, there were several years of back-and-forth with what Lew felt was a vague and insufficient admission of possible errors, which ended up with Lew and colleagues publishing a comment on an erratum – the only time I’ve ever heard of that happening in a scientific journal. Continue reading →

Breathless headline-grabbing press releases based on modest findings. Investigations driven by confirmation bias. Broad generalizations based on tiny samples.

I am talking, of course, about the final report of the Diederik Stapel investigation.

Regular readers of my blog will know that I have been beating the drum for reform for quite a while. I absolutely think psychology in general, and perhaps social psychology especially, can and must work to improve its methods and practices.

But in reading the commission’s press release, which talks about “a general culture of careless, selective and uncritical handling of research and data” in social psychology, I am struck that those conclusions are based on a retrospective review of a known fraud case — a case that the commissions were specifically charged with finding an explanation for. So when they wag their fingers about a field rife with elementary statistical errors and confirmation bias, it’s a bit much for me.

I am writing this as a first reaction based on what I’ve seen in the press. At some point when I have the time and the stomach I plan to dig into the full 100-page commission report. I hope that — as is often the case when you go from a press release to an actual report — it takes a more sober and cautious tone. Continue reading →

The tools we have available to us affect the way we interact with and even think about the world. “If all you have is a hammer” etc. Along these lines, I’ve been wondering what would happen if the makers of data analysis software like SPSS, SAS, etc. changed some of the defaults and options. Sort of in the spirit of Nudge — don’t necessarily change the list of what is ultimately possible to do, but make changes to make some things easier and other things harder (like via defaults and options).

Would people think about their data differently? Here’s my list of how I might change regression procedures, and what I think these changes might do:

1. Let users write common transformations of variables directly into the syntax. Things like centering, z-scoring, log-transforming, multiplying variables into interactions, etc. This is already part of some packages (it’s easy to do in R), but not others. In particular, running interactions in SPSS is a huge royal pain. For example, to do a simple 2-way interaction with centered variables, you have to write all this crap *and* cycle back and forth between the code and the output along the way:

desc x1 x2. Continue reading →

Imagine that you have entered a charity drawing to win a free iPad. The charity organizer draws a ticket, and it’s your number. Hooray! But wait, someone else is cheering too. After a little investigation it turns out that due to a printing error, two different tickets had the same winning number. You don’t want to be a jerk and make the charity buy another iPad, and you can’t saw it in half. So you have to decide who gets the iPad.

Suppose that someone proposes to flip a coin to decide who gets the iPad. Sounds pretty fair, right?

But suppose that the other guy with a winning ticket — let’s call him Pete — instead proposes the following procedure. First the organizer will flip a coin. If Pete wins that flip, he gets the iPad. But if you win the flip, then the organizer will toss the coin 2 more times. Continue reading →

Pretty big news. Psychological Science is seriously discussing 3 new reform initiatives. They are outlined in a letter being circulated by Eric Eich, editor of the journal, and they come from a working group that includes top people from APS and several other scientists who have been active in working for reforms.

After reading it through (which I encourage everybody to do), here are my initial takes on the 3 initiatives:

Initiative 1: Create tutorials on power, effect size, and confidence intervals. There’s plenty of stuff out there already, but if PSci creates a good new source and funnels authors to it, it could be a good thing.

Initiative 2: Disclosure statements about research process (such as how sample size was determined, unreported measures, etc.) This could end up being a good thing, but it will be complicated. Simine Vazire, one of the working group members who is quoted in the proposal, puts it well:

We are essentially asking people to “incriminate” themselves — i.e., reveal information that, in the past, editors have treated as reasons not to publish a paper. If we want authors to be honest, I think they will want some explicit acknowledgement that some degree of messiness (e. Continue reading →

This morning felt quite ignominious indeed, and naturally it reminded me of William James. From the Principles of Psychology, chapter 26, “Will”:

We know what it is to get out of bed on a freezing morning in a room without a fire, and how the very vital principle within us protests against the ordeal. Probably most persons have lain on certain mornings for an hour at a time unable to brace themselves to the resolve. We think how late we shall be, how the duties of the day will suffer; we say, “I must get up, this is ignominious,” etc.; but still the warm couch feels too delicious, the cold outside too cruel, and resolution faints away and postpones itself again and again just as it seemed on the verge of bursting the resistance and passing over into the decisive act. Now how do we ever get up under such circumstances? If I may generalize from my own experience, we more often than not get up without any struggle or decision at all. We suddenly find that we have got up. A fortunate lapse of consciousness occurs; we forget both the warmth and the cold; we fall into some revery connected with the day’s life, in the course of which the idea flashes across us, “Hollo! I must lie here no longer” – an idea which at that lucky instant awakens no contradictory or paralyzing suggestions, and consequently produces immediately its appropriate motor effects. It was our acute consciousness of both the warmth and the cold during the period of struggle, which paralyzed our activity then and kept our idea of rising in the condition of wish and not of will. The moment these inhibitory ideas ceased, the original idea exerted its effects.

James’s visible presence in contemporary psychology seem mostly limited to 2 roles. Continue reading →