Category Archives: sometimes i’m wrong

it’s halftime in america.  or at least it’s halftime in game five of the clippers-rockets series.  although i had planned to root for the clippers, mostly because they managed to eliminate the spurs and the holder of the best nickname ever,** i am finding it hard to resist james harden’s mohawk/beard combo.  clippers or rockets?  so hard to choose. this is a common problem for me.  i like to have it both ways.  i am a personality psychologist and a social psychologist. i’m an atheist who is baptized and had my first communion, but if anyone in iran asks, i’m muslim. i think eating meat is mostly wrong but i just ate a bbq cheeseburger with pepperjack cheese.  i am vast. i contain multitudes. (and now also a bbq cheeseburger). Continue reading →

Check yourself before you wreck yourself

by Michael Inzlicht

Things have gone sideways in social psychology. And, I am not sure this is necessarily a recent trend. The rot in my chosen field might have been festering for a very long time. I say this because not a day goes by when I do not hear about one or another of our cherished findings falling under disrepute. You have all heard about the depressingly high number of failed replications, be they one-offs or large scale coordinated lab attempts. You have also, no doubt, heard about problems with publication bias, low statistical power, and the widespread use of questionable research practices. Taken individually, none of these might be terribly upsetting; together they are an unmitigated disaster. Perhaps more upsetting than the real problems facing my beloved social psychology is the level of denial I see. I have heard suggestions that things aren’t so bad. “Yes, there are a few effects that are probably not real,” say the defenders, “but we have made real and extraordinary breakthroughs and most of the stuff we study is rock solid”. I have also heard that every twenty to thirty years our field goes through its routine hand-wringing, but that it always passes. Continue reading →

this idea keeps popping up: if you conduct a replication study and get a null result, you need to explain why the original study found a significant effect and you didn't.

what's wrong with this idea?  a few things.

first, it seems to discount the possibility that the original finding was a fluke - a false positive that made it look like there is an effect when in fact there isn't.  here's an analogy:

null hypothesis: my coin is fair

research hypothesis: my coin is weighted

original study: i flip the coin 20 times and get 15 heads (p = .041) replication study: i flip the coin another 20 times and get 10 heads (p = 1.0) do i need to explain why i got 15 heads the first time?

maybe.  or maybe the first study was just a fluke.  that happens sometimes (4.1% of the time, to be exact).

what if the replication study was: i flip the same coin 100 times and get 50 heads?  now isn't the evidence pretty strong that the null is true, and the original study was just a fluke? Continue reading →

Not nutting up or shutting up: Notes on the demographic disconnect in our field’s best practices conversation

Alison Ledgerwood, Elizabeth Haines, and Kate Ratliff

A few weeks ago, two of us chaired a symposium on best practices at SPSP focusing on concrete steps that researchers can take right now to maximize the information they get from the work that they do. Before starting, we paused briefly to ask a couple simple questions about the field’s ongoing conversation on these issues. Our goal was to take a step back for a moment and consider both who is doing the talking as well as how are we talking about these issues. Apparently our brief pause sounded strident to some ears, and precipitated an email debate that was ultimately publicized on two blogs. The thing is, the issues we originally wanted to raise seemed to be getting a little lost in translation. And somehow, despite the absolute best of intentions of the two people having the (cordial, reasonable, interesting) debate, we had become literally invisible in the conversation that was taking place. So we thought maybe we would chime in, and Simine graciously allowed us to guest blog.* As we said in our symposium, a conversation about where the field as a whole is going should involve the field as a whole. And yet, when we look at the demographics of the voices involved in the conversation on best practices and the demographics of the field, it’s clear that there’s a disconnect.** For instance, the SPSP membership is about 56% female. Continue reading →

california. not relevant.  just nice.

below is a joint blog entry with lee jussim.  the title, pictures, and post-script are mine, the rest is posted on both of our blogs. joint introduction recently, two friends and scholars who are working together on scientific integrity stuff had a very sane and civil email discussion about gender representation in the scientific integrity debate and in STEM more generally.  at the end of the discussion, neither had convinced the other but they decided it was still an interesting and informative discussion, and they decided to post it on their blogs for the world to see.  you're welcome, world.   you can find simine’s blog here:   Continue reading →

keri russell plotting her next QRP

i am teaching a seminar called 'oh you like that finding do you?  well it's probably FALSE.' the students are a little bit shell-shocked. i am having the time of my life. the hardest part is explaining how perfectly smart, truth-seeking scientists can repeatedly do idiotic things.  happily, i have plenty of examples from my own life. just this week, i almost p-hacked.  the details are a little dry, but i think it's worth telling this tale of mundane p-hackery, because this is what p-hacking looks like in the wild.  it is not sinister and dark like a good episode of the americans.  it is super boring like the stories you read to your children at night. Continue reading →

  i am not going to say anything original in this post. my philosophy of science friends tell me that i should abandon popper.  before i do, i'd like to have one last fling with him.

i am teaching two research methods classes this quarter, so i'm teaching, you know, how scientists are objective and our theories are falsifiable and we seek to disconfirm them. (hello psc41 students!)today's topic: modus tollens and falsifiability.

for those of you who don't remember day 2 of research methods, here is modus tollens (aka, denying by denying, a valid form of logical argument):

if p then q

~q

therefore ~p

why is modus tollens important for empirical research?  i am cribbing from meehl 1978 here (who himself was stealing from others, including of course popper), but basically, it's how we make our theories falsifiable.  we make risky predictions of the sort:

if my theory is correct, then i should observe a certain pattern of data.

then we collect data.

if the data show what we predicted, then our theory is left standing. of course it doesn't prove that our theory is correct, that would be affirming the consequent, and we all know that's a logical fallacy. Continue reading →

  i have been re-watching the west wing*, and i think i just discovered the meaning of life.  in the episode i just saw, president bartlett comes back from church and complains about the pastor** getting ephesians wrong.  the point of ephesians, he says, is: be subject to one another. i am not religious***, and a quick glance at the rest of ephesians 5 tells me that i probably don't have much in common with paul****.  but the phrase resonated with me. and it reminded me of a passage i read years ago:

'You fight your superficiality, your shallowness, so as to try to come at people without unreal expectations, without an overload of bias or hope or arrogance, as untanklike as you can be, sans cannon and machine guns and steel plating half a foot thick; you come at them unmenacingly on your own ten toes instead of tearing up the turf with your caterpillar treads [...]' -philip roth, american pastoral*****

it applies to scientific inquiry - it means subjecting ourselves to different opinions, different perspectives, disagreement.  and actually being open to changing. Continue reading →

i was recently engaged in a totally civil disagreement on facebook* about whether things are changing in our field in response to all of the concerns about the robustness of our scientific findings, and the integrity of our methods.  i said i was optimistic.  someone asked me if i could elaborate on why i am optimistic.  this is for you, brent roberts.

1. the obvious reasons

the fact that the open science framework exists, that the reproducibility project exists, that social psychology did a special issue on replications, that there have been workshops on reproducibility at NSF, that SPSP had a taskforce on best practices, that APS had a taskforce that led to changes in submission guidelines and procedures at psych science, that psych science commissioned and published an article and tutorials on effect estimation approaches by geoff cumming, that perspectives coordinates and publishes registered replication reports, and that every conference i've been to for at least the last two years has included some presentations on replicability.  everyone already knows all of this, but it's easy to take it for granted.  i'm pretty sure that if i had told someone in 2007 that all of this would happen in the next seven years, they would not have believed me.

2. i am not a pariah

i am shocked and amazed that people still tolerate me after i spew my not-always-entirely-well-thought-out opinions on my blog.  when i started blogging i never thought that 1,500 people would see my posts.  i'd like to think this has something to do with my biting sense of humor, but i know it's because there is a huge amount of interest in this topic. other blogs, like  sanjay srivastava's and daniel lakens's get tons of readers, i'm sure.  betsy levy paluck has 1,500 followers on twitter.  dorothy bishop has 16,000. Continue reading →

 

 

dear editors, i love journals.  i love editors.  i love editors of journals.  that's why i want to help.  we need more quality control in our journals, and you are the ones who can do it. in july 2012, in a comment to a blog post, chris fraley wrote 'What we might need, in other words, is a formal “consumer reports” for our leading journals.' i was so excited by this idea that i wrote to him and told him it was the best actionable idea that has come out of the replicability discussion.* fast forward 27 months, and our paper, 'N-Pact Factor: Evaluating the quality of empirical journals with respect to sample size and statistical power' is out. you can read the actual paper here, and you can read chris's blog post about it here. Continue reading →