Category Archives: sometimes i’m wrong

many people have written about how bem's esp paper was one of the major factors that triggered the latest scientific integrity movement in social/personality psychology.  that is an interesting story.  but that is not the bem paper i want to talk about today.

i have come here today to discuss bem's chapter, 'writing the empirical journal article' (2003) that i - and i suspect many others - used to assign to every undergrad and graduate student taking our psychology research methods classes.  there are many extremely wise points in that chapter.  but there are also many pieces of advice that seem entirely antiquated today.  if the bem chapter is no longer the gold standard for how to write an empirical article, what is? (see also: laura king's article for the spsp dialogue (pdf, p. 6)).

i was reminded of the complexity of this question when a historian friend of mine suggested i read 'the question of narrative in contemporary historical theory' by hayden white (1984). i will share a few quotes with you:

'but it is precisely because the narrative mode of representation is so natural to human consciousness, so much an aspect of everyday speech and ordinary discourse, that its use in any field of study aspiring to the status of a science must be suspect. for whatever else a science may be, it is also a practice which must be as critical about the way it describes its objects of study as it is about the way it explains their structures and processes.'

'a discipline that produces narrative accounts of its subject matter as an end in itself seems methodologically unsound; one that investigates its data in the interest of telling a story about them appears theoretically deficient. Continue reading →

one of the most fascinating dimensions along which psychology researchers differ is in their reaction to counterintuitive findings.  when some people say 'i can't believe it!' they mean it as a compliment.  when others say it, you should take cover.  how should we feel about counterintuitive findings?

i'll come out and say it: i have not drunk the bayesian kool-aid. i do like the idea that the amount of evidence required to support a claim should depend on the plausibility of that claim to begin with, but the reason i'm not a whole-hearted bayesian is that i am skeptical that there will be much consensus in psychology about which claims are more or less probable.  (have you ever asked a group of psychologists what proportion of our a priori hypotheses are likely to be right? you should try it, it's a fun party trick.) but i have seen cases where pretty much everyone agrees that a conclusion is very counterintuitive (in fact, there are quite a few cases where the authors themselves appeal to this as a selling point of their work).  and in those cases we can ask: given that we all agree this is surprising, should we hold the research to a higher standard? do the authors need more evidence if the claim they are making is widely acknowledged to be shocking?

Continue reading →

 

should replications be held to a higher standard than original research?

i have seen some very bright and influential people argue that they should, mainly because of the potential damage that a failed replication could do to the original author's reputation.  according to this argument, shoddy original research may be 'irresponsible' but shoddy replication is a 'sin'.* 

i have several objections to this.

1. original research is often treated as precedent.  my impression is that people see the original finding as very likely to be true, and require a lot of new evidence to be convinced otherwise.  this is problematic for many reasons, but if it's true, it's all the more reason to hold original research to a very high standard.  giving original research a pass is dangerous given how hard it is to overturn a finding once it is in the literature.

2. Continue reading →

 

magical things can still happen in used bookstores.

i was killing time in one and came across a book by richard feynman called 'the meaning of it all'.  i noticed that the first essay was called 'the uncertainty of science'. i bought it.  it was the best $4 i've spent in a while.

it is tempting to just re-type the entire essay here, but in the interest of trying not to violate copyright, and trying to contribute something of my own, i will share some excerpts and reflections.

first, feynman on p-hacking:

'it is necessary to look at the results of observation objectively, because you, the experimenter, might like one result better than another. you perform the experiment several times, and because of irregularities, like pieces of dirt falling in, the result varies from time to time. you do not have everything under control. Continue reading →

 

 this blogging thing is pretty rad.  (also: twitter.  wow.)

today's topic: having it all.  i'm not talking about the work/life problem.  i'm talking about the sample size/methodological rigor problem. 

let's start with sample size.  by now, everyone knows that bigger is better.  you can't have too large a sample.  there is no 'double-edged sword'.  there are no downsides to a large sample.  more evidence is always better, and larger samples = more evidence.

this seems very obvious but i've seen at least three different editors criticize manuscripts for having samples that are too big.  so i want to be very clear: there is no such thing as a sample that is too big. Continue reading →

i was having breakfast with friends at beauty's bagel when i saw this picture.  i could not stop staring.  the jesus-like erotic figure is nice of course, but what mesmerized me was the sentiment.  this is the phrase i've been searching for.  i can finally start my blog.

this blog is about being wrong.  specifically, i plan to write about two things:
1. the search for truth in psychological research.
2. the search for self-knowledge in everyday life.

my thesis: both of these pursuits are riddled with obstacles, and should be approached with a great deal of skepticism and humility. Continue reading →