Category Archives: sometimes i’m wrong

with permission, an excerpt from john doris's forthcoming book. i wanted to post this because i think it is an excellent summary of the current crisis in social/personality psychology from a well-informed 'outsider'.  this will be review for many psychologists, but should be useful for new researchers, and for those outside of psychology who want some background/analysis from someone who is not in the trenches but has read and thought a lot about this literature and these issues.  

art by melissa dominiak

Doris, J. M. 2015. Talking to Our Selves: Reflection, Ignorance, and Agency. Oxford:Oxford University Press.   […]  At this writing, social psychology is being shaken by charges that many published findings, including numerous iconic findings, do not replicate when tested by independent investigators. Continue reading →

 

i have been sitting on this paul meehl gem for a few months now, ruminating on how it relates to our current situation:

"The two opposite errors to which psychologists, especially clinical psychologists, are tempted are the simpleminded and the muddleheaded (as Whitehead and Russell labeled each other in a famous dinner exchange). The simpleminded, due to their hypercriticality and superscientism and their acceptance of a variant of operationalist philosophy of science (that hardly any historian or logician of science has defended unqualifiedly for at least 30 years), tend to have a difficult time discovering anything interesting or exciting about the mind. The muddleheads, per contra, have a tendency to discover a lot of interesting things that are not so. I have never been able, despite my Minnesota “simpleminded” training, to decide between these two evils. At times it has seemed to me that the best solution is sort of like the political one, namely, we wait for clever muddleheads to cook up interesting possibilities and the task of the simpleminded contingent is then to sift the wheat from the chaff. But I do not really believe this, partly because I have become increasingly convinced that you cannot do the right kind of research on an interesting theoretical position if you are too simpleminded to enter into its frame of reference fully (see, e.g., Meehl, 1970b). One hardly knows how to choose between these two methodological sins." *

here is what i have come up with (i am trying to fit what probably belongs in several separate blog posts into one because i think the points are interconnected.  bear with me.)

1. another way to describe these groups is that the simpleminded are terrified of type I error while the muddleheaded are terrified of type II error. Continue reading →

i know a thing or two about pain.  just the other day, i was enjoying* a giants game when a foul ball landed on my knee.  (i failed to catch the ball.  this guy (above) ended up with it.  i do, however, still have the bruise. i would post a picture of that but i don't do selfies.) some things in life just hurt.  expecting them not to hurt is like expecting kale to taste good.  

self-correction is one of those things.  it is never going to be pleasant.  whether it is in the domain of science or in the personal domain, self-improvement sucks.  if we say that we should only self-correct if we can do it in a way that feels good for everyone, then we are giving up on the idea of self-correction. if we are that delicate, we are doomed. Continue reading →

Anti-scientific-integrity vs. Pro-witch-hunt: The problem with median splits

In addition to agreeing with everything Simine said (and almost* everything she ever says) I share her single-quotation-mark-denoted skepticism of ‘sides’ in the replicability controversy. I started thinking about this as I caught myself approaching each tweet, or status update, or blog post over the past few weeks with a “wait, what side is this person on?” mentality. After coming across several that were difficult to categorize, and also considering what it would look like to have some kind of ‘balanced’ discussion, I started to realize that thinking about these issues in terms of sides is at best misguided, and at worst extremely divisive and counter-productive.

This isn’t to say that people don’t have different perspectives, and that these perspectives never come into conflict. Certainly there are several mutually incompatible views currently being entertained by very smart people, and arguments between these people have the potential to be very fruitful.

But, I think most people’s views fall on a continuum (or, probably more accurately, on multiple continua). After all, median splits are bad, right? When it comes to our data we’re taught not to artificially truncate the natural variability that exists amongst participants. Doing so inaccurately portrays people’s responses as categorical when, in all likelihood, two categories probably do a very poor job of characterizing the complexity of people’s perspectives.

As far as I can tell very few people are “anti-scientific-integrity.” I also come across very few people who think that our field would be perfectly fine operating exactly as it has been. Continue reading →

because i can't very well have a blog about scientific integrity and not blog about this incident, some thoughts:

1. there are assholes on both 'sides.'**  of everything.  there are assholes everywhere. there is probably one under your chair right now.

2. most people on both sides are not assholes.  in this case, none of the main characters are assholes.  the record shows that those with significant roles in the special issue (original authors, replicators, editors) have tried very hard to be fair and balanced.  this is impressive.  i admire all of them for their dedication to the scientific process.  they all got more than they bargained for, and we owe them our thanks. thank you. Continue reading →

i don't mean to pile on,* but want to share two quick thoughts about the jens förster case.

1.

first, i think the evidence is pretty overwhelming that the data in the paper in question are not real.  in addition to the original report and LOWI committee's report, additional analyses by the data colada guys show that the pattern of results could never have happened without data manipulation. 

this case is interesting in part because the evidence for fraud comes from statistical analyses of the published results, rather than from a whistleblower inside the lab, or a confession.  this makes some people uncomfortable.  i agree that concluding fraud based on probabilities could be problematic - this case makes us wonder what we would think if the odds were, say, 1 in 10,000.  how small does the probability have to be for us to conclude fraud?

i don't know, but i know 1 in 508 quintillion is improbable enough.  i agree it is worth thinking about what we should do with borderline cases, but it is also important to recognize that this is not a borderline case. Continue reading →

the question came up a few weeks ago: if two measures are correlated .60, should they be aggregated into a single measure or kept separate?  this might seem like a narrow question, but it raises some deep and complicated issues.  at the heart of the matter is: when are two measures measuring the same construct?

 

thinking about this reminded me of the jingle and jangle fallacies.  i have not read the original papers (the jingle fallacy apparently dates back to edward thorndike, 1904, and the jangle fallacy to truman kelley, 1927.  i have not tracked down either book/paper.  sorry.) the jingle fallacy is calling two things by the same name that are actually different constructs.  the jangle fallacy is using two different names for things that are actually the same construct.

 

let's get specific.  let's take the constructs 'neuroticism' and 'negative affect'.  are these two the same thing? (maybe, maybe not).

one obvious difference is that neuroticism is usually thought of as a stable personality trait, whereas negative affect is typically thought of as a momentary state. Continue reading →

someone recently asked my why i do what i do.  it's easy to come up with just-so stories. for all i know, i could just as easily have ended up inventing new ice cream flavors for tara's ice cream (i still haven't ruled it out).  and if i had, i could probably make up a story about why it was always meant to be.  narratives are deceptively easy to construct, and amazingly convincing after the fact. it would be especially ironic for a self-knowledge researcher to deceive herself about why she studies self-knowledge.  so i mostly try to resist giving an explanation.  but then there's this:

this is a note from my best friend, written when we were 15. (i have honored her request to remain anonymous -- she undoubtedly has much more embarrasing material on me.)

a few excerpts:

'at miriam's birthday party, i got the idea that you wanted an honest evaluation from me about you.'

Continue reading →

 

'those who venture to criticize us perform a remarkable act of friendship,
for to undertake to wound and offend a man for his own good
is to have a healthy love for him.'

-michel de montaigne*

(all this time, reviewers were just expressing their healthy love for me!)

today i am going to talk about self-deception. don't worry, i will connect it back to scientific integrity.

being a researcher who studies self-knowledge and self-deception is a little nerve-wracking.  watching other people delude themselves and be entirely convinced by their self-deception, you start to wonder whether all of your own self-beliefs might not also be delusional. it can make a person paranoid. so i started wondering, could there be an internal marker of self-deception? a red flag that, if trained, one could detect and catch oneself in the act of self-deception? Continue reading →

i learned a new word the other day. bucketing. it almost made me cry.

one of the most common mistakes i see when reviewing papers is authors who take a continuous variable and, for no good reason, mutilate it by turning it into a categorical variable. our old friend the median split is one example.  (whose idea was it to befriend the median split? and why won’t he stop harassing us?)

bucketing, from what i can tell, is another such technique.  i had a hard time finding a definition, but i think it’s basically creating categories out of multiple response options and grouping the data that way. for example, you can turn the continuous variable ‘age’ into a categorical variable by categorizing people into age ‘buckets’ (e.g, 20-29, 30-39, etc.). Continue reading →