How reliable are psychiatric diagnoses?

DSM-5 Cover.png

This post, which mostly provides useful citations, was originally written as an answer to the following Quora question:

Is ADHD one of the only psychiatric conditions that can be diagnosed objectively?

It really depends on what you mean by “objective”, but the answer is “probably not”.

Since we do not understand the underlying causes of ADHD — or any major psychiatric disorder — we diagnose them based on clusters of symptoms.

In the United States and several other countries, a large number of psychiatrists use a book called Diagnostic and Statistical Manual of Mental Disorders (DSM).

DSM (now in it’s 5th, revised edition, DSM-5) essentially uses a system of checklists to enable a clinician to assess if a person has a given disorder. This is a controversial book for various reasons, but for now it is what most psychiatrists use.

Instead of the complex philosophical question of ‘objectivity’, the usefulness of DSM can be assessed using statistical measures of ‘reliability’.

Given that one clinician uses DSM-5 to give the diagnosis of ADHD, how likely is another clinician to do so using the DSM-5? Measures of “test-retest reliability” capture this probability.

Here is a paper that explains the statistical measurement of reliability in some detail:

DSM-5: How Reliable Is Reliable Enough? [pdf]

There are conflicting reports on the reliability of DSM-5, but here is one paper that reports statistical assessments:

DSM-5 field trials in the United States and Canada, Part II: test-retest reliability of selected categorical diagnoses.

“There were a total of 15 adult and eight child/adolescent diagnoses for which adequate sample sizes were obtained to report adequately precise estimates of the intraclass kappa. Overall, five diagnoses were in the very good range(kappa=0.60–0.79), nine in the good range(kappa=0.40–0.59), six in the questionable range (kappa = 0.20–0.39), and three in the unacceptable range (kappa values,0.20). Eight diagnoses had insufficient sample sizes to generate precise kappa estimates at any site.”

[…]

“Two were in the very good (kappa=0.60–0.79) range: autism spectrum disorder and ADHD.”

For more on the quantity reported here, kappa, see this paper:

Interrater reliability: the kappa statistic

The quantity kappa ranges from 0 to 1. Zero means that there was no agreement between raters (clinicians in this case), and 1 means there was perfect agreement.


As I said before, the DSM is controversial — and not just because of reliability issues. Here is a sampling of papers and popular articles on the general topic:

Academic articles

Reliability in Psychiatric Diagnosis with the DSM: Old Wine in New Barrels

“However, the standards for evaluating κ-statistics have relaxed substantially over time. In the early days of systematic reliability research, Spitzer and Fleiss [4] suggested that in psychiatric research κ-values ≥0.90 are excellent; values between 0.70 and 0.90 are good, while values ≤0.70 are unacceptable. In 1977, Landis and Koch [5] proposed the frequently used thresholds: values ≥0.75 are excellent; values between 0.40 and 0.75 indicate fair to good reliability, and values ≤0.40 indicate poor reliability. More recently, Baer and Blais [6] suggested that κ-values >0.70 are excellent; values between 0.60 and 0.70 are good; values between 0.41 and 0.59 are questionable, and values ≤0.40 are poor. Considering these standards, the norms used in the DSM-5 field trial are unacceptably generous.”

The Reliability of Psychiatric Diagnoses: Point—Our psychiatric Diagnoses are Still Unreliable

“Today, 26 years later, did the DSM system succeed in improving the reliability of psychiatric diagnoses? Two answers exist. The DSM did improve the reliability of psychiatric diagnoses at the research level. If a researcher or a clinician can afford to spend 2 to 3 hours per patient using the DSM criteria and a structured interview or a rating scale, the reliability would improve. [13] For psychiatrists and clinicians, who live in a world without hours to spare, the reliability of psychiatric diagnoses is still poor. [2,3]”

Diagnostic Issues and Controversies in DSM-5: Return of the False Positives Problem.

“The fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) was the most controversial in the manual’s history. This review selectively surveys some of the most important changes in DSM-5, including structural/organizational changes, modifications of diagnostic criteria, and newly introduced categories. It analyzes why these changes led to such heated controversies, which included objections to the revision’s process, its goals, and the content of altered criteria and new categories. The central focus is on disputes concerning the false positives problem of setting a valid boundary between disorder and normal variation. Finally, this review highlights key problems and issues that currently remain unresolved and need to be addressed in the future, including systematically identifying false positive weaknesses in criteria, distinguishing risk from disorder, including context in diagnostic criteria, clarifying how to handle fuzzy boundaries, and improving the guidelines for “other specified” diagnosis.”

Popular articles

The DSM-5 Controversy

“You will need to display fewer and fewer symptoms to get labeled with certain disorders, for example Attention Deficit Disorder and Generalized Anxiety Disorder. Children will have more and more mental disorder labels available to pin on them. These are clearly boons to the mental health industry but are they legitimate additions to the manual that mental health professionals use to diagnose their clients?”

DSM 5 Is Guide Not Bible—Ignore Its Ten Worst Changes

“This is the saddest moment in my 45 year career of studying, practicing, and teaching psychiatry. The Board of Trustees of the American Psychiatric Association has given its final approval to a deeply flawed DSM 5 containing many changes that seem clearly unsafe and scientifically unsound. My best advice to clinicians, to the press, and to the general public – be skeptical and don’t follow DSM 5 blindly down a road likely to lead to massive over-diagnosis and harmful over-medication. Just ignore the ten changes that make no sense.”

Normal or Not? New Psychiatric Manual Stirs Controversy

“Among the flashpoints: Asperger’s disorder will be folded into autism spectrum disorder; grief will no longer exempt someone from a diagnosis of depression; irritable children who throw frequent temper tantrums can be diagnosed with disruptive mood dysregulation disorder. [Hypersex to Hoarding: 7 New Psychological Disorders]

“One prominent critic has been Allen Frances, a professor emeritus of psychiatry at Duke University who chaired the DSM-IV task force.

“Frances charges that through a combination of new disorders and lowered thresholds, the DSM-5 is expanding the boundaries of psychiatry to encompass many whom he describes as the “worried well.””

Advertisements

The head and the heart

I was asked the following question on Quora some time ago:

Why do people point/refer to their chest (heart) when they talk about the mind, when the head (brain) is the organ used for doing so?

Here’s my answer:

People who have had a western-style education generally point to their head when they are talking about their mind, and to their chest when they are talking about their heart, which is the “metaphorical container” for many if not most emotions.

Around the world, people have always associated the heart with intense emotions — anger, love, fear and so on. This may be because these emotions are actually felt in the heart and lungs. When you are aroused by strong anger, love, or fear, you may feel that your chest is pounding — and it often is! Your emotional state can affect your heart-rate and your breathing.

So using the heart as the metaphorical container for emotion is quite understandable.

What is harder for us to understand is why some cultures — such as Ancient Egypt — use(d) the heart as their metaphorical container for all mental concepts, including intelligence.

Didn’t Ancient Egyptians know that head injuries affected behavior and intelligence? The answer is yes they did know*, but for some reason this knowledge wasn’t prominent in their literary culture, which was happy to stick with the heart metaphor.

We can speculate that certain cultures — both ancient and modern — identify the agent or person with the seat of emotion, rather than the seat of seeing, hearing, smelling and tasting (which is easily identified with the head). Personhood is a complex concept, and to this day no one fully understands what it is. There is nothing intrinsically wrong with identifying the Self with emotions, and then assigning the heart as the symbol for emotional experience.

Our language and gestures are symbolic, and ultimately any symbol will suffice to communicate a basic idea. You might wonder why we don’t point to some specific part of the head, for example, when we talk about a particular aspect of cognition — after all, we have rough scientific conceptions of neural processes now. The answer is that it doesn’t matter all that much for the purpose of communication.

Having said that, it’s helpful for understanding (and it’s also aesthetically pleasing!) if our symbols partially reflect the underlying biological process, which is why using the heart as a metaphor for the seat of emotion is still quite acceptable. In the same way, we might say that a surgeon or pianist has “good hands”, even though neuroscientists tend to agree that dexterity is largely achieved by neural connections in the head, not the hands.


One of my favorite books is The Origin of Consciousness in the Breakdown of the Bicameral Mind, by Julian Jaynes. It’s a deeply strange book, so you have to take all its conclusions with a hefty pinch of salt. But the main reason I love it is because Jaynes asks questions that many people simply neglect to ask in the first place.

For example, Jaynes asks how exactly body parts became “metaphorical containers” for abstract qualities. How did courage become associated with the gut, or emotion with the heart, or life-force with the air in the lungs (from which the word ‘psyche’ ultimately derives)?

Jaynes’s answer — which you don’t have to believe, of course — is that humans discovered the associations between body parts and attributes through violence and death. He thinks that ancient battlefields might have taught people where abstract attributes were ‘located’. A stomach injury might make a person decidedly less brave. And when a person exhales for the last time their life-force seems to leave the body.

This way of thinking can seem quite primitive, but the way we conduct neuroscience now is basically an outgrowth of the same logic. We see what function is lost when a particular brain area breaks down, and then we label that brain area as the seat of that function.


Note

* See this answer for some quotes that show that at least some Ancient Egyptians were well aware of the importance of the head:

Israel Ramirez’s answer to What did people think the brain was before its actual function was found?

 

Yanny or Laurel? A perspective from the science of mind and brain

I really like the Yanny versus Laurel meme, which exploded yesterday. It helps illustrate some key points about human perception:

  1. In some situations people can differ wildly in their experience of low-level perception.
  2. Active top-down expectations (and other, weirder processes) have a strong effect on low-level perception.

So basically, it’s an auditory version of #ThatDress.

Continue reading

Is a memory a bunch of atoms? And does this mean we can transfer exact memories?

I was asked the following question on Quora.

Are specific memories just arrangements of atoms in our brains? Could you put certain molecules in someones head and give them an exact memory that you had?

Short answer: No.


Modern science has shown that every thing is an arrangement of atoms: neurons, apples, tables, rockets, asteroids, aardvarks… they are all made up of atoms.

The question now is this: is a memory a thing?

Continue reading

“The first rule of intelligence: Don’t talk about your intelligence”

That line is from an article in The Atlantic about how poor people are at self-assessment:

People Don’t Actually Know Themselves Very Well

“The first rule of intelligence: Don’t talk about your intelligence. It’s something you prove, not something you claim. As comedian Patton Oswalt quipped about humor, the only person who goes around saying “I’m funny” is a not-funny person. If you were really funny, you’d just make people laugh.”

To me this kind of thing is pretty obvious, but I guess some people really need to be reminded of it.

Here’s another paragraph with several important reminders, particularly for people who blather about intelligence and cognitive biases:

“This is why people consistently overestimate their intelligence, a pattern that seems to be more pronounced among men than women. It’s also why people overestimate their generosity: It’s a desirable trait. And it’s why people fall victim to my new favorite bias: the I’m-not-biased bias, where people tend to believethey have fewer biases than the average American. But you can’t judge whether you’re biased, because when it comes to yourself, you’re the most biased judge of all. And the more objective people think they are, the more they discriminate, because they don’t realize how vulnerable they are to bias.”

The kind of brain training that actually works!

Just saw this on twitter:

Here’s the link to the (thus far unreviewed!) study:

How much does education improve intelligence? A meta-analysis

“… we found consistent evidence for beneficial effects of education on cognitive abilities, of approximately 1 to 5 IQ points for an additional year of education. Moderator analyses indicated that the effects persisted across the lifespan, and were present on all broad categories of cognitive ability studied. Education appears to be the most consistent, robust, and durable method yet to be identified for raising intelligence.”