How reliable are psychiatric diagnoses?

DSM-5 Cover.png

This post, which mostly provides useful citations, was originally written as an answer to the following Quora question:

Is ADHD one of the only psychiatric conditions that can be diagnosed objectively?

It really depends on what you mean by “objective”, but the answer is “probably not”.

Since we do not understand the underlying causes of ADHD — or any major psychiatric disorder — we diagnose them based on clusters of symptoms.

In the United States and several other countries, a large number of psychiatrists use a book called Diagnostic and Statistical Manual of Mental Disorders (DSM).

DSM (now in it’s 5th, revised edition, DSM-5) essentially uses a system of checklists to enable a clinician to assess if a person has a given disorder. This is a controversial book for various reasons, but for now it is what most psychiatrists use.

Instead of the complex philosophical question of ‘objectivity’, the usefulness of DSM can be assessed using statistical measures of ‘reliability’.

Given that one clinician uses DSM-5 to give the diagnosis of ADHD, how likely is another clinician to do so using the DSM-5? Measures of “test-retest reliability” capture this probability.

Here is a paper that explains the statistical measurement of reliability in some detail:

DSM-5: How Reliable Is Reliable Enough? [pdf]

There are conflicting reports on the reliability of DSM-5, but here is one paper that reports statistical assessments:

DSM-5 field trials in the United States and Canada, Part II: test-retest reliability of selected categorical diagnoses.

“There were a total of 15 adult and eight child/adolescent diagnoses for which adequate sample sizes were obtained to report adequately precise estimates of the intraclass kappa. Overall, five diagnoses were in the very good range(kappa=0.60–0.79), nine in the good range(kappa=0.40–0.59), six in the questionable range (kappa = 0.20–0.39), and three in the unacceptable range (kappa values,0.20). Eight diagnoses had insufficient sample sizes to generate precise kappa estimates at any site.”


“Two were in the very good (kappa=0.60–0.79) range: autism spectrum disorder and ADHD.”

For more on the quantity reported here, kappa, see this paper:

Interrater reliability: the kappa statistic

The quantity kappa ranges from 0 to 1. Zero means that there was no agreement between raters (clinicians in this case), and 1 means there was perfect agreement.

As I said before, the DSM is controversial — and not just because of reliability issues. Here is a sampling of papers and popular articles on the general topic:

Academic articles

Reliability in Psychiatric Diagnosis with the DSM: Old Wine in New Barrels

“However, the standards for evaluating κ-statistics have relaxed substantially over time. In the early days of systematic reliability research, Spitzer and Fleiss [4] suggested that in psychiatric research κ-values ≥0.90 are excellent; values between 0.70 and 0.90 are good, while values ≤0.70 are unacceptable. In 1977, Landis and Koch [5] proposed the frequently used thresholds: values ≥0.75 are excellent; values between 0.40 and 0.75 indicate fair to good reliability, and values ≤0.40 indicate poor reliability. More recently, Baer and Blais [6] suggested that κ-values >0.70 are excellent; values between 0.60 and 0.70 are good; values between 0.41 and 0.59 are questionable, and values ≤0.40 are poor. Considering these standards, the norms used in the DSM-5 field trial are unacceptably generous.”

The Reliability of Psychiatric Diagnoses: Point—Our psychiatric Diagnoses are Still Unreliable

“Today, 26 years later, did the DSM system succeed in improving the reliability of psychiatric diagnoses? Two answers exist. The DSM did improve the reliability of psychiatric diagnoses at the research level. If a researcher or a clinician can afford to spend 2 to 3 hours per patient using the DSM criteria and a structured interview or a rating scale, the reliability would improve. [13] For psychiatrists and clinicians, who live in a world without hours to spare, the reliability of psychiatric diagnoses is still poor. [2,3]”

Diagnostic Issues and Controversies in DSM-5: Return of the False Positives Problem.

“The fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) was the most controversial in the manual’s history. This review selectively surveys some of the most important changes in DSM-5, including structural/organizational changes, modifications of diagnostic criteria, and newly introduced categories. It analyzes why these changes led to such heated controversies, which included objections to the revision’s process, its goals, and the content of altered criteria and new categories. The central focus is on disputes concerning the false positives problem of setting a valid boundary between disorder and normal variation. Finally, this review highlights key problems and issues that currently remain unresolved and need to be addressed in the future, including systematically identifying false positive weaknesses in criteria, distinguishing risk from disorder, including context in diagnostic criteria, clarifying how to handle fuzzy boundaries, and improving the guidelines for “other specified” diagnosis.”

Popular articles

The DSM-5 Controversy

“You will need to display fewer and fewer symptoms to get labeled with certain disorders, for example Attention Deficit Disorder and Generalized Anxiety Disorder. Children will have more and more mental disorder labels available to pin on them. These are clearly boons to the mental health industry but are they legitimate additions to the manual that mental health professionals use to diagnose their clients?”

DSM 5 Is Guide Not Bible—Ignore Its Ten Worst Changes

“This is the saddest moment in my 45 year career of studying, practicing, and teaching psychiatry. The Board of Trustees of the American Psychiatric Association has given its final approval to a deeply flawed DSM 5 containing many changes that seem clearly unsafe and scientifically unsound. My best advice to clinicians, to the press, and to the general public – be skeptical and don’t follow DSM 5 blindly down a road likely to lead to massive over-diagnosis and harmful over-medication. Just ignore the ten changes that make no sense.”

Normal or Not? New Psychiatric Manual Stirs Controversy

“Among the flashpoints: Asperger’s disorder will be folded into autism spectrum disorder; grief will no longer exempt someone from a diagnosis of depression; irritable children who throw frequent temper tantrums can be diagnosed with disruptive mood dysregulation disorder. [Hypersex to Hoarding: 7 New Psychological Disorders]

“One prominent critic has been Allen Frances, a professor emeritus of psychiatry at Duke University who chaired the DSM-IV task force.

“Frances charges that through a combination of new disorders and lowered thresholds, the DSM-5 is expanding the boundaries of psychiatry to encompass many whom he describes as the “worried well.””


On sleep and creativity

800px-francisco_josc3a9_de_goya_y_lucientes_-_the_sleep_of_reason_produces_monsters_28no-_43292c_from_los_caprichos_-_google_art_projectJust came across a nice little article by Ed Yong on how the two major phases of sleep — REM and slow-wave sleep — might contribute to creativity. These ideas have been floating around for a while, but it’s nice to see them in a pop sci article.

A New Theory Linking Sleep and Creativity

Continue reading

Why human memory is not a bit like a computer’s

DisintegrationofPersistence(This is a cross-post of a 3 Quarks Daily article I wrote last year.)

A few months ago I attended a rather peculiar seminar at MIT’s Department of Brain and Cognitive Sciences. A neuroscientist colleague of mine named Robert Ajemian had invited an unusual speaker: a man named Jim Karol, who was billed as having the world’s best memory. According to his website, his abilities include “knowing over 80,000 zip codes, thousands of digits of Pi, the Scrabble dictionary, sports almanacs, MEDICAL journals, and thousands of other facts.” He has memorized the day of the week for every date stretching back to 1AD. And his abilities are not simply matter of superhuman willingness to spend hours memorizing lists. He can add new items to his memory rapidly, on the fly. After a quick look at a deck of cards, he can recall perfectly the order in which they were shuffled. I witnessed him do this last ‘trick’, as well as a few others, so I can testify that his abilities are truly extraordinary [1].

Continue reading

“Are thoughts just a bunch of electrical and chemical signals being tossed around inside the brain, or is there more to it than that?”

“In our world,” said Eustace, “a star is a huge ball of flaming gas.”

“Even in your world, my son, that is not what a star is but only what it is made of…”

The Voyage of the Dawn Treader, CS Lewis

I really like the quote above, which is from the Chronicles of Narnia. It raises a neat little metaphysical question:

Why do we assume that what a thing is made up of is what a thing is?

As a neuroscientist I have to point out that no one really knows what a thought is from a scientific perspective. This means that we don’t know what we would need to measure in order to ‘decode’ a person’s thoughts. For the foreseeable future, I cannot look at a brain scan and say, “This person is definitely thinking about pineapples!”

Of course, thoughts seem to be closely linked with neural patterns in the brain, and those patterns are clearly linked with electro-chemical signaling. Tinkering with the signaling clearly tinkers with the thinking. Otherwise the effects of drugs such as alcohol and coffee on thought would be a mystery. Perhaps some day we will have a scanner that tells us what a person is thinking of.

Matter and form

While I admit that electro-chemical signals being tossed about is a necessary precondition for thinking — no phenomenon that lacks such tossing will be unanimously labeled as thinking — I think that material constituency is a less than stellar guide for thinking about what something is.

Consider charcoal, diamonds, graphite, and graphene. These are made up of carbon. But is that all there is to the story of what they are? I hope the answer is an emphatic no, since they all have radically different properties. Charcoal is black and relatively soft. Diamonds are transparent and exceptionally hard. Graphene and graphite conduct electricity whereas other forms do not.

What explains the differences between the various allotropes of carbon? Clearly it isn’t what they are made of — it’s the same stuff in each case.

Eight allotropes of carbon: a) diamond, b) graphite, c) lonsdaleite, d) C60 buckminsterfullerene, e) C540, Fullerite f) C70, g) amorphous carbon, and h) single-walled carbon nanotube. Source: Wikipedia


What differs among the allotropes is the arrangements of carbon atoms. In other words, form is as important as ‘content’. Depending on how you arrange carbon atoms, you will end up with something soft and opaque or hard and transparent. Clearly the properties of the substances are not to be found in the properties of the atoms.

This is generally true for most complex and interesting objects and processes. You can boil them down to some set of elements — and these elements maybe subatomic particles, atoms, molecules, genes, cells, neurotransmitters — but some defining feature of the overarching process will be missing from the constituent parts considered in isolation, just as transparency or opaqueness are missing from individual carbon atoms.

In complex systems theory and in condensed matter physics, the word emergence is often used to describe the phenomenon by which collections of matter acquire new properties as a result of arrangement or sheer scale. Chemistry is full of examples. Oxygen is a gas. Hydrogen is a gas. But when they combine together in the right way, they produce water, which is a liquid — and one with all kinds of properties that can’t be predicted from first principles through analyzing the constituent parts.

Are thoughts emergent?

Since we don’t know exactly what thoughts are, we cannot say for sure whether they are emergent phenomena or not. But we can indirectly infer that they are by considering properties of thoughts and comparing them with properties of chemicals being tossed around in the brain.

A hallmark of thoughts is that they are about things. When you are thinking about a pineapple, there is an “aboutness” relation between the thought and the pineapple. Thoughts refer to things — which may be real things in the world, or imaginary things like dragons. This is a distinctive feature of mental phenomena, and the philosophers call it intentionality. (Note that intentionality has nothing to do with intentions or motivations — it’s not the best term, but that’s where you’ll find the relevant writings.)

This “aboutness” or “intentionality” is not a feature of chemical tossing patterns. A pattern is a pattern is a pattern, and isn’t intrinsically about any other pattern. At the very least, we can say that modern physics and chemistry have had no reason to invent an “aboutness” concept so far. In other words, there is no purely physical theory of reference.

So it seems reasonable to at least consider the possibility that the property of “aboutness” emerges when matter is arranged in just the right way.

“Is there more to it?”

This admittedly abstract concept is not really going to satisfy people who were hoping that thoughts were actually composed of “magic dust”, as Sam Moss quite rightly termed it. Thoughts are not “made up” of some special secret sauce. If you look at a brain — or any other tissue — under a microscope, all you see are cells. And cells are made up of atoms — mostly carbon, hydrogen and oxygen, with some crucial cameos by nitrogen, calcium, phosphorus, sulfur, sodium, potassium, magnesium and choride.

So is that all a brain or a body is? A stew of a dozen elements? If you followed the story with carbon, then you’ll know that the answer is no. The arrangement of the atoms makes all the difference in the world.

But does this mean that “there is more to it”? If “more” implies a substance of which thoughts are made, then the answer is most likely no.

In any case, given that matter makes up everything, saying that something is “just” matter seems a bit unfair to matter — it’s about as magical a dust as you could possibly hope for!

Matter gives you the universe and you ask if there is more to it?! 😉

If you like, you can call arrangement or form the special “something more”. Arrangement is the “something more” that distinguishes charcoal from diamonds, and thought from nonsense.

But arrangement will not fulfill all the duties of magical dust. It is not the same as the traditional notion of a soul. A soul can live on without a body. But a form has no meaning without the constituent matter that is arranged.

So here’s the compromise: thoughts are made up of electro-chemical signals tossing around, but that is not what they are, since this definition does not distinguish in any useful way between thoughts and perceptions, feelings, moods, emotions or sensations — or even unconscious neural processes for that matter — all of which are also made up of electro-chemical signals.

So saying thoughts are electro-chemical signals is about as useful as saying diamonds are carbon. It’s true, but not in an especially interesting or informative sense.


If you’ve made it this far, well done! I guess I was having a slow Friday evening! 🙂

I know that what I’ve written is quite abstract, but that goes with the territory if you are thinking about thoughts.

Here are some answers that may be of interest:


This post was originally a Quora answer.

Yanny or Laurel? A perspective from the science of mind and brain

I really like the Yanny versus Laurel meme, which exploded yesterday. It helps illustrate some key points about human perception:

  1. In some situations people can differ wildly in their experience of low-level perception.
  2. Active top-down expectations (and other, weirder processes) have a strong effect on low-level perception.

So basically, it’s an auditory version of #ThatDress.

Continue reading

What are emotions?

What indeed?

My research is on cognitive-emotional interaction, so I suppose I am qualified to answer this question. 🙂

But my answer cannot be the answer, since there is actually no consensus among scientists concerning the definition of emotions.

[Illustration of grief from Charles Darwin’s book The Expression of the Emotions in Man and Animals.]

Continue reading

Dopamine is not the “feel good” molecule (and the very concept of a feel good molecule is meaningless)


Dopamine is not the feel good molecule or the basis of pleasure. The idea that any molecule considered in isolation could be the basis of a subjective experience is basically nonsense.

For people who can’t really reason through this idea, there is plenty of experimental evidence showing the complexity of each and every “celebrity” neurochemical — dopamine, serotonin, oxytocin, and so on.

Continue reading