How reliable are psychiatric diagnoses?

DSM-5 Cover.png

This post, which mostly provides useful citations, was originally written as an answer to the following Quora question:

Is ADHD one of the only psychiatric conditions that can be diagnosed objectively?

It really depends on what you mean by “objective”, but the answer is “probably not”.

Since we do not understand the underlying causes of ADHD — or any major psychiatric disorder — we diagnose them based on clusters of symptoms.

In the United States and several other countries, a large number of psychiatrists use a book called Diagnostic and Statistical Manual of Mental Disorders (DSM).

DSM (now in it’s 5th, revised edition, DSM-5) essentially uses a system of checklists to enable a clinician to assess if a person has a given disorder. This is a controversial book for various reasons, but for now it is what most psychiatrists use.

Instead of the complex philosophical question of ‘objectivity’, the usefulness of DSM can be assessed using statistical measures of ‘reliability’.

Given that one clinician uses DSM-5 to give the diagnosis of ADHD, how likely is another clinician to do so using the DSM-5? Measures of “test-retest reliability” capture this probability.

Here is a paper that explains the statistical measurement of reliability in some detail:

DSM-5: How Reliable Is Reliable Enough? [pdf]

There are conflicting reports on the reliability of DSM-5, but here is one paper that reports statistical assessments:

DSM-5 field trials in the United States and Canada, Part II: test-retest reliability of selected categorical diagnoses.

“There were a total of 15 adult and eight child/adolescent diagnoses for which adequate sample sizes were obtained to report adequately precise estimates of the intraclass kappa. Overall, five diagnoses were in the very good range(kappa=0.60–0.79), nine in the good range(kappa=0.40–0.59), six in the questionable range (kappa = 0.20–0.39), and three in the unacceptable range (kappa values,0.20). Eight diagnoses had insufficient sample sizes to generate precise kappa estimates at any site.”


“Two were in the very good (kappa=0.60–0.79) range: autism spectrum disorder and ADHD.”

For more on the quantity reported here, kappa, see this paper:

Interrater reliability: the kappa statistic

The quantity kappa ranges from 0 to 1. Zero means that there was no agreement between raters (clinicians in this case), and 1 means there was perfect agreement.

As I said before, the DSM is controversial — and not just because of reliability issues. Here is a sampling of papers and popular articles on the general topic:

Academic articles

Reliability in Psychiatric Diagnosis with the DSM: Old Wine in New Barrels

“However, the standards for evaluating κ-statistics have relaxed substantially over time. In the early days of systematic reliability research, Spitzer and Fleiss [4] suggested that in psychiatric research κ-values ≥0.90 are excellent; values between 0.70 and 0.90 are good, while values ≤0.70 are unacceptable. In 1977, Landis and Koch [5] proposed the frequently used thresholds: values ≥0.75 are excellent; values between 0.40 and 0.75 indicate fair to good reliability, and values ≤0.40 indicate poor reliability. More recently, Baer and Blais [6] suggested that κ-values >0.70 are excellent; values between 0.60 and 0.70 are good; values between 0.41 and 0.59 are questionable, and values ≤0.40 are poor. Considering these standards, the norms used in the DSM-5 field trial are unacceptably generous.”

The Reliability of Psychiatric Diagnoses: Point—Our psychiatric Diagnoses are Still Unreliable

“Today, 26 years later, did the DSM system succeed in improving the reliability of psychiatric diagnoses? Two answers exist. The DSM did improve the reliability of psychiatric diagnoses at the research level. If a researcher or a clinician can afford to spend 2 to 3 hours per patient using the DSM criteria and a structured interview or a rating scale, the reliability would improve. [13] For psychiatrists and clinicians, who live in a world without hours to spare, the reliability of psychiatric diagnoses is still poor. [2,3]”

Diagnostic Issues and Controversies in DSM-5: Return of the False Positives Problem.

“The fifth revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) was the most controversial in the manual’s history. This review selectively surveys some of the most important changes in DSM-5, including structural/organizational changes, modifications of diagnostic criteria, and newly introduced categories. It analyzes why these changes led to such heated controversies, which included objections to the revision’s process, its goals, and the content of altered criteria and new categories. The central focus is on disputes concerning the false positives problem of setting a valid boundary between disorder and normal variation. Finally, this review highlights key problems and issues that currently remain unresolved and need to be addressed in the future, including systematically identifying false positive weaknesses in criteria, distinguishing risk from disorder, including context in diagnostic criteria, clarifying how to handle fuzzy boundaries, and improving the guidelines for “other specified” diagnosis.”

Popular articles

The DSM-5 Controversy

“You will need to display fewer and fewer symptoms to get labeled with certain disorders, for example Attention Deficit Disorder and Generalized Anxiety Disorder. Children will have more and more mental disorder labels available to pin on them. These are clearly boons to the mental health industry but are they legitimate additions to the manual that mental health professionals use to diagnose their clients?”

DSM 5 Is Guide Not Bible—Ignore Its Ten Worst Changes

“This is the saddest moment in my 45 year career of studying, practicing, and teaching psychiatry. The Board of Trustees of the American Psychiatric Association has given its final approval to a deeply flawed DSM 5 containing many changes that seem clearly unsafe and scientifically unsound. My best advice to clinicians, to the press, and to the general public – be skeptical and don’t follow DSM 5 blindly down a road likely to lead to massive over-diagnosis and harmful over-medication. Just ignore the ten changes that make no sense.”

Normal or Not? New Psychiatric Manual Stirs Controversy

“Among the flashpoints: Asperger’s disorder will be folded into autism spectrum disorder; grief will no longer exempt someone from a diagnosis of depression; irritable children who throw frequent temper tantrums can be diagnosed with disruptive mood dysregulation disorder. [Hypersex to Hoarding: 7 New Psychological Disorders]

“One prominent critic has been Allen Frances, a professor emeritus of psychiatry at Duke University who chaired the DSM-IV task force.

“Frances charges that through a combination of new disorders and lowered thresholds, the DSM-5 is expanding the boundaries of psychiatry to encompass many whom he describes as the “worried well.””


Perhaps neurogenesis doesn’t happen in adult humans?

A new study suggests that new neurons are not born very often in human adults.

Birth of New Neurons in the Human Hippocampus Ends in Childhood

“The lab’s new research, based on careful analysis of 59 samples of human hippocampus from UCSF and collaborators around the world, suggests new neurons may not be born in the adult human brain at all. The findings present a challenge to a large body of research which has proposed that boosting the birth of new neurons could help to treat brain diseases such as Alzheimer’s disease and depression. But the authors said it also opens the door to exciting new questions about how the human brain learns and adapts without a supply of new neurons, as in seen in mice and other animals.”

My labmates are all monkey neuroanatomists, and for years they have been skeptical about the neurogenesis narrative, particularly in primates. Another famous dissenter is Pasko Rakic. Read about his complaints in this Guardian article from 2012:

Does your brain produce new cells?


Are mental disorders the same as brain disorders? Maybe not!

I am currently reading an excellent paper that will be published in Behavioral and Brain Sciences soon. It raises some very important issues with popular conceptions of mental illness.

Brain disorders? Not really… Why network structures block reductionism in psychopathology research

These two figures capture some of the key points:

Here is the abstract:

“In the past decades, reductionism has dominated both research directions and funding policies in clinical psychology and psychiatry. However, the intense search for the biological basis of mental disorders has not resulted in conclusive reductionist explanations of psychopathology. Recently, network models have been proposed as an alternative framework for the analysis of mental disorders, in which mental disorders arise from the causal interplay between symptoms. In this paper, we show that this conceptualization can help understand why reductionist approaches in psychiatry and clinical psychology are on the wrong track. First, symptom networks preclude the identification of a common cause of symptomatology with a neurobiological condition, because in symptom networks there is no such common cause. Second, symptom network relations depend on the content of mental states and as such feature intentionality. Third, the strength of network relations is highly likely to partially depend on cultural and historical contexts as well as external mechanisms in the environment. Taken together, these properties suggest that, if mental disorders are indeed networks of causally related symptoms, reductionist accounts cannot achieve the level of success associated with reductionist disease models in modern medicine. As an alternative strategy, we propose to interpret network structures in terms of D. C. Dennett’s (1987) notion of real patterns, and suggest that, instead of being reducible to a biological basis, mental disorders feature biological and psychological factors that are deeply intertwined in feedback loops. This suggests that neither psychological nor biological levels can claim causal or explanatory priority, and that a holistic research strategy is necessary for progress in the study of mental disorders.”

Behavioral and Brain Sciences is one of the premier journals for “big thinking” in cognitive science and neuroscience, so it’s great to see these ideas there.

Fifty terms to avoid in psychology and psychiatry?

The excellent blog Mind Hacks shared a recent Frontiers in Psychology paper entitled “Fifty psychological and psychiatric terms to avoid: a list of inaccurate, misleading, misused, ambiguous, and logically confused words and phrases”.

As mentioned in the Mind Hacks post, the advice in this article may not always be spot-on, but it’s still worth reading. Here are some excerpts:

(7) Chemical imbalance. Thanks in part to the success of direct-to-consumer marketing campaigns by drug companies, the notion that major depression and allied disorders are caused by a “chemical imbalance” of neurotransmitters, such as serotonin and norepinephrine, has become a virtual truism in the eyes of the public […] Nevertheless, the evidence for the chemical imbalance model is at best slim […]  There is no known “optimal” level of neurotransmitters in the brain, so it is unclear what would constitute an “imbalance.” Nor is there evidence for an optimal ratio among different neurotransmitter levels.”

“(9) Genetically determined. Few if any psychological capacities are genetically “determined”; at most, they are genetically influenced. Even schizophrenia, which is among the most heritable of all mental disorders, appears to have a heritability of between 70 and 90% as estimated by twin designs”

“(12) Hard-wired. The term “hard-wired” has become enormously popular in press accounts and academic writings in reference to human psychological capacities that are presumed by some scholars to be partially innate, such as religion, cognitive biases, prejudice, or aggression. For example, one author team reported that males are more sensitive than females to negative news stories and conjectured that males may be “hard wired for negative news” […] Nevertheless, growing data on neural plasticity suggest that, with the possible exception of inborn reflexes, remarkably few psychological capacities in humans are genuinely hard-wired, that is, inflexible in their behavioral expression”

“(27) The scientific method. Many science textbooks, including those in psychology, present science as a monolithic “method.” Most often, they describe this method as a hypothetical-deductive recipe, in which scientists begin with an overarching theory, deduce hypotheses (predictions) from that theory, test these hypotheses, and examine the fit between data and theory. If the data are inconsistent with the theory, the theory is modified or abandoned. It’s a nice story, but it rarely works this way”

“(42) Personality type. Although typologies have a lengthy history in personality psychology harkening back to the writings of the Roman physician Galen and later, Swiss psychiatrist Carl Jung, the assertion that personality traits fall into distinct categories (e.g., introvert vs. extravert) has received minimal scientific support. Taxometric studies consistently suggest that normal-range personality traits, such as extraversion and impulsivity, are underpinned by dimensions rather than taxa, that is, categories in nature”

Lilienfeld, S. O., Sauvigné, K. C., Lynn, S. J., Cautin, R. L., Latzman, R. D., & Waldman, I. D. (2015). Fifty psychological and psychiatric terms to avoid: a list of inaccurate, misleading, misused, ambiguous, and logically confused words and phrases. Frontiers in Psychology, 6, 1100.