Concerns About Resiliency Training
“Extraordinary claims require extraordinary evidence.”
-Carl Sagan
It would be absolutely amazing if there were a proven, scalable tool to prevent things like anxiety, depression, PTSD, or suicide without the need for actual clinical psychologists. It would be even better if people could be trained to deliver it in as little as a couple of weeks. Unfortunately, most professionals and researchers in that space would tell you it’s more complicated than that. Most, but not all. A handful of enterprising psychologists are willing to make those claims, and they’re the ones who win contracts with organizations ranging from school districts to the military.
Many of the following arguments are derived from journalist Jesse Singal’s book The Quick Fix: Why Fad Psychology Can’t Cure Our Social Ills. The book addresses eight different examples of seemingly overhyped pop psychology, including a chapter on Comprehensive Soldier Fitness, which includes Master Resilience Trainer. A version of that chapter was also published as a standalone article in The Chronicle of Higher Education under the title Positive Psychology Goes to War: How the Army Adopted an Untested, Evidence-Free Approach to Fighting PTSD.
Cognitive Behavioral Therapy (CBT) is a proven form of psychotherapy that focuses on identifying and then challenging cognitive distortions. Some better-known examples include all-or-nothing thinking, catastrophizing, and discounting the positive. While it is certainly possible for non-professionals to learn and apply the ability to recognize cognitive distortions (Greg Lukianoff and Jonathan Haidt offer some useful resources here) professional cognitive behavioral therapists generally have at least a master's degree along with requirements for supervised hours. These professionals must meet state licensure requirements to practice, and the leading certifications require years of experience.
As professor and historian Daniel Horowitz noted in his book Happier early positive psychologists worried about “the dangers of popularization,” where theoretical claims might be marketed as proven methods without sufficient evidence. There is perhaps no greater example of popularization than US military leadership’s embrace of resiliency training with essentially no evidence that it was capable of achieving what they wanted it to (namely, solving a dual crisis of PTSD and suicide).
The Master Resilience Trainer (MRT) program that has been embraced by the Army is based on the Penn Resilience Program (PRP) popularized by Dr. Martin Seligman’s Positive Psychology Center at the University of Pennsylvania. Despite many strong claims made by the center, several rigorous analyses raise questions about the program’s effectiveness, especially the core aspect that the trainings are usually conducted not by licensed mental health professionals but by non-professionals trained in just a few days.
A meta-analysis co-written by the creator of PRP came to the conclusion that “Future PRP research should examine whether PRP's effects on depressive symptoms lead to clinically meaningful benefits for its participants, whether the program is cost-effective, whether CB skills mediate program effects, and whether PRP is effective when delivered under real-world conditions.” This is hardly the full-throated endorsement you would expect from Jane Gillham, who originally developed the program.
A follow-up analysis published in the Journal of Adolescence reviewed nine controlled PRP trials from Australia, the Netherlands, and the US, ultimately coming to an even harsher conclusion: “No evidence of PRP in reducing depression or anxiety and improving explanatory style was found. The large-scale roll-out of PRP cannot be recommended. The content and structure of universal PRP should be re-considered.”
Knowing that MRT was built on a foundation of PRP is also troubling because the aims of the two programs are significantly different. PRP was designed for use among 10 to 14-year-old students, usually delivered to groups of 6 to 15 students at a time over a total of roughly 20 hours. It aimed to build skills among healthy young people that would help prevent depression and anxiety later on. MRT, on the other hand, was established specifically to address PTSD and suicide among soldiers. It is most frequently delivered to large audiences of soldiers as an hour's worth of mandatory training annually.
Prior to Penn’s partnership with the Army, no one involved in PRP had made any claims related to PTSD or suicide. Nick Brown, a scientific integrity researcher at Linnaeus University, laid out the problem in stark terms in his critical review of the program: “The idea that techniques that have demonstrated, at best, marginal effects in reducing depressive symptoms in school-age children could also prevent the onset of a condition that is associated with some of the most extreme situations with which humans can be confronted is a remarkable one that does not seem to be backed up by empirical evidence.”
So, a decade and a half since the initial implementation of MRT, what evidence has the Army produced for the program’s effectiveness? The Comprehensive Soldier Fitness team’s non-peer-reviewed, internally developed series of reports reached some positive conclusions, but these results have been heavily criticized by outside experts. Clinical Psychologists Roy Eidelson and Stephen Soldz highlighted ethical, empirical, and methodological concerns in a working paper and issued a call for retraction of the Army’s report. Even internal Army publications reached the conclusion that there were “significant unmitigated biases that threaten the present CSF program’s validity.”
In a 2014 Institute of Medicine report on military resiliency and prevention programs, the authors synthesized results from both these internal and external reviews, finding of CSF that “this committee does not find these results meaningful, given the low level of improvement and the very small effect size.” Of the fourth and final technical report released by the CSF team, the reviewers found that “Findings revealed no change in the GAT factors and no difference in diagnosis among those receiving the intervention. Therefore, the subsequent mediation analysis performed by the authors cannot be interpreted as evidence of intervention/program impact.”
For impacts this small, how much does the program cost? The initial cost was announced as $125 million (including a $31 million no-bid contract with Penn, not counting the money required to operate or sustain the program). Firm figures have been difficult to come by since then, but an Army spokesperson stated in 2017 that CSF cost $43.7 million the previous year. A report from 2013 described it as a $50 million per year program, slated to expand to $75 million per year by 2019.
This program has trained millions of soldiers over more than a decade. How much more training, and how many more millions of dollars do we need to invest before the lack of measurable results becomes unacceptable?