SCOTT D Miller - For the latest and greatest information on Feedback Informed Treatment

  • About
    • About Scott
    • Publications
  • Training and Consultation
  • Workshop Calendar
  • FIT Measures Licensing
  • FIT Software Tools
  • Online Store
  • Top Performance Blog
  • Contact Scott
info@scottdmiller.com 773.404.5130

Making Sense of Negative Research Results about Feedback Informed Treatment

January 16, 2020 By scottdm 10 Comments

blindfolded-professionalA ship’s captain who successfully sails through a strait at night learns nothing, and adds nothing, to their knowledge of the world.

(Please hang with me.  I promise this post will not be a long, metaphysical rant).

Returning to the example.  As paradoxical as it may strike one at first blush, a captain whose ship founders on the rocks while sailing through the strait both learns and adds to their knowledge.  As philosopher Ernst von Glasersfeld once opined, “The only aspect of that ‘real’ world that actually enters into the realm of experience is its constraints.”dart

The principle identified by von Glasersfeld applies not only to life lessons, but also to scientific advancement and, of course, feedback informed treatment (FIT).  Indeed, identifying and learning from “constraints” — that is, when things go wrong — is the very purpose of FIT.

It’s why, for example, when a client refuses to complete the outcome and alliance measures, my first impulse is to “lean in” and explore their reasons, rather than instantly set the scales aside.   It’s also why I’m most intrigued by studies which find that FIT fails to improve outcome (1, 2).  In both instances, my curiosity is piqued.  “Finally,” I think, “a chance to learn something …”.   Doing so, cognitive science has long shown, is not as easy or straightforward as simply adjusting our beliefs in light of new facts.  Quite to the contrary.

blindersWe are prone to see what we expect, fit the “different” into our current way of viewing the world or ignore it altogether.  One brief example before turning attention to FIT (aka Routine Outcome Monitoring [ROM]).  For most of the history of the field, the failure to engage in and respond to psychological intervention has been attributed to a host of client variables (e.g., degree or type of dysfunction, poor attachment history, IQ, etc.).  Therapists, for their part, have been held accountable for making the correct diagnosis and administering the right treatment.

And yet, despite continuous growth in the size of the DSM, and number of treatment approaches, no improvement in the outcome of psychotherapy has occurred in the last 50 years — a fact I first talked publicly about in 2014 and which über-researchers James Prochaska and John Norcross finally acknowledged in the most recent issue of the American Psychologist.  While some have argued that the field’s flat outcomes indicate the effectiveness therapy has reached a natural limit, an alternate point of view is that we should consider looking beyond the current ways of thinking about what matters most in successful treatment.

On this score, one possibility has been staring the field ninetysevenin the face for decades: the impact of the individual therapist on outcome.  Research has long shown, for example, that who does the treatment contributes 5 to 9 times more to outcome than the type of therapy, psychiatric diagnosis, or client history.  The same body of evidence documents some practitioners are consistently more effective than others.  When researcher Scott Baldwin and colleagues looked into why, they found 97% of the difference was attributable to therapist variability in the alliance.  Said another way, more effective therapists are able to establish a strong working relationship with a broader and more diverse group of clients.  I hope you’re seeing new possibilities for improving effectiveness.  If you’re a regular reader of my blog, you already know my colleagues and I published the only study to date documenting that a focus on therapist development via routine outcome measurement, feedback, and deliberate practice improves both agency and individual practitioner outcomes.

Turning to FIT, in my first post of the New Year, I talked about the strong sense of “anticipointment” I felt when thinking about the future of our field.  A colleague from the Netherlands, Kai Hjulsted, responded, saying he’d been having the same feeling about FIT!  The source, he said, was a study by a Dutch researcher conducted in a crisis intervention setting which, “contrary to expectations,” found, “Patients with psychiatric problems and severe distress seeking emergency psychiatric help did not benefit from direct feedback.”

I was well aware of this study, having served on the researcher’s dissertation committee.  And over the last decade, multiple studies have been published showing little or no benefit from feedback (e.g., 1, 2, 3).

How to make sense of such findings?  Having spoken with numerous practitioners (and even some researchers), I can tell you the tendency is to fit the results into our current way of viewing the world.  So, seen through a traditional medicopsychiatric lens, the inevitable conclusion is FIT does not work with people with certain, specific diagnosis (e.g., severe distress, in crisis, or those with eating disorders).  Such a conclusion makes no sense, however, if the totality of evidence is considered.  Why?  Because the results are decidedly mixed.  Thus, in one study, FIT makes a difference with people in crisis, in another it does not.  With one group of “severely distressed” clients, feedback appears to make matters worse, with another, chances of improvement increase 2.5 times.

Making SenseWhat then can we conclude?

An answer begins to emerge as soon as we’re able to get beyond thinking of FIT as just one more in a long list of treatment methods rather than a fundamental, organizing principle of agency and practice culture.  As is hopefully obvious, learning to administer measurement scales takes little time.

Cultural change, by contrast, is a much longer process.  How long?  Norwegian researcher Heidi Brattland and colleagues found it took three years ongoing training and support to successfully implement FIT.  Had they stopped to evaluate, like all other studies to date, after an average of 4 hours of instruction, no impact on outcomes would have been recorded.

While its now clear that time and support are critical to keep-calm-there-s-more-to-comesuccessful implementation, these two variables alone are not sufficient to make sense of emerging, apparently unsupportive studies of FIT.  Addressing such findings requires we look at the type of design used in most research: the randomized controlled trial.   That I’ll do in my next post, in particular addressing two, top notch, well-executed studies many have assumed show FIT is not effective in psychological care for people with eating disorders and severe distress.

And so, as I asked at the outset, please “hang with me.”

Until next time,

Scott

Scott D. Miller, Ph.D.
Director, International Center for Clinical Excellence
March 2020

 

 

 

Filed Under: Feedback Informed Treatment - FIT

Comments

  1. Rachel Barbara-May says

    January 16, 2020 at 10:15 pm

    Hi Scott,
    I have been learning over many years about the challenge faced, in my service at least, in getting beyond thinking of FIT as just one more in a long list of treatment methods rather than a fundamental, organizing principle of agency and practice culture. Your blogs continue to help me keep my despair at bay and I regularly re-visit the learnings from spectacular implementation studies like Heidi’s to help manage my ‘anticipointment’. As I continue to try and help transform the agency and practice culture of a service that provides psychological care for people with eating disorders and also those in severe distress I am eagerly hanging in there with you, until next time…
    Rachel

    Reply
  2. Dianne Lobes says

    January 16, 2020 at 10:30 pm

    I understand the benefits of mistakes – I’ve learned much from mine. But I disagree that one learns “nothing” by going through the strait successfully at night. It depends on focus and awareness – if one is paying close attention, one learns how to go through the strait successfully at night, such that she can reprieve this behavior in the future. That’s hardly nothing.

    Reply
  3. Chris. Potter says

    January 16, 2020 at 10:34 pm

    I appreciate you non-defensive response to the ‘allegation’ that FIT does not work with certain populations (as listed). It is a delight to see your ongoing effort at integrating practice, continuous outcome feedback, and research. During my days on Oregon’s Evidence Based Practice committees, I helped move the discussion from ‘process’ (EBP) focus to an ‘outcomes’ focus. I credit Joel Weinberger, George Stricker, and Dr Beutler for providing me with encouragement in doing this. This linked publication is a wonderful body of work that identifies the complexity of common factors (etc) that affect outcomes more readily than any specific therapeutic technique. Keep up the good work! https://psycnet.apa.org/record/2006-10050-004

    Reply
  4. Chris. Potter says

    January 16, 2020 at 10:40 pm

    https://www.researchgate.net/publication/14639910_The_Local_Clinical_Scientist_A_Bridge_Between_Science_and_Practice

    Reply
  5. Lawrence Goh says

    January 17, 2020 at 2:21 am

    Very much agree on the shift in culture needed. Therapists are human just like clients, thus any fundamental change needs to undergo a process. For a team the supervisors and management need to show the way to “hang on”.

    Reply
  6. scott says

    January 17, 2020 at 5:09 am

    I’m intrigued, but wondering that FIT is intended for the therapist rather than the client, and what the response to lower feedback scores was during client contact. Next episode!

    Reply
  7. Jane says

    January 17, 2020 at 7:35 am

    The results of Dr. van Oenen’s study are surprising to me. I also work in a crisis setting (hospital emergency departments). I did a study of the rates of patient follow-up with psychiatry or psychotherapy after ED visits and found that it was dismally low (below 10%), even when appointments were set up to occur within 48-72 hours after the patient’s ED visit. So I knew that the one and only opportunity to help the patient might be the ED encounter, and which prompted me to see if the FIT model might be useful.

    The nature of the ED visit involves many interruptions for physicians, labs to be drawn, registration, meds provided, etc. , so a “session” has many breaks. At the resumption of each interval after the interruption, I would ask the feedback questions to ensure I was on track regarding understanding the patient’s problem and to learn if our talk was helpful or if a midstream adjustment was needed. With acutely psychotic or highly intoxicated patients or people seeking drugs or shelter, it often was not, but most lucid patients who were in acute distress expressed considerable relief when it was used.

    The best evidence is probably that the ED does not have the revolving door patterns with patients that some of my colleagues in other area EDs who use the traditional methods. The FIT model requires some adaptation for an ED because it takes more time than a screening and there are many people are involved in patient care doing different tasks that break up the consultation into segments, but my experience suggests that the FIT model has advantages for many patients that counseling without the feedback seems to lack.

    Of course a controlled study (unfortunately not possible) would provide the necessary data to assess effectiveness, but anecdotally, the FIT model seems to work better than previous methods I have tried.

    Reply
  8. Gary Sweeten says

    January 17, 2020 at 4:27 pm

    I like your writing style. It always pulls me into the narrative.

    Scandalous means “To make a person stumble.” I think your presentations are the opposite.

    Reply
  9. Matt E says

    January 25, 2020 at 10:19 pm

    Hello, I am working on a book on integrative/whole-person constructs of care for rehabilitation professionals. This is great and consistent with the understandings we are trying to present – focusing on relationship, attunement, co-regulation, and presence over throwing fix-it tools at folks. I was alerted to this blog by and ACT page. I am wondering if you can point me to the best journal/peer-reviewed reference for the specific quantitative statement that “who does the treatment contributes 5 to 9 times more to outcome than the type of therapy, psychiatric diagnosis, or client history.” Thank you!

    Reply
    • scottdm says

      January 27, 2020 at 10:10 pm

      Hopefully, the third time is a charm. I’ve replied twice, only to have what I’ve written disappear. Anyway, two sources:

      1. The Great Psychotherapy Debate
      2. The Heart and Soul of Change

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

 

SEARCH

Subscribe for updates from my blog.

  

Upcoming Training

Mar
17

Feedback Informed Treatment (FIT) Intensive ONLINE


Mar
22

FIT Supervision Intensive 2021 ONLINE


Mar
30

FIT SPRING CAFÉ


Aug
02

FIT Implementation Intensive 2021


Aug
04

Training of Trainers 2021

FIT Software tools

FIT Software tools

NREPP Certified

HTML tutorial

LinkedIn

Topics of Interest:

  • Behavioral Health (110)
  • behavioral health (4)
  • Brain-based Research (2)
  • CDOI (14)
  • Conferences and Training (67)
  • deliberate practice (27)
  • Dodo Verdict (9)
  • Drug and Alcohol (3)
  • evidence-based practice (65)
  • excellence (61)
  • Feedback (38)
  • Feedback Informed Treatment – FIT (204)
  • FIT (25)
  • FIT Software Tools (12)
  • ICCE (26)
  • Implementation (7)
  • medication adherence (3)
  • obesity (1)
  • PCOMS (11)
  • Practice Based Evidence (38)
  • PTSD (4)
  • Suicide (1)
  • supervision (1)
  • Termination (1)
  • Therapeutic Relationship (8)
  • Top Performance (39)

Recent Posts

  • Developing a Sustainable Deliberate Practice Plan
  • Making Sense of Client Feedback
  • Umpires and Psychotherapists
  • Augmenting the Two-Dimensional Sensory Input of Online Psychotherapy
  • Death of a Friend

Recent Comments

  • Asta on The Expert on Expertise: An Interview with K. Anders Ericsson
  • Michael McCarthy on Culture and Psychotherapy: What Does the Research Say?
  • Jim Reynolds on Culture and Psychotherapy: What Does the Research Say?
  • gloria sayler on Culture and Psychotherapy: What Does the Research Say?
  • Joseph Maizlish on Culture and Psychotherapy: What Does the Research Say?

Tags

addiction Alliance behavioral health brief therapy Carl Rogers CBT cdoi common factors conferences continuing education denmark evidence based medicine evidence based practice Evolution of Psychotherapy excellence feedback feedback informed treatment healthcare holland icce international center for cliniclal excellence medicine mental health meta-analysis Norway NREPP ors outcome measurement outcome rating scale post traumatic stress practice-based evidence psychology psychometrics psychotherapy psychotherapy networker public behavioral health randomized clinical trial SAMHSA session rating scale srs supershrinks sweden Therapist Effects therapy Training