SCOTT D Miller - For the latest and greatest information on Feedback Informed Treatment

  • About
    • About Scott
    • Publications
  • Training and Consultation
  • Workshop Calendar
  • FIT Measures Licensing
  • FIT Software Tools
  • Online Store
  • Top Performance Blog
  • Contact Scott
info@scottdmiller.com 773.404.5130

Final Making Sense of Making Sense of Negative Research Results about Feedback Informed Treatment

February 19, 2020 By scottdm 21 Comments

toilet 2“Everyone understands how a toilet works, don’t you?” ask cognitive scientists Sloman and Fernbach.

The answer, according to their research, is likely no.  Turns out, peoples’ confidence in their knowledge far outstrips their ability to explain how any number of simple, every day items work — a coffeemaker, zipper, bicycle and yes, a toilet.   More troubling, as complexity increases, the problem only worsens.  Thus, if you struggle to explain how glue holds two pieces of paper together — and most, despite being certain they can, cannot — good luck accounting for how an activity as complicated as psychotherapy works.

So pronounced is our inability to recognize the depth of our ignorance, the two researchers have given the phenomenon a name: the “Illusion of Explanatory Depth.”  To be sure, in most instances, not being able to adequately and accurately explain isn’t a problem.  Put simply, knowing how to make something work is more important in everyday life than knowing how it actually works:

  • Push the handle on the toilet and the water goes down the drain, replaced by fresh water from the tank;
  • Depress the lever on the toaster, threads of bare wire heat up, and the bread begins to roast;
  • Replace negative cognitions with positive ones and depression lifts.

Simple, right?

Our limited understanding serves us well until we need to build or improve upon any one of the foregoing.  In those instances, lacking true understanding, I know itwe could literally believe anything — in the case of the toilet, a little man in the rowboat inside the tank makes it flush — and be just as successful.   While such apparent human frailty might, at first pass, arouse feelings of shame or stupidity, truth is operating on a “need to know” basis makes good sense.  It’s both pragmatic and economical.  In life, you cannot possibly, and don’t really need to know everything.

And yet, therein lies the paradox: we passionately believe we do.  That is, until we are asked to provide a detailed, step-by-step, scientifically sound accounting — only then, does humility and the potential for learning enter the picture.

When research on routine outcome monitoring (ROM) first began to appear, the reported impact on outcomes was astonishing.  Some claimed it was the most important development in the field since the invention of psychotherapy!  They were also quite certain how it worked: like a blood test, outcome and alliance measures enabled clinicians to check progress and make adjustments when needed.  Voila!

Eight years ago, I drew attention to the assertions being made about ROM, warning “caution was warranted. ” It was not a bold statement, rather a reasoned one.   After all, throughout the waning decades of the last millennium and into the present, proponents of cognitive (CT) and cognitive behavioral therapy (CBT) had similarly overreached, claiming not only that their methods were superior in effect to all others, but that the mechanisms responsible were well understood.  Both proved false.  As I’ve written extensively on my blog, CT and CBT are no more effective in head to head comparisons with other approaches.  More, studies dating back to 1996 have not found any of the ingredients, touted by experts as critical, necessary to success (1, 2, 3).

ToiletThat’s why I was excited when researchers Mikeal, Gillaspy, Scoles, and Murphy (2016) published the first dismantling study of the Outcome and Session Rating Scales, showing that using the measures in combination, or just one or the other, resulted in similar outcomes.  Some were dismayed by these findings.  They wrote to me questioning the value of the tools.  For me, however, it proved what I’d said back in 2012, “focusing on the measures misses the point.”  Figure out why their use improves outcomes and we stop conflating features with causes, and are poised to build on what most matters.

On this score, what do the data say?  When it comes to feedback informed treatment, two key factors count:

  1. The therapist administering the measures; and
  2. The quality of the therapeutic relationship.

As is true of psychotherapy-in-general, the evidence indicates that who uses the scales is more important that what measures are used (1, 2).  Here’s what we know:

  • Therapists with an open attitude towards getting feedback reach faster progress with their patients;
  • Clinicians able to create an environment in which clients provide critical (e.g., negative) feedback in the form of lower alliance scores early on in care have better outcomes (1, 2); and
  • The more time a therapists spend consulting the data generated by routinely administering outcome and alliance measures, the greater their growth in effectiveness over time.

In terms of how FIT helps, two lines of research are noteworthy:

  • In a “first of its kind” study, psychologist Heidi Brattland found that the strength of the therapeutic relationship improvedThe Therapeutic Relationship more over the course of care when clinicians used the Outcome and Session Rating Scales (ORS & SRS) compared to when they did not.  Critically, such improvements resulted in better outcomes for clients, ultimately accounting for nearly a quarter of the effect of FIT.
  • Brattland also found therapists, “significantly differed in the influence of … [FIT] on the alliance, in the influence of the alliance on outcomes, and the residual direct effect of [FIT] … posttreatment” (p. 10).  Consistent with other studies, such findings indicate routine measurement can be used to identify a clinician’s “growth edge” — what, where, and with whom — they might improve their ability to relate to and help the diverse clients met in daily work.  Indeed, the combination of FIT, use of aggregate data to identify personal learning objectives, and subsequent engagement in deliberate practice has, in the only study in history of psychotherapy to date, been shown to improve effectiveness at the individual practitioner level.

“Inch by inch, centimeter by centimeter,” I wrote back in 2012, “the results of [new] studies will advance our understanding and effectiveness.”  I’m hopeful that the discussion in this and my two prior posts (1, 2) will help those interested in improving their results avoid the vicious cycle of hope and despair that frequently accompanies new ideas in our field, embracing the findings and what they can teach us rather than looking for the next best thing.

Until next time,

Scott

Scott D. Miller, Ph.D.
Director, International Center for Clinical Excellence

P.S: The March Intensives are sold out.  Register now for the summer trainings to ensure your spot:
ICCE FIT Implementation Intensive 2020Picture1ICCE FIT Deliberate Practice Intensive 2020

Filed Under: Feedback Informed Treatment - FIT

Comments

  1. Derek Smith says

    February 20, 2020 at 1:41 pm

    As a therapist who has read superficially about FIT and is interested in deliberate practice, it has been my assumption that if I dug in further I would find that many of my worries about the application of the ORS and SRS were addressed in specific training with them. Perhaps some of the push-back I’ve seen you write about is due to a similarly shallow understanding?

    Reply
  2. Carlada Razmus says

    February 20, 2020 at 7:21 pm

    I really enjoyed the analogy of a toilet. This also really resonates with me “subsequent engagement in deliberate practice has, in the only study in history of psychotherapy to date, been shown to improve effectiveness at the individual practitioner level.” Kind of scary when you think about it.

    Reply
  3. Jelena Zulevic says

    February 20, 2020 at 9:08 pm

    I’m looking forward to your book! I’ve found out about FIT/ROM through your blog and I’m very eager to learn more, thank you for great posts!

    Reply
  4. Laura says

    February 20, 2020 at 9:28 pm

    Great stuff. Looking forward to listening to your new podcast.

    Reply
  5. Ryan Seedall says

    February 21, 2020 at 5:37 am

    I just listened to the podcast. It was a great reminder that deliberate practice is not a noun. I think it has been easy for me to treat it as a noun rather than verb. Also, I was intrigued by your initial statement of concern with deliberate practice focused on specific models. I would love to understand a bit more about this.

    Reply
  6. Rowan says

    February 21, 2020 at 12:10 pm

    Very excited to read the new book as soon as I can get my hands on it!

    Reply
  7. H Paul says

    February 21, 2020 at 6:20 pm

    I love reading and listening to your presentations. Your content is always thought provoking and interesting. Thanks for what you do.

    Reply
  8. Carmen Iuga says

    February 21, 2020 at 6:48 pm

    I find myself discussing with my colleagues over and over again about how psychotherapy works and what frustrates me the most is the lack of interest in this topic. I found that a majority of psychotherapists around me don’t even know or care about the common factors or the client’s preferences, not to mention that they are quite rigid in their views about what deliberate practice is. So, for me, your work is a mine gold and I’m thankful for your constant effort. I look forward to learn more about FIT.

    Reply
  9. Natasha says

    February 22, 2020 at 2:22 am

    “focusing on the measures misses the point.” Figure out why their use improves outcomes and we stop conflating features with causes, and are poised to build on what most matters.

    – I find the above quotes from the post to account for all things therapy. When the relationship is established, tools take a backseat. The combination of the relationship and more of what works for that individual is how we aide/bear witness to/guide progress in the lives of those we serve.

    Reply
  10. Maria says

    February 22, 2020 at 3:49 pm

    I appreciate these blog posts and your willingness to tackle some of the challenges that come up with FIT. Starting to use FIT more consistently in my practice has challenged me in ways I did not expect. It’s one thing to believe you’re really open to feedback and another thing to be faced with negative feedback and figure out how to learn from it gracefully! An uncomfortable, but necessary spot to be in for growth.

    Reply
  11. Kathryn Pusateri says

    February 22, 2020 at 4:49 pm

    Just pre-ordered your new book “Deliberate Practice”. Can’t wait til it arrives.

    Reply
  12. Jesse Carlock says

    February 22, 2020 at 5:52 pm

    Working at my growing edge is scary but the other side of this is excitement. I appreciate what you are saying and look forward to reading “Better Results” when it comes out.

    Reply
  13. madeorakouma Bahomda says

    February 24, 2020 at 6:08 pm

    I listened to you and Dr. Daryl Chow presentation about the “Psychotherapy’s Fatal Flaw and How to Fix It”. I gained a lot off insights on how we can become more effective by identifying our deficit and seeking coaching/supervision to overcome this deficit. Most of counselors may think only of their effectiveness ignoring their weakness.
    Thank, you for sharing.

    Reply
  14. cheryl korneychuk-waddy says

    February 25, 2020 at 2:38 am

    I was part of training of the trainers years ago, and I am constantly awed by the consistency of the messaging around outcomes-right up to your newest interview with Daryl Chow; it’s not about the measures. But it’s ALL about the measures :)

    Reply
  15. Alan Scheuermann says

    February 25, 2020 at 3:01 am

    Just listened to your podcast “Psychotherapy’s Fatal Flaw and How to Fix It”. 12 years later, after attending the 2008 workshop in Chicago, and after 16 years of utilizing the ORS/SRS and consistently incorporating feedback into my work, the essence of identifying growing edges out of each of the potential factors contributing to positive change and outcomes was never more clear. I’ve gotten lost on chasing down one or the other of the factors, but this synthesis was quite elegant in its practical simplicity, yet daunting in its own way. Thank you!!

    Reply
  16. Cathy Hasty says

    February 25, 2020 at 4:46 pm

    I have followed your work for years and look forward to the book. I was trained as a pastoral psychotherapist and clinical educator with the association for clinical pastoral education which included a baseline of 4 years full time residency and with about 9 hours a week of intensive group and individual supervision for my clinical practice as an educator for spiritual care and my role as a psychotherapist. I continued in many hours of individual supervision to achieve a certification that no longer exists; AAPC Diplomate. Our training included verbatim, co-visitation and tapes of our work. While I have learned many modalities and theories, the most important was the individual accurate and often very painful feedback on my individual ways to avoid, resist and obscure the relationships. I agree with you on so much of what you are saying which is subversive in psychotherapy. I hope your upcoming book and work will include a small section to promote local organizing of people who are interested in improving therapy, not just participating in the programs, trainings and consultation that you offer. Thank you for your work!

    Reply
  17. Kate Bothwell-Wendel says

    February 26, 2020 at 9:44 pm

    I just finished listening to the podcast “Psychotherapy’s Fatal Flaw and once again am in awe and so intrigued by your work. Years ago, I recall being taught about the importance of the relationship and now, as the podcast host noted, I get constant emails about if I just learn X technique, then somehow that will be the missing link that will make all the difference in my effectiveness with clients.
    I appreciated the analogy of the technique being the vehicle and the vehicle not going anywhere without first deciding what are we doing and where are we going. I am also trained as a coach and as I was reflecting on your discussions of the 4 pillars, (Individualized learning objectives, feedback, successive refinement, role of coach to guide) goal consensus, relationship factors, therapist factors and hope and expectancy, I realized that the coach training I took was quite helpful in exploring these areas and how rarely they have come up in any therapy training I have been to. Thanks to both of you for highlighting and researching their enormous value in the therapeutic relationship and outcomes!

    Reply
  18. Luis says

    March 4, 2020 at 11:48 pm

    I do enjoy your blog. I did enjoy Psychotherapy’s Fatal Flaw And How to Fix It.
    thank you for your work.

    Reply
  19. Mary Hosbrough says

    March 5, 2020 at 12:05 pm

    I just posted the comment I’m including below on Clearly Clinical blog, but wanted to share the impact of this podcast on some professional development decisions with you, because I love seeing you at Evolution of Psychotherapy, but I’m deciding my time and money will be better utilized in pursuing learning FIT, ROM, and Deliberate Practice this year and I appreciate the accessibility of the upcoming Deliberate Practice Book, the eLearning course, and I finally purchased the FIT manuals as a starting point. I truly believe 1) I have an open attitude about receiving feedback, 2) I can improve creating an environment in which clients provide critical feedback, and 3) I am willing to spend time consulting the data generated by routinely administering outcome and alliance measures (once I learn how to do that). I feel excited (and a little scared) and hopeful!

    comment about podcast- This podcast was so thought (and practice) provoking for me. Elizabeth’s comments in the interview about clinicians’ limited time really nailed it for me. I work for a nonprofit agency with too much paperwork and too little time (and money), and I often set goals that are within that realm of “clinical knowledge” Daryl described, such as “read this treatment manual about Adults with ADHD and start an ADHD group,” or even “go to Evolution of Psychotherapy conference to learn from experts in the field and choose area for professional development to work on until the next Ev of Psych.” I went to my first Evolution of Psychotherapy as a grad student in 2014 and then again in 2018 and have been trying to figure out the time and finances to go in 2020… I love that experience, but while listening to this I’m realizing I spend close to $2,000 (of my own money) attending that conference, all the while telling myself I can’t afford to attend a FIT or Deliberate Practice training in Chicago, and likely not even pursuing such training, worried about the time, money, and vulnerability of embarking on this journey. After listening to this podcast, I find myself rethinking my strategies. I just purchased the FIT manuals. I will preorder the Deliberate Practice book. I may have missed the contest for the e-learning course (?), but I’m so grateful this has been put on my radar as a way to pursue learning to use FIT. $179 will be affordable to me, and I feel more prepared to dedicate my time, energy, and finances in learning to use FIT, ROM, and Deliberate Practice. Thank you.

    Reply
    • scottdm says

      March 6, 2020 at 4:33 pm

      Mary, email me at: scottdmiller@talkingcure.com and I’ll send you a discount code to attend this summer’s trainings in Chicago!

      S

      Reply
  20. Wendy says

    March 7, 2020 at 7:21 am

    I’ve just started learning about FIT and I have to say that at first I was intimidated by the idea due to the possibility of receiving negative feedback. But I put ego to the side and dug deep into what the fear was and discovered that it’s more imposter syndrome than anything else. Now I’m of the mindset that FIT is a growth tool and can take someone from good to great.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

 

SEARCH

Subscribe for updates from my blog.

  

Upcoming Training

 ICCE FIT Intensive Training with Dr. Scott Miller
ICCE FIT Supervision Intensive Online Scott D Miller

FIT Software tools

FIT Software tools

NREPP Certified

HTML tutorial

LinkedIn

Topics of Interest:

  • Behavioral Health (110)
  • behavioral health (4)
  • Brain-based Research (2)
  • CDOI (14)
  • Conferences and Training (67)
  • deliberate practice (27)
  • Dodo Verdict (9)
  • Drug and Alcohol (3)
  • evidence-based practice (65)
  • excellence (61)
  • Feedback (38)
  • Feedback Informed Treatment – FIT (203)
  • FIT (25)
  • FIT Software Tools (12)
  • ICCE (26)
  • Implementation (7)
  • medication adherence (3)
  • obesity (1)
  • PCOMS (11)
  • Practice Based Evidence (38)
  • PTSD (4)
  • Suicide (1)
  • supervision (1)
  • Termination (1)
  • Therapeutic Relationship (8)
  • Top Performance (39)

Recent Posts

  • Making Sense of Client Feedback
  • Umpires and Psychotherapists
  • Augmenting the Two-Dimensional Sensory Input of Online Psychotherapy
  • Death of a Friend
  • The Cost of Caring

Recent Comments

  • Asta on The Expert on Expertise: An Interview with K. Anders Ericsson
  • Michael McCarthy on Culture and Psychotherapy: What Does the Research Say?
  • Jim Reynolds on Culture and Psychotherapy: What Does the Research Say?
  • gloria sayler on Culture and Psychotherapy: What Does the Research Say?
  • Joseph Maizlish on Culture and Psychotherapy: What Does the Research Say?

Tags

addiction Alliance behavioral health brief therapy Carl Rogers CBT cdoi common factors conferences continuing education denmark evidence based medicine evidence based practice Evolution of Psychotherapy excellence feedback feedback informed treatment healthcare holland icce international center for cliniclal excellence medicine mental health meta-analysis Norway NREPP ors outcome measurement outcome rating scale post traumatic stress practice-based evidence psychology psychometrics psychotherapy psychotherapy networker public behavioral health randomized clinical trial SAMHSA session rating scale srs supershrinks sweden Therapist Effects therapy Training