Nearly three years have passed since I blogged about claims being made about the impact of routine outcome monitoring (ROM) on the quality and outcome of mental health services. While a small number of studies showed promise, others results indicated that therapists did not learn from nor become more effective over time as a result of being exposed to ongoing feedback. Such findings suggested that the focus on measures and monitoring might be misguided–or at least a “dead end.”
Well, the verdict is in: feedback is not enough to improve outcomes. Indeed, researchers are finding it hard to replicate the medium to large effects sizes enthusiastically reported in early studies, a well-known phenomenon called the “decline effect,” observed across a wide range of scientific disciplines.
In a naturalistic multisite randomized clinical trial (RCT) in Norway, for example, Amble, Gude, Stubdal, Andersen, and Wampold (2014) found the main effect of feedback to be much smaller (d = 0.32), than the meta-analytic estimate reported by Lambert and Shimokawa (2011 [d = 0.69]). A more recent study (Rise, Eriksen, Grimstad, and Steinsbeck, 2015) found that routine use of the ORS and SRS had no impact on either patient activation or mental health symptoms among people treated in an outpatient setting. Importantly, the clinicians in the study were trained by someone with an allegiance to the use of the scales as routine outcome measures.
Fortunately, a large and growing body of literature points in a more productive direction. Consider the recent study by De Jong, van Sluis, Nugter, Heiser, and Spinhoven (2012), which found that a variety of therapist factors moderated the effect ROM had on outcome. Said another way, in order to realize the potential of feedback for improving the quality and outcome of psychotherapy, emphasis must shift away from measurement and monitoring and toward the development of more effective therapists.
What’s the best way to enhance the effectiveness of therapists? Studies on expertise and expert performance document a single, underlying trait shared by top performers across a variety of endeavors: deep domain-specific knowledge. In short, the best know more, see more and, accordingly, are able to do more. The same research identifies a universal set of processes that both account for how domain-specific knowledge is acquired and furnish step-by-step directions anyone can follow to improve their performance within a particular discipline. Miller, Hubble, Chow, & Seidel (2013) identified and provided detailed descriptions of three essential activities giving rise to superior performance. These include: (1) determining a baseline level of effectiveness; (2) obtaining systematic, ongoing feedback; and (3) engaging in deliberate practice.
I discussed these three steps and more, in a recent interview for the IMAGO Relationships Think Tank. Although intended for their members, the organizers graciously agreed to allow me to make the interview available here on my blog. Be sure and leave a comment after you’ve had a chance to listen!
Until next time,
Scott
Scott D. Miller, Ph.D.
www.whatispcoms.com
www.iccexcellence.com
Scott, have you ever found clients to be resistant to filling out the ORS and SRS at every single session? If so, what do you recommend to overcome that resistance?
I am confused. Isn’t this what FIT was all about? Are you changing your mind?
Hi Allyson, I think what Scott is impressing upon us is that what has been learned about Feedback Informed Therapy is that anything that is applied in a mechanical, manualized or routine way does not by itself improve outcomes. It is when all three components are applied as Scott wrote: “Miller, Hubble, Chow, & Seidel (2013) identified and provided detailed descriptions of three essential activities giving rise to superior performance. These include: (1) determining a baseline level of effectiveness; (2) obtaining systematic, ongoing feedback; and (3) engaging in deliberate practice.” Without the feedback component, we would not be alerted to any areas for improving outcome with this particular client or as a therapist in general, nor would we be able to identify areas for our own improvement. He is pointing out that research on exceptional performance is adding the deliberate practice component here, that feedback by itself offers little promise, but doing something intentional with that feedback is the very spirit of FIT. That is my understanding anyway, for what its worth. Hope that makes sense.
EXACTLY George! Thanks so much for your comment. Also, I’m working hard to counter the sole emphasis on measurement.
Thanks for the clarification!
I am also confused. Here, you seem to say feedback (e.g. ORS & SRS) is not significantly effective, yet in the interview with IMAGO you seem to say it is. Are you saying that feedback ALONE is not sufficient? If so, when have you suggested otherwise?? Would the take away then be: Continue to use systematic feedback tools, but know that these are not enough to make you a better therapist.
Thanks!
Timely article. Just looking at quality measures for service delivery and the use of attestations or consumer feedback and the frequency of using this form of measure. There is still apprehension by many in having the counselor or provider of the services survey the recipient of their services (bias). We have trusted the ORS due in part because as it is delivered it becomes part of the care and consumer investment in the care. I have rarely heard or seen anyone use the information to measure capacity or quality of the provider. It’s almost always interpreted as consumer readiness or engagement. (I recognize I may not be well informed in all the features of these rating tools, but ) I’m wondering about provider rating or evaluation by the provider themselves or the consumer. It has almost always been rejected for various reasons but is key to the point made in this article that to really impact improved services we have to improve the provider’s performance/service delivery. Interesting musings, although no less clear…. is there a next step?
Hi Scott,
I’m a little confused!
While I was not able to access the entire article by de Jong K1, van Sluis P, Nugter MA, Heiser WJ, Spinhoven P., I was able to read the abstract.
They seem to be concluding that feedback is not effective under all circumstances, and that therapist factors are important for not-on-track cases. Furthermore, when feedback was effective it appeared to be related to therapist factors such as, “internal feedback propensity, self-efficacy, and commitment to use the feedback.”
If you can, please explain what “internal feedback propensity” and “self-efficacy” mean. The reason I ask is that I would like to know how they are similar to (or different from) the factors you (and others) assert to be the three components critical for superior performance:
(1) determining a baseline level of effectiveness;
(2) obtaining systematic, ongoing, formal feedback; and
(3) engaging in deliberate practice.
It seems to me that “internal feedback propensity” (and even the drive for self-efficacy) may be more of a natural character trait or – for want of a better word – talent. If so, would that be contrary to your belief that “A fundamental finding of the research on superior performance is that talent is not a function of genetics…”?
Furthermore, where does the tendency to engage in deliberate practice come from?
I guess I’m not yet convinced that “top performers” do not come into the world with certain “gifts”.
Best!
Patrick
Hi Patrick:
Thanks for taking the time to reply. Our field has been in search of a method that improves outcomes for decades. Every inventor that comes along claims as much. Ultimately, and in each instance, the effectiveness of the tools are shown to vary depending on WHO uses them. Measurement is no different. Their helpfulness depends on the user. At present, researchers are working hard to understand what makes for an effective therapist. As George notes below, I’ve long maintained that administering measures is NOT enough. Still, hyperbolic claims continue, with some claiming they are the most important development in the history of the field since the very invention of therapy.
For my part, I am really excited about what we are learning. Finally out from under the search for specific methods for specific diagnoses, an entire new focus is being brought to the work. Chow’s brand new study shows that top performers spend more time engaged in deliberate practice. We are currently, along with a number of other researchers, exploring how to incorporate such findings into training of clinicians. For example, we know that more effective clinicians, form better alliances, so we have a research study going training therapists to respond empathically to difficult situations. Other researchers are looking at other variables, such as “internal feedback propensity,” or the willingness to seek and use feedback. While using different terms, the Chow study, along with others, identified similar findings. For example, more effective therapists evince more surprise in response to feedback, the precursor to learning. Said another way, if you are not surprised by feedback, you either: (1) didn’t hear it; or (2) did hear it, but did not consider it newsworthy. Anyway, stay tuned. And hey, join us in August 2016, for our professional development intensive. There, we lay out all of the current findings as well as help participants apply them to their practice: http://www.eventbrite.ie/e/fit-professional-development-intensive-2016-tickets-17740785166
The finding reflects my experience that just collecting data/obtaining feedback is not enough – it’s what the practitioner does with the information and the extent to which the practitioner engages in ‘deliberate practice’ – i.e. ‘Feedback…… informed.. practice’ and how the learning is applied.
Exactly Wendy. And this is why I’ve been warning about claims made by some advocates for nearly three years. The real risk is that research aimed at investigating what really matters will end once the headline grabbing claims about measurement are proven hyperbolic.
Unfortunately, the sound quality was poor on my system and I struggled to hear so gave up on the recording. However, in regard to the three essential activities, with which I wholeheartedly agree, in my experience, the following is a better description of the second….
“Obtaining systematic, ongoing feedback and exploring the findings, with the client, during therapy.” Whilst this may come under the umbrella of ‘deliberate practice’, I consider that it is so critical to good outcomes that it should be clearly identified.
I have been managing cases for upwards of ten years and, during that time have been privileged to work with outstanding practitioners. Unfortunately, I am, frequently, in the position of having to find a new therapist in a particular geographic location and, often, I have to speak with three or four individuals, or more, before finding someone who really understands (or is willing to learn) the value of feedback informed treatment. These days, many therapists assert that they utilise a number of outcome measurements but, in my experience, few truly understand – or even consider their value in therapy. Perhaps the message should be….. It’s not just the Data. Stupid! It’s what the data may be suggesting, that we can tease out during therapy. My practice was founded on the basis of being outcome informed and each practitioner is required to utilise the ORS/SRS instruments. At the outset, most practitioners were not familiar with the ORS/SRS or, indeed, any psychometric questionnaires. The disinterested were allowed to fall by the wayside and I worked closely with the others, coaching them to examine, carefully, the sometimes ‘hidden’ messages in the data and to explore these with the client during the session.
The practice specialises in the assessment and treatment of psychological trauma and we often work with clients who have sustained severe physical injuries and posttraumatic stress. For two years we have been working with survivors and family members of a major critical incident in Scotland, with the majority of the cases being handled by two highly effective therapists……..working from different orientations – one traditional CBT & EMDR, the other styles herself as a ‘Human Givens Therapist’.
The only common factors are the client population/presenting symptoms and the same set of psychometric questionnaires that include the ORS/SRS, which are introduced at every session. The outcome data reflect that most clients are enabled to move forward, achieving clinically reliable reductions in symptoms and, importantly, via the ORS/SRS, a corresponding increase in wellbeing, in 6 – 8 sessions.
Hi Scott
We have had a discussion about this recent blog via email, as I did not know how to access this ‘comment’ page previously. To try to summarise that discussion as briefly as I can from my perspective.
I think the blog generated some initial confusion as it seemed to to imply that our response to the feedback (ors/srs) in the room was not important, the ‘deliberate practice’ that took place between sessions was the only key to increasing effectiveness. I had a number of friends ask me if that was what you were saying. But reading the papers you were references made it clearer.
The feedback we make in the room is vitally important to keeping the therapy on track, especially when any indications arise that it might be going off track. The problem has been, as de Jong & co (2012) found, half of the therapists in their study did not use the feedback – and I see this, especially in those clinics where the tools have been mandated. There are numerous therapists who think the tools are going to do the work for them (or that they are just another damn form they have to get completed). Lets call that response in the room, 1st order responding – following systems theorists like Bateson or Argyris.
Now 2nd order responding is what I see Scott emphasising now in his study of expertise – the work we go and do outside the therapy room to improve our performance in it. And we can tie that to the tools by looking for patterns in our results – for example I number crunched my outcomes to see if I performed better in winter versus summer, or with men versus women, acting out vs internalising cases, etc. Continuing education, supervision, and personal therapy don’t have great stats to support them as pathways to greater effectiveness; but Scott tells us there are some longitudinal studies now underway which hopefully will show that when tied to ors/srs stats result in improved effectiveness.
I also discussed the Dreyfus brothers’ model of expertise development – theirs is more philosophically rooted as one is a Wittgensteinian scholar – and they give emphasis to how a shift from ‘know that’ knowledge to ‘know how’ occurs as expertise developed. It fits well with the extended, enactive, embodied cognition paradigm – the car and the road all feel like part of you when you drive. I pointed out the Dreyfuses had a chess grandmaster counting out loud whilst playing multiple games, and Scott noted they can also play blindfolded – so when you get there (expertise) its no longer ‘know that’ but now know how – ‘trust the force Luke’. But the research indicates that it takes a lot of know that study (or maybe just a lot of ‘know how’ practice – I’ve seen some “wounded healers” who are very effective in a narrow domain ‘cos life has taught them in a harsh environment: Scott may differ with me on that point?).
Anyway – we agree that as the expertise develops, through doing all this 2nd order work, we will be able to stay present in the room better dealing with this 1st order responding – getting the therapy back on track. We will be more skilful at ‘facilitative interpersonal skills’ (Anderson & Ogles 2009), or ‘resource activation’ (Gassmann & Grawe 2006) or in NZ Maori ‘manaakitanga’ – raising the mana of other.
Is that a fair summary of our discussion Scott?