Looking for Results in All the Wrong Places: What Makes Feedback Work?

September 16, 2012 By scottdm Leave a Comment

As anyone knows who reads this blog or has been to one of my workshops, I am a fan of feedback. Back in the mid-1990’s, I began using Lynn Johnson’s 10-item Session Rating Scale in my clinical work. His book, Psychotherapy in the Age of Accountability, and our long relationship, convinced me that I needed to check in regularly with my clients. At the same time, I started using the Outcome Questionnaire (OQ-45). The developer, Michael Lambert, a professor and mentor, was finding that routinely measuring outcome helped clinicians catch and prevent deterioration in treatment. In time, I worked with colleagues to develop a set of tools, the brevity of which made the process of asking for and receiving feedback about the relationship and outcome of care, feasible.

Initial research on the measures and feedback process was promising. Formally and routinely asking for feedback was associated with improved outcomes, decreased drop-out rates, and cost savings in service delivery! As I warned in my blogpost last February, however, such results, while important, were merely “first steps” in a scientific journey. Most importantly, the research to date said nothing about why the use of the measures improved outcomes. Given the history of our field, it would be easy to begin thinking of the measures as an “intervention” that, if faithfully adopted and used, would result in better outcomes. Not surprisingly, this is exactly what has happened, with some claiming that the measures improve outcomes more than anything since the beginning of psychotherapy. Sadly, such claims rarely live up to their initial promise. For decades the quest for the holy grail has locked the field into a vicious cycle of hope and despair, one that ultimately eclipses the opportunity to conduct the very research needed to facilitate understanding of the complex processes at work in any intervention.

In February, I wrote about several indirect, but empirically robust, avenues of evidence indicating that another variable might be responsible for the effect found in the initial feedback research. Now, before I go on, let me remind you that I’m a fan of feedback, a big fan. At the same time, its important to understand why it works and, specifically, what factors are responsible for the effect. Doing otherwise risks mistaking method with cause, what we believe with reality. Yes, it could be the measures. But, the type research conducted at the time did not make it possible to reach that conclusion. Plus, it seemed to me, other data pointed elsewhere; namely to the therapist. Consider, for example, the following findings: (1) therapists did not appear to learn from the feedback provided by measures of the alliance and outcome; (2) therapists did not become more effective over time as a result of being exposed to feedback. In other words, as with every other “intervention” in the history of psychotherapy, the effect of routinely monitoring the alliance and outcome seems to vary by therapist.

Such results, if true, would have significant implications for the feedback movement (and the field of behavioral health in general). Instead of focusing on methods and interventions, efforts to improve the outcome of behavioral health practice should focus on those providing the service. And guess what? This is precisely what the latest research on routine outcome measurement (ROM) has now found. Hot off the press, in the latest issue of the journal, Psychotherapy Research, Dutch investigators de Jong, van Sluis, Nugter, Heiser, and Spinhoven (2012) found that feedback was not effective under all circumstances. What variable was responsible for the difference? You guessed it: the therapist–in particular, their interest in receiving feedback, sense of self-efficacy, commitment to use the tools to receive feedback, and…their gender (with women being more willing to use the measures). Consistent with ICCE’s emphasis on supporting organizations with implementation, other research points to the significant role setting and structure plays in success. Simon, Simon, Harris and Lambert (2011), Reimer and Bickman (2012), and de Jong (2012) have all found that organizational and administrative issues loom large in mediating the use and impact of feedback in care.

Together with colleagues, we are currently investigating both the individual therapist and contextual variables that enable clinicians to benefit from feedback. The results are enticing. The first will be presented at the upcoming Achieving Clinical Excellence conference in Holland, May 16-18th. Other results will be reported in the 50th anniversayry issue of the journal, Psychotherapy, to which we’ve been asked to contribute. Stay tuned.

REACHing the Next Level of Clinical Performance: What it Really Takes

September 1, 2012 By scottdm 2 Comments

Do any of these people look familiar? Well, of course, I’m the guy in the middle pointing. To my left is the rock and roll guitarist Joe Walsh. On my right is world-renowned, card mechanic Richard Turner. Why have I pictured myself sandwiched between these two? Because they are both inspiring examples of what can be accomplished when individuals push beyond the “tried and true,” to reach the next level of performance.

Back in June, I read an article about Joe Walsh in the Chicago Tribune. Buried deep in the piece was a brief biography of the guitarist that exemplifies what it takes to achieve excellence. Walsh, who is 67 years old, began playing back in the late 1960’s. Like many of his generation, he was inspired by the Beatles. One day, he was listening to the band’s song, “And Your Bird can Sing,” which contains a “ridiculously finger stretching George Harrison guitar solo.” According to the article, Walsh worked tirelessly until he mastered the riff. It was only years later, long after he’d become famous, that he met Ringo Starr. Walsh related the story to the drummer who “looked at Walsh like he was nuts.” Harrison, Ringo told him, had played two guitar parts separately and tracked them on top of each other and later tracked them together in the studio.

Good thing no one told Walsh the truth. As a result, he did what no one–even now–thought posssible.

On to Richard Turner. Unless you are into magic or gambling, this may be the first time you’ve ever heard of this person. Author Alex Stone, in his phenomenally fun and informative book, Fooling Houdini, describes him as “a card handler without equal, a man whose prowess with a deck borders on the supernatural.” The supernaturnal? Really? I would have deemd such praise so much more hype, typical of “hollywood” and the media, if I hadn’t meet Turner personally and seen him work. Simply put, there’s nothing he can’t do with a deck of cards.

Watch the brief video below filmed at this year’s “Training of Trainers” event in Chicago. At all training events, we bring in top performers to entertain, inspire, and inform participants about what it takes to achieve excellence.

Not bad eh? Especially when one considers that Turner is blind. And the video above is only the tiniest snippet of his performance. At one point, he dealt out hands of poker and black jack, asking audience members which position they would like to have dealt the winning cards. Sure enough, whatever position was called, luck struck there and only there. “Give me a number between 1 and 52,” he asked. Whatever number was called out, he cut the cards to that exact position in the deck. Did I mention he’s also a 6th degree black belt? Simply put, Turner is a performer that is always pushing the limits. Once he was cited for a driving motor cycle while blind! How does he do it? Practice. Yep, seventeen hours a day! For years, he slept with a deck of cards. Like Walsh, he persisted until he mastered moves that no one considered possible or, more accurately, no one ever even imagined.

So, what can mental health professionals do to REACH the next level of clinical performance? Over the last few years, together with my colleagues, we’ve been writing about the steps thrapists can take to achieving excellence. This year, I was privileged to summarize the current state of the research on the subject in a keynote address at the Psychotherapy Networker conference in Washington, DC. Here, for the first time, is “Part 1” of that address (the second half will follow soon). In it, I lay out what the evidence says it takes to excel.

Obesity Redux: The RFL Results and complex Nature of Truth and Science

August 28, 2012 By scottdm 2 Comments

Back in April, I blogged about research published by Ryan Sorrell on the use of feedback-informed treatment in a telephonically-divered weight management program. The study, which appeared in the journal Disease Management*, not only found that the program and feedback led to weight loss, but also significant improvements in distress, health eating behaviors (70%), exercise (65%), and presenteeism on the job (64%)–the latter being critical to employers who were paying for the service.

Despite these results, the post garnered no attention until four months later during the first week of August when three clinicians posted comments on the very same day–that’s the beauty of the web, a long memory and an even longer reach.

What can I say? I’m having to eat my hat (or, the bird on my shoulder is…). I learned a great deal from the feedback:

Despite having sourced the figure from the American Academy of Child and Adolescent Psychiatry, the claim that weight gain due to poor diet and a lack of exercise was responsible for 300,000 deaths was false. According to the comments, the figure is closer to 26,000, a mere 10% of the number claimed!
The same was true regarding the reported annual cost of obesity. The 100 billion dollar figure reported on the AACAP website is, I was told, “grossly inflated” and worse, missed the point. By focusing on BMI, the writer counseled, “we will have wasted money spent on the 51% of the healthy people who are deemed ‘unhealthy’ based on weight and the 18% unhealthy ones who are overlooked because their weight looks fine (see Wildman et al., 2008).”

Solid points both. Thankfully, one of the writers noted what was supposed to have been the main point of the post; namely, ” the importance of “practice-based” evidence” in guiding service delivery, “making clear that finding the ‘right’ or ‘evidence-based’ approach for obesity (or any problem for that matter) is less important than finding out “what works” for each person in need of help.”

I want to make sure readers have access to the results of the study because they are an impressive demonstration of what’s possible when the feedback is sought from and used to guide service to people “in care.” Weight loss aside, Ryan also reported significant improvements in distress, healthy eating behaviors (70%), exercise (65%), and presenteeism on the job (64%). All this by using two simple, 4-question scales.

*Sorrell, R. (September, 2007). Application of an Outcome-Directed Behavioral Modification Model for Obesity on a Telephonic, Web-based Platform.Disease Management, 10, Supplement 1, 23-26.

PS: An AP article that came out this last weekend and was discussed on NPR suggests the truth about the “weight of the nation” may be more complicated than either I or those who commented on my blog may realize. Among the many changes that have occured over the last decades, the piece declares, “Who are we? Fatter. The average woman has gained 18 pounds since 1990, to 160 pounds; the average man is up 16 pounds, to 196.” Hmm.