Feedback is NOT Enough: A Brief Update about the Empirical Evidence

September 25, 2018 By scottdm 1 Comment

The use of routine outcome monitoring (ROM) is on the rise. In the United States and abroad, regulatory bodies are actually mandating the gathering of outcome data as the new “standard of care.”

As agencies rush to implement–often at great cost in terms of time and money–the question remains: just how much does ROM contribute to improved retention and effectiveness?

Over 20 years ago, I began using outcome and alliance scales in my work as a therapist, asking clients at each visit to give me feedback about the qaulity of our relationship and their experience of progress. Eventually, together with colleagues, I developed two, brief measures: the Outcome and Session Rating Scales.

When studies using the scales began to appear in the literature, I was immediately concerned. In my opinion, the results were just “too good to be true.” First, the results were confounded by allegiance effects, having been done exclusively by people with a significant investment in the results. More to the point, however, I was worried that the studies focused on the measures rather than on therapists.

Soon, as I predicted, other studies appeared with far more modest results. And now, a meta-analysis of all studies using the ORS and SRS has been published, confirming that routinely measuring performance, improves outcome but not as much as reported in the original studies (viz., .27 versus .50).

For those involved in and advocating FIT (Feedback-Informed Treatment), this is an IMPORTANT study. It makes clear that when working feedback-informed, improving effectiveness requires more than the use of two measures. Indeed, it’s not really about the measures at all. Rather, it’s about therapists using feedback to identify opportunities for their own professional development.

As my colleague and fellow psychologist, Birgit Valla, is fond of saying, “A stopwatch will not make you a better runner. It’s not about the clock. It’s how you use the information to identify small, specific aspects of your performance that could be improved and then practicing.”

That’s what the team at ICCE and I have been exploring these past 7 years. The latest article summarizing that research was published just this week.

All the best,

Scott

Scott D. Miller, Ph.D.
Director, International Center for Clinical Excellence

P.S.: Registration for the Spring Intensives is open. Click on the links below to reserve your spot!

Science is Real (confusing)

September 17, 2018 By scottdm 9 Comments

The graphic above is a small sample of the many posts I encountered on social media last week. Obviously, science has a great deal of currency in public discourse.

Now, look at the bottom row. On the far left, we are told that drinking wine will help you live a longer life. On the right, the exact opposite claim is made: no level of alcohol consumption is safe.

Can anyone blame us for being confused? What is the truth? Isn’t that what science is supposed to help us sort out? Judging from the slogans printed on T-shirts, posters, and lawn signs, apparently so:

Science matters!
Science will not be silenced!
In science we trust!

Or, in the words of “Science Guy,” Bill Nye, “If you don’t believe in science, you are holding everyone back.”

How can one respond to that, except to say, “Ouch!”

And yet, at the risk of holding everyone back, I actually think much of the current confusion about what is and is not true comes precisely from believing in science. To me, its a bit like saying, “I believe in hammers.” Yes, each word makes sense, but the resulting sentence is absurd.

Science is not something to believe in or not. Like a hammer, it is merely a tool — one that, as the founder of American psychology, William James (1896), noted, is “first of all a certain dispassionate method.”

James then continued, offering a warning particularly suited to our media-saturated times, “To suppose that [science] means a certain set of results that one should pin one’s faith upon and hug forever is sadly to mistake its genius, and degrades the scientific body to the status of a sect.”

Real world science is a messy affair, with partial, inconclusive, and often contradictory results the norm rather than the exception. When done well and thoroughly understood, it can help tip the scales in one direction or another. Rarely, however, does it offer us a mirror of the universe.

Here’s a recent example from my own work. Are superior performers in sports, art, music, programming, and psychotherapy born or made?

About a decade ago, a slew of books and articles appeared boldly asserting, “Greatness isn’t born. It’s grown” (Coyle, 2009). Anyone, they promised, could accomplish anything if they just practiced long enough (Colvin, 2009; Gladwell, 2008; Shenk, 2010; Syed, 2010).

Then, in 2014, a group of researchers published a meta-analysis questioning the strength of the association between practice and performance. In a popular magazine , the banner for an article penned by one of the study’s authors even claimed the whole idea of improving performance via practice, “perpetuates a cruel myth” as it promotes the false belief, “people can help themselves to the same degree if they just try hard enough.”

What are we to believe?

Sorting out the seemingly contradictory results requires a deep dive into the literature: who did the studies, what questions did they ask, and how was the data analyzed? In other words, longer than the 2 – 4 minute “reads” promised in the social media posts pictured above. In fact, from the start to the publication of our new study on the subject, my co-investigators and I spent hundreds of hours spread out over a three year period examining the question. Here’s what we found:

The correlation researchers cited as demonstrating practice is “not as important as has been argued” (.35 [p. 1, Macnamara et al., 2014]) was greater than the association between mortality (e.g., death) and obesity (.13), excessive drinking (.21), and taking prescribed medications correctly (.23).

When the data set was reanalyzed including only those studies judged by independent, blind raters to be bona fide instances of research on the link between practice and performance, the correlation increased to.40.

So, you decide: if you want to improve your effectiveness –as a pole vaulter, chess player, surgeon, or psychotherapist — should you practice? Please share your thoughts below.

WAIT! Three new science posts just came across my social media feed:

What to do?

Start talking to my dog. Check!
Begin my three day fast. Check!
Nah, I’m just going to watch TV.

All the best,

Scott

Scott D. Miller, Ph.D.
Director, International Center for Clinical Excellence

Better Results through Deliberate Practice

January 16, 2018 By scottdm 1 Comment

The legendary cellist Pablo Casals was once interviewed by comedian George Carlin. When asked why, at age 93, he continued to practice three hours a day, Casals replied, “I’m beginning to show some improvement!”

Hard not to feel inspired and humbled by such dedication, eh? And while humorous, Casals was not joking. Across a wide variety of domains (e.g., sports, computer programming, teaching), deliberate practice leads to better results. Indeed, our recent study of mental health practitioners documented a growth in effectiveness consistent with performance improvements obtained by elite atheletes.

The January issue of the APA monitor includes a detailed article on the subject. Staff writer Tori DeAngelis lays out the process of applying deliberate practice strategies to clinical work in clear, step-by-step terms. Best of all, it’s free–even continuing education credits are available if you need them.

As mentioned in the article, each summer the International Center for Clincal Excellence sponsors a two-day, intensive training on deliberate practice for therapists. Daryl Chow, Ph.D. and I will be teaching together, presenting the latest scientific and practical information from our forthcoming book, Better Results: Using Deliberate Practice to Improve Therapeutic Effectiveness (APA, 2019).

As in prior years, we promise you will be participating in an intimate, cutting-edge, and highly-personalized learning experience. Many practitioners return to year after year. “I’ve attended the Deliberate Practice Intensive for three years in a row,” says therapist Jim Reynolds, “because there is such a warm camraderie. We are all trying to do the best we can with our clients, but we go beyond that. To do that, I need contact with others who are striving to do better.”

Until next time,

Scott

Scott D. Miller, Ph.D.