Are you any good as a therapist? The Legacy of Paul W. Clement

March 26, 2014 By scottdm 4 Comments

Twenty years ago, I came across an article published in the journal, Professional Psychology. It was written by a psychologist in private practice, Paul Clement. The piece caught my eye for a number of reasons. First, although we’d never met, Paul lived and worked in a town near my childhood home: Pasadena, California. Second, the question he opened his article with was provocative, to say the least, “Are you any good?” In other words, how effective are YOU as a psychotherapist? Third, and most important, he had compiled and was reporting a quantitative analysis of his results over the last 26 years as a practicing clinician. It was both riveting and stunning. No one I knew had ever done had published something similar before.

In graduate school, I’d learned to administer a variety of tests (achievement, vocational, personality, projective, IQ, etc.). Not once, however, did I attend a course or sit in a lecture about how to measure my results. I was forced to wonder, “How could that be?” Six years in graduate school and not a word about evaluating one’s outcomes. After all, if we don’t know how effective we are, how are any of us supposed to improve?

What was the reason for the absence of measurement, evaluation, and analysis? It certainly wasn’t because psychotherapy wasn’t effective. A massive amount of research existed documenting the effectiveness of treatment. Paul’s research confirmed these results. Of those he’d worked with, 75% were improved at termination. Moreover, such results were obtained in a relatively brief period of time, the median number of sessions used being 12.

Other results he reported were not so easy to accept. In short, Paul’s analysis showed that his outcomes had not improved over the course of his career. At the conclusion of the piece, he observed, “I had expected to find that I had gotten better and better over the years, but my data failed to suggest any systematic change in my therapeutic effectiveness across the 26 years in question…it was a bad surprise for me.” (p. 175).

For years, I carried the article with me in my briefcase, hoping that one day, I might better understand his findings. Maybe, I thought, Clement was simply an outlier? Surely, we get better with experience. It was hard for me to believe I hadn’t improved since my first, ham-handed sessions with clients. Then again, I didn’t really know. I wasn’t measuring my results in any meaningful way.

The rest is history. Within a few short years, I was routinely monitoring the outcome and alliance at every session I did with clients. Thanks to my undergraduate professor, Michael Lambert, Ph.D., I began using the OQ 45 to assess outcomes. Another mentor, Dr. Lynn Johnson had developed a 10-item scale for evaluating the quality of the therapeutic relationship, know as the Session Rating Scale. Both tools became an integral part of the way I worked. Eventually, a suggestion by Haim Omer, Ph.D., led me to consider creating shorter, less time consuming visual analogue versions of both measures. In time, together with colleagues, the ORS and SRS were developed and tested. Throughout this process, Paul Clement, and his original study remained an important, motivating force.

Just over a year ago, Paul sent me an article evaluating 40 years of his work as a psychotherapist. Once again, I was inspired by his bold, brave, and utterly transparent example. Not only had his outcomes not improved, he reported, they’d actually deteriorated! Leave it to him to point the way! As readers of this blog know, our group is busy at work researching what it takes to forestall such deterioration and improve effectiveness. Last year, we summarized our findings in the 50th Anniversary issue of Psychotherapy. As I write, we are preparing a more detailed report for publication in the same journal.

Yesterday, I was drafting an email, responding to one I’d recently received from him, when I learned Paul had died. I will miss him. In this, I know I’m not alone.

Looking for Results in All the Wrong Places: What Makes Feedback Work?

September 16, 2012 By scottdm Leave a Comment

As anyone knows who reads this blog or has been to one of my workshops, I am a fan of feedback. Back in the mid-1990’s, I began using Lynn Johnson’s 10-item Session Rating Scale in my clinical work. His book, Psychotherapy in the Age of Accountability, and our long relationship, convinced me that I needed to check in regularly with my clients. At the same time, I started using the Outcome Questionnaire (OQ-45). The developer, Michael Lambert, a professor and mentor, was finding that routinely measuring outcome helped clinicians catch and prevent deterioration in treatment. In time, I worked with colleagues to develop a set of tools, the brevity of which made the process of asking for and receiving feedback about the relationship and outcome of care, feasible.

Initial research on the measures and feedback process was promising. Formally and routinely asking for feedback was associated with improved outcomes, decreased drop-out rates, and cost savings in service delivery! As I warned in my blogpost last February, however, such results, while important, were merely “first steps” in a scientific journey. Most importantly, the research to date said nothing about why the use of the measures improved outcomes. Given the history of our field, it would be easy to begin thinking of the measures as an “intervention” that, if faithfully adopted and used, would result in better outcomes. Not surprisingly, this is exactly what has happened, with some claiming that the measures improve outcomes more than anything since the beginning of psychotherapy. Sadly, such claims rarely live up to their initial promise. For decades the quest for the holy grail has locked the field into a vicious cycle of hope and despair, one that ultimately eclipses the opportunity to conduct the very research needed to facilitate understanding of the complex processes at work in any intervention.

In February, I wrote about several indirect, but empirically robust, avenues of evidence indicating that another variable might be responsible for the effect found in the initial feedback research. Now, before I go on, let me remind you that I’m a fan of feedback, a big fan. At the same time, its important to understand why it works and, specifically, what factors are responsible for the effect. Doing otherwise risks mistaking method with cause, what we believe with reality. Yes, it could be the measures. But, the type research conducted at the time did not make it possible to reach that conclusion. Plus, it seemed to me, other data pointed elsewhere; namely to the therapist. Consider, for example, the following findings: (1) therapists did not appear to learn from the feedback provided by measures of the alliance and outcome; (2) therapists did not become more effective over time as a result of being exposed to feedback. In other words, as with every other “intervention” in the history of psychotherapy, the effect of routinely monitoring the alliance and outcome seems to vary by therapist.

Such results, if true, would have significant implications for the feedback movement (and the field of behavioral health in general). Instead of focusing on methods and interventions, efforts to improve the outcome of behavioral health practice should focus on those providing the service. And guess what? This is precisely what the latest research on routine outcome measurement (ROM) has now found. Hot off the press, in the latest issue of the journal, Psychotherapy Research, Dutch investigators de Jong, van Sluis, Nugter, Heiser, and Spinhoven (2012) found that feedback was not effective under all circumstances. What variable was responsible for the difference? You guessed it: the therapist–in particular, their interest in receiving feedback, sense of self-efficacy, commitment to use the tools to receive feedback, and…their gender (with women being more willing to use the measures). Consistent with ICCE’s emphasis on supporting organizations with implementation, other research points to the significant role setting and structure plays in success. Simon, Simon, Harris and Lambert (2011), Reimer and Bickman (2012), and de Jong (2012) have all found that organizational and administrative issues loom large in mediating the use and impact of feedback in care.

Together with colleagues, we are currently investigating both the individual therapist and contextual variables that enable clinicians to benefit from feedback. The results are enticing. The first will be presented at the upcoming Achieving Clinical Excellence conference in Holland, May 16-18th. Other results will be reported in the 50th anniversayry issue of the journal, Psychotherapy, to which we’ve been asked to contribute. Stay tuned.

The Impact of Mentors

July 20, 2010 By scottdm Leave a Comment

Brendan Madden Scott D. Miller Jeffrey K. Zeig

A little over month ago, I blogged about how the outcome and session rating scales were originally conceived of and developed. A few days prior to that, I wrote about where the whole idea of using measures to solicit feedback had started. In both instances, my teachers and supervisors played a significant role. Immediately following a two day workshop I’d given in Israel, psychologist Haim Omer suggested developing a visual analog scale that could be used in lieu of the longer Outcome Questionnaire 45.2–an idea that literally changed the entire arc of my professional career. Drs. Lynn Johnson and Michael Lambert–a supervisor and professor I met and worked with as a graduate student–were the first to pioneer feedback-informed treatment (FIT). Some twenty plus years into my career, I remain in contact with both, calling, seeking input, discussing ideas, and soliciting feedback.

“Professional coaches,” says the noted “expert on experts” K. Anders Erickson, “…play an essential role in guiding…future experts in a safe and effective manner” (p. 698). Needless to say, I’ve been very fortunate to have such visionary mentors. One more story.

In 1984, I wrote a letter to Dr. Jeffrey K. Zeig, the director of the Milton H. Erickson Institute. I was in my second year of a Ph.D. program in psychology and, like many graduate students, dead broke. While taking a course on hypnosis as part of my studies, I’d become interested in the work of Milton Erickson.

“I’d like to learn more,” I wrote at the time, “Would it be possible for me to visit the Institute, watch some videos and have a chance to talk with you?” I wasn’t too far away. I could drive to Phoenix where the Institute was located. I could even arrange to stay with friends to save money. “Dr. Zeig,” I continued, “I’m a graduate student and don’t have much money, but I’d be willing to do some work in kind.” I’d pasted mailing labels on thousands of brochures for the local hypnosis and therapy organizations, for example, in exchange for being able to attend professional continuing education events. “I’ll vacuum and clean the office, wash vehicles, do filing. Whatever might be helpful to you or the Institute.”

Within a couple of weeks, an envelop from the Milton H. Erickson Institute arrived. In it was a letter that was brief and to the point. “Please call me,” it said, and was signed Jeffrey K. Zeig, Ph.D. Needless to say, I called straight away. We chatted for a few minutes. He told me that I was welcome to visit the Institute, watch videos, talk with some of the staff and even spend some time with him. And then he asked, “Do you think you could afford five dollars?” I was floored.

Ever since meeting him on that hot summer day in Phoenix, he’s been an important teacher and mentor. It’s particularly noteworthy that whenever we talk–by phone, email, or in chance meetings on airplanes while criss-crossing the globe–he invariably asks, “What are you learning?” And then he listens, intently.

Last week, we were catching up on the phone and Jeff told me that his long-held desire to open an international psychotherapy training and research facility had finally been fulfilled. Briefly, The Institute for Applied Therapeutic Change is a real clinic where professionals and students can learn the latest in behavioral healthcare from leading experts in the field and while working with real clients (click on the text above for the complete press release).

“I can hardly wait to attend some of the events,” I said. “And when are you available to teach?” he responded. Stunned again. I’m so fortunate and can hardly wait to participate in the Institute activities as both a presenter and student. Stay tuned to the Foundation website for more details!

Feedback, Friends, and Outcome in Behavioral Health

May 1, 2010 By scottdm Leave a Comment

My first year in college, my declared major was accounting. What can I say? My family didn’t have much money and my mother–who chose my major for me–thought that the next best thing to wealth was being close to money.

Much to her disappointment I switched from accounting to psychology in my sophomore year. That’s when I first met Dr. Michael Lambert.

Michael J. Lambert, Ph.D.

It was 1979 and I was enrolled in a required course taught by him on “tests and measures.” He made an impression to be sure. He was young and hip–the only professor I met while earning my Bachelor’s degree who insisted the students call him by his first name. What’s more, his knowledge and passion made what everyone considered the “deadliest” class in the entire curriculum seem positively exciting. (The text, Cronbach’s classic Essentials of Psychological Testing, 3rd Edition, still sits on my bookshelf–one of the few from my undergraduate days). Within a year, I was volunteering as a “research assistant,” reading and then writing up short summaries of research articles.

Even then, Michael was concerned about deterioration in psychotherapy. “There is ample evidence,” he wrote in his 1979 book, The Effects of Psychotherapy (Volume 1), “that psychotherapy can and does cause harm to a portion of those it is intended to help” (p. 6). And where the entire field was focused on methods, he was hot on the trail of what later research would firmly establish as the single largest source of variation in outcome: the therapist. “The therapist’s contribution to effective psychotherapy is evident,” he wrote, “…training and selection on dimensions of…empathy, warmth, and genuineness…is advised, although little research supports the efficacy of current training procedures.” In a passage that would greatly influence the arc of my own career, he continued, “Client perception…of the relationship correlate more highly with outcome that objective judges’ ratings” (Lambert, 1979, p. 32).

Fast forward 32 years. Recently, Michael sent me a pre-publication copy of a mega-analysis of his work on using feedback to improve outcome and reduce deterioration in psychotherapy. Mega-analysis combines original, raw data from multiple studies–in this case 6–to create a large, representative data set of the impact of feedback on outcome. In his accompanying email, he said, “our new study shows what the individual studies have shown.” Routine, ongoing feedback from consumers of behavioral health services not only improves overall outcome but reduces risk of deterioration by nearly two thirds! The article will soon appear in the Journal of Consulting and Clinical Psychology.

Such results were not available when I first began using Lambert’s measure–the OQ 45–in my clinical work. It was late 1996. My colleagues and I had just put the finishing touches on Escape from Babel, our first book together on the “common factors.”

That’s when I received a letter from my colleague and mentor, Dr. Lynn Johnson.

Lynn D. Johnson, Ph.D.

In the envelop was a copy of an article Lynn had written for the journal, Psychotherapy entitled, “Improving Quality in Psychotherapy” in which he argued for the routine measurement of outcome in psychotherapy. He cited three reasons: (1) providing proof of effectiveness to payers; (2) enabling continuous analysis and improvement of service delivery; and (3) giving consumers voice and choice in treatment. (If you’ve never read the article, I highly recommend it–if for no other reason than its historical significance. I’m convinced that the field would be in far better shape now had Lynn’s suggestions been heeded then).

Anyway, I was hooked. I soon had a bootleg copy of the OQ and was using it in combination with Lynn’s Session Rating Scale with every person I met.

It wasn’t always easy. The measure took time and more than a few of my clients had difficulty reading and comprehending the items on the measure. I was determined however, and so persisted, occasionally extending sessions to 90 minutes so the client and I could read and score the 45-items together.

Almost immediately, routinely measuring and talking about the alliance and outcome had an impact on my work. My average number of sessions began slowly “creeping up” as the number of single-session therapies, missed appointments, and no shows dropped. For the first time in my career, I knew when I was and was not effective. I was also able to determine my overall success rate as a therapist. These early experiences also figured prominently in development of the Outcome Rating Scale and revision of the Session Rating Scale.

More on how the two measures–the OQ 45 and original 10-item SRS–changed from lengthy Likert scales to short, 4-item visual analog measures later. At this point, suffice it to say I’ve been extremely fortunate to have such generous and gifted teachers, mentors, and friends.