Dealing with Scientific Objections to the Outcome and Session Rating Scales: Real and Bogus

December 15, 2012 By scottdm Leave a Comment

The available evidence is clear: seeking formal feedback from consumers of behavioral health services decreases drop out and deterioration while simultanesouly improving effectiveness. When teaching practitioners how to use the ORS and SRS to elicit feedback regarding progress and the therapeutic relationship, three common and important concerns are raised:

How can such simple and brief scales provide meaningful information?
Are consumers going to be honest?
Aren’t these measures merely assessing satisfaction rather than anything meaninful?

Recently, I was discussing these concerns with ICCE Associate and Certified Trainer, Dan Buccino.

Briefly, Dan is a clinical supervisor and student coordinator in the Adult Outpatience Community Psychiatry program at Johns Hopkins. He’d not only encountered the concerns noted above but several additional objections. As he said in his email, “they were at once baffling and yet exciting, because they were so unusal and rigorous.”

“It’s a sign of the times,” I replied, “As FIT (feedback informed treatment) becomes more widespread, the supporting evidence will be scrutinized more carefully. It’s a good sign.”

Together with Psychologist and ICCE Senior Associate and Trainer, Jason Seidel, Dan crafted detailed response. When I told them that I believed the ICCE community would value having access to the document they created, both agreed to let me publish it on the Top Performance blog. So…here it is. Please read and feel free to pass it along to others.

Implementation Science, FIT, and the Training of Trainers

March 8, 2012 By scottdm Leave a Comment

The International Center for Clinical Excellence (ICCE) is pleased to announce the 6th annual Training of Trainers event to be held in Chicago, Illinois August 6th-10th, 2012. As always, the ICCE TOT prepares participants provide training, consultation, and supervision to therapists, agencies, and healthcare systems in Feedback-Informed Treatment (FIT). Attendees leave the intensive, hands-on training with detailed knowledge and skills for:

Training clinicians in the Core Competencies of Feedback Informed Treatment (FIT/CDOI);
Using FIT in supervision;
Methods and practices for implementing FIT in agencies, group practices, and healthcare settings;.
Conducting top training sessions, learning and mastery exercises, and transformational presentations.

Multiple randomized clinical trials document that implementing FIT leads to improved outcomes and retention rates while simultanesouly decreasing the cost of services.

This year’s “state of the art” faculty include: ICCE Director, Scott D. Miller, Ph.D., ICCE Training Director, Julie Tilsen, Ph.D., and special guest lecturer and ICCE Coordinator of Professional Development, Cynthia Maeschalck, M.A.

Join colleagues from around the world who are working to improve the quality and outcome of behavioral healthcare via the use of ongoing feedback. Space is limited. Click here to register online today. Last year, one participants said the training was, “truly masterful. Seeing the connection between everything that has been orchestrated leaves me amazed at the thought, preparation, and talent that has cone into this training.” Here’s what others had to say:

The Impact of Mentors

July 20, 2010 By scottdm Leave a Comment

Brendan Madden Scott D. Miller Jeffrey K. Zeig

A little over month ago, I blogged about how the outcome and session rating scales were originally conceived of and developed. A few days prior to that, I wrote about where the whole idea of using measures to solicit feedback had started. In both instances, my teachers and supervisors played a significant role. Immediately following a two day workshop I’d given in Israel, psychologist Haim Omer suggested developing a visual analog scale that could be used in lieu of the longer Outcome Questionnaire 45.2–an idea that literally changed the entire arc of my professional career. Drs. Lynn Johnson and Michael Lambert–a supervisor and professor I met and worked with as a graduate student–were the first to pioneer feedback-informed treatment (FIT). Some twenty plus years into my career, I remain in contact with both, calling, seeking input, discussing ideas, and soliciting feedback.

“Professional coaches,” says the noted “expert on experts” K. Anders Erickson, “…play an essential role in guiding…future experts in a safe and effective manner” (p. 698). Needless to say, I’ve been very fortunate to have such visionary mentors. One more story.

In 1984, I wrote a letter to Dr. Jeffrey K. Zeig, the director of the Milton H. Erickson Institute. I was in my second year of a Ph.D. program in psychology and, like many graduate students, dead broke. While taking a course on hypnosis as part of my studies, I’d become interested in the work of Milton Erickson.

“I’d like to learn more,” I wrote at the time, “Would it be possible for me to visit the Institute, watch some videos and have a chance to talk with you?” I wasn’t too far away. I could drive to Phoenix where the Institute was located. I could even arrange to stay with friends to save money. “Dr. Zeig,” I continued, “I’m a graduate student and don’t have much money, but I’d be willing to do some work in kind.” I’d pasted mailing labels on thousands of brochures for the local hypnosis and therapy organizations, for example, in exchange for being able to attend professional continuing education events. “I’ll vacuum and clean the office, wash vehicles, do filing. Whatever might be helpful to you or the Institute.”

Within a couple of weeks, an envelop from the Milton H. Erickson Institute arrived. In it was a letter that was brief and to the point. “Please call me,” it said, and was signed Jeffrey K. Zeig, Ph.D. Needless to say, I called straight away. We chatted for a few minutes. He told me that I was welcome to visit the Institute, watch videos, talk with some of the staff and even spend some time with him. And then he asked, “Do you think you could afford five dollars?” I was floored.

Ever since meeting him on that hot summer day in Phoenix, he’s been an important teacher and mentor. It’s particularly noteworthy that whenever we talk–by phone, email, or in chance meetings on airplanes while criss-crossing the globe–he invariably asks, “What are you learning?” And then he listens, intently.

Last week, we were catching up on the phone and Jeff told me that his long-held desire to open an international psychotherapy training and research facility had finally been fulfilled. Briefly, The Institute for Applied Therapeutic Change is a real clinic where professionals and students can learn the latest in behavioral healthcare from leading experts in the field and while working with real clients (click on the text above for the complete press release).

“I can hardly wait to attend some of the events,” I said. “And when are you available to teach?” he responded. Stunned again. I’m so fortunate and can hardly wait to participate in the Institute activities as both a presenter and student. Stay tuned to the Foundation website for more details!

Implementing Consumer-Driven, Outcome-Informed (CDOI) Behavioral Health Services: The ICCE and 2010 Training of Trainers Event

June 8, 2010 By scottdm Leave a Comment

This week I’m in Calgary, Canada. Last week, I was in Charleston, South Carolina. Next week, I’ll be in Marion, Ohio and Bay City, Michigan. In each instance, I’m working with the management and staff of public behavioral health agencies that are busy implementing consumer-driven, outcome-informed clinical work.

Some of the groups are just beginning the process. Others, as reported here on my blog, have been at it long enough to document significant improvements in outcome, retention, and productivity (i.e., in Ohio and Virginia). All have told me that implementing the seemingly simple ideas of outcome-informed practice is incredibly hard work–impacting nearly every aspect of agency life. Being able to access the expertise and experience of fellow clinicians and agency directors in real time when questions and challenges arise is, I’ve also learned, critical in maintaining the momentum necessary for successful implementation.

Enter the ICCE: The International Center for Clinical Excellence. Briefly, the ICCE is a web-based community of clinicians, researchers, agency managers, and policy makers dedicated to excellence in behavioral health. Many of the groups I’m working with have joined the site providing them with 24/7/365 access to a deeply knowledgeable world-wide community. In addition to the numerous topic-specific discussion groups and member-generated videos, organizations can set up private forums where management and clinicians can have confidential discussions and coordinate implementation efforts.

If you are a clinician or agency director and are not already a member, you and/or your organization can access the ICCE community today by visiting the website at: www.centerforclinicalexcellence.com. Membership is free. In the video below, I talk with Arjan Van der Weijde, about groups in Holland that are meeting on on the ICCE for practitioners to discuss their implementation of feedback-informed work in the Netherlands. Check it out.

I’ve also included a brief video about the upcoming “Training of Trainers” course, held each year in August in Chicago. As in prior years, professionals from all over the world will be joining me and the state-of-the-art faculty for four intensive days of training. Agencies both public and private, in the U.S. and abroad, are sending staff to the event to learn the skills necessary to lead transformation projects. Space is already limited so register soon.

The Training of Trainers

Learning, Mastery, and Achieving One’s Personal Best

April 25, 2010 By scottdm Leave a Comment

Dateline: Sunday, April 25th, 2010 Chicago, IL

There’s a feeling I get whenever I’m learning something new. It’s a combination of wonder and possibility. Even though I’ve been traveling and teaching full time for over 18 years, I still feel that get that feeling of excitement whenever I step on a plane: What will I see? Who will I meet? What will I learn? Move over Indiana Jones, you’ve got nothing on me!

On my desk right now are stacks of books on the subject of expertise and expert performance: The Talent Code: Greatness Isn’t Born. It’s Grown. Here’s How, The Genius in All of Us: Why Everything You’ve Been Told About Genetics, Talent, and IQ Is Wrong, The Cambridge Handbook of Creativity, The Psychology of Abilities, Competencies, and Expertise, Why We Make Mistakes: How We Look Without Seeing, Forget Things in Seconds, and Are All Pretty Sure We Are Way Above Average, and many, many more.

On the floor, arranged in neat little piles, are reams of research articles, newspaper clippings, and pages torn out of magazines. Literally, all on the same subject: how can we clinicians reliably achieve better results?

I’ve never been one to “settle” for very long. It’s the journey not the destination I find appealing. Thus, I began exploring the common factors when it became clear that treatment models contributed little if anything to outcome (click here to read the history of this transition). When I became convinced that the common factors held little promise for improving results in psychotherapy, I followed the lead of two my mentors, professor Michael Lambert (who I worked with as an undergraduate) and psychologist Lynn Johnson (who trained and supervised me), and began measuring outcome and seeking feedback. Now that research has firmly established that using measures of the alliance and outcome to guide service delivery significantly enhances performance (see the comprehensive summary of research to date below), I’ve grown restless again.

In truth, I find discussions about the ORS and SRS a bit, well, boring. That doesn’t mean that I’m not using or teaching others to use the measures. Learning about the tools is an important first step. Getting clinicians to actually use them is also important. And yet, there is a danger if we stop there.

Right now, we have zero evidence that measurement and feedback improves the performance of clinicians over time. More troubling, the evidence we do have strongly suggests that clinicians do not learn from the feedback they receive from outcome and alliance measures. Said another way, while the outcome of each particular episode of care improves, clinicians overall ability does not. And that’s precisely why I’m feeling excited–the journey is beginning…

…and leads directly to Kansas City where, on October 20-22nd, 2010, leading researchers and clinicians will gather to learn the latest, evidence-based information and skills for improving performance in the field of behavioral health. As of today, talented professionals from Australia, Sweden, Norway, Denmark, Germany, England, Israel, and the United States have registered for the international “Achieving Clinical Excellence” conference. Some common questions about the event include:

1. What will I learn?

How to determine your overall effectiveness and what specifically you can do to improve your outcomes.

2. Is the content new?

Entirely. This is no repeat of a basic workshop or prior conferences. You won’t hear the same presentations on the common factors, dodo verdict, or ORS and SRS. You will learn the skills necessary to achieve your personal best.

3. Are continuing education credits available?

Absolutely–up to 18 hours depending on whether you attend the pre-conference “law and ethics” training. By the way, if you register now, you’ll get the pre-conference workshop essentially free! Three days for one low price.

4. Will I have fun?

Guaranteed. In between each plenary address and skill building workshop, we’ve invited superior performers from sports, music, and entertainment to perform and inspire . If you’ve never been to Kansas City, you’ll enjoy the music, food, attractions, and architecture.

Feel free to email me with any questions or click here to register for the conference. Want a peak at some of what will be covered? Watch the video below, which I recorded last week in Sweden while “trapped” behind the cloud of volcanic ash. In it, I talk about the “Therapists Most Likely to Succeed.”

Measures and feedback 2016 from Scott Miller

Is Professional Training a Waste of Time?

March 18, 2010 By scottdm 6 Comments

Every year, thousands of students graduate from professional programs with degrees enabling them to work in the field of behavioral health. Many more who have already graduated and are working as a social worker, psychologist, counselor, or marriage and family therapist attend—often by legal mandate—continuing education events. The costs of such training in terms of time and money are not insignificant.

Most graduates enter the professional world in significant debt, taking years to pay back student loans and recoup income that was lost during the years they were out of the job market attending school. Continuing professional education is also costly for agencies and individuals in practice, having to arrange time off from work and pay for training.

To most, the need for training seems self-evident. And yet, in the field of behavioral health the evidence is at best discouraging. While in traveling in New Zealand this week, my long-time colleague and friend, Dr. Bob Bertolino forwarded an article on the subject appearing in the latest issue of the Journal of Counseling and Development (volume 88, number 2, pages 204-209). In it, researchers Nyman and Nafziger reported results of their study on the relationship between therapist effectiveness and level of training.

First, the good news: “clients who obtained services…experienced moderate symptom relief over the course of six sessions.” Now the bad news: it didn’t matter if the client was “seen by a licensed doctoral –level counselor, a pre-doctoral intern, or a practicum student” (p. 206, emphasis added). The authors conclude, “It may be that researchers are loathe to face the possibility that the extensive efforts involved in educating graduate students to become licensed professionals result in no observable differences in client outcome” (p. 208, emphasis added).

In case you were wondering, such findings are not an anomaly. Not long ago, Atkins and Christensen (2001) reviewed the available evidence in an article published in the Australian Psychologist and concluded much the same (volume 36, pages 122-130); to wit, professional training has little if any impact on outcome. As for continuing professional education, you know if you’ve been reading my blog that there is not a single supportive study in the literature.

“How,” you may wonder, “could this be?” The answer is: content and methods. First of all, training at both the graduate and professional level continues to focus on the weakest link in the outcome chain—that is, model and technique. Recall, available evidence indicates that the approach used accounts for 1% or less of the variance in treatment outcome (see Wampold’s chapter in the latest edition of the Heart and Soul of Change). As just one example, consider workshops being conduced around the United States using precious resources to train clinicians in the methods studied in the “Cannabis Youth Treatment” (CYT) project–a study which found that the treatment methods used contributed zero to the variance in treatment outcome. Let me just say, where I come from zero is really close to nothing!

Second, and even more important, traditional methods of training (i.e., classroom lecture, reading, attending conferences) simply do not work. And sadly, behavioral health is one of the few professions that continue to rely on such outdated and ineffective training methods.

The literature on expertise and expert performance provides clear, compelling, and evidence-based guidelines about the qualities of effective training. I’ve highlighted such data in a number of recent blogposts. The information has already had a profound impact on the way how the ICCE organizes and conducts trainings. Thanks to Cynthia Maeschalck, Rob Axsen, and Bob, the entire curriculum and methods used for the annual “Training of Trainers” event have been entirely revamped. Suffice it to say, agencies and individuals who invest precious time and resources attending the training will not only learn but be able to document the impact of the training on performance. More later.

The Future of Behavioral Health: Integrated Care & Entrepreneurship

February 2, 2010 By scottdm Leave a Comment

Nicholas Cummings, Ph.D.

Sometime in late 1986 I wrote a letter to Dr. Nicholas Cummings. As a then student-member of the American Psychological Association (APA), I was automatically subscribed to and receiving the American Psychologist. In the April issue, Dr. Cummings published an article, provocatively titled, “The Dismantling of Our Health System: Strategies for the Survival of Psychological Practice.” Change was in the air. “Traditional psychology practice is both inefficient and ineffective,” he argued, and a “growing revolution in heath care” was a clear and present danger to the very survival of the field.

What can I say? As someone at the beginning of his career (with large student loans to repay and a long time to go before retirement), the five page article spooked me. So I did what I’d done before: I wrote a letter. I’d been writing and calling prominent researchers and clinicians ever since I was an undergraduate. During the 70’s, I’d exchanged letters and even phone calls with B.F. Skinner!

In all honesty, I didn’t know anything about Dr. Cummings–for example, that he was a former president of the APA, launched the Professional Schools of Psychology movement and founded four campuses of the California School of Professional Psychology, wrote the freedom-of-choice legislation requiring insurers to reimburse non-medical, behavioral health providers, and started American Biodyne in the hopes that psychologists could own rather than be owned–as eventually happened–by managed care (read a history here). Neither did I know that he was a member of Psychology’s famed “Dirty Dozen.” Without a doubt, however, the decision to write him, changed the arc of my career.

Fast forward 25 years, and 86-year-old Nicholas Cummings is at it again–not only sounding a warning klaxon but identifying the opportunities available in the dramatically changing healthcare environment. Sadly, the field (and professional psychology in particular) ignored the counsel he’d given back in 1986. As a result, business interests took over managed care, resulting–just as he predicted–in low wages and the near complete lack of professional autonomy.

So, what can clinicians do now to survive and thrive? According to Dr. Cummings, two words best capture the future of behavioral health: (1) integrated care; and (2) entrepreneurship. Let’s face the unpleasant reality and say it out loud: independent practice is on life support. Agency work is no picnic either given the constant threats to funding and never ending amount of regulation and paperwork. Finally, when it comes to practitioner income, its a buyer’s market.

That said, it’s not all doom and gloom. Far from it. There is a tremendous need (and opportunity) in the present reform-driven healthcare marketplace for clinicians who are able to blend behavioral interventions, medical literacy, knowledge about healthcare delivery systems, and entrepreneurship skills. Possibilities do exist. The real question is, “Do we have the will to change?” Here’s where the power of one simple action–in this instance, a phone call–can have such a profound effect on one’s life and success.

Though we never formally worked together, I’ve been calling and writing Nick off and on for the two and a half decades. Late last fall, my partners and co-creators of the International Center for Clinical Excellence, Brendan Madden and Enda Madden, flew to Reno to seek his advice on our business plan. We simply called him. He said, “When can you get here?” The result? His sage counsel helped us win the InterTrade Seedcorn Regional Prize for “Best Emerging Company” as well as secure investors in the most restrictive venture capital environment since the Great Depression. And that’s not all…

Chief Technology Officer Enda Madden Chief Executive Officer Brendan Madden

Just last week, I flew to Phoenix, Arizona to give a presentation on using outcomes to improve behavioral healthcare service delivery at Arizona State University. Nick was there to meet me, along with the director of an entirely new program for behavioral health entrepreneurs, Dr. Ron O’Donnell. Briefly, the “Doctor of Behavioral Health” is the culmination of Nick’s vision of creating a doctoral training program tailored to the emerging need for innovative behavioral clinicians to practice in primary care and medical settings. Response has been overwhelming to say the least. Fifty plus post-graduate clinicians are enrolled. That’s right, post-graduate. In other words, these are practicing clinicians returning to add “integrated care expert and behavioral health entrepreneur” to their resume.

School of Letters and Sciences

As it turns out, I’ll be traveling from Chicago to Phoenix a fair amount in the future. When he stood to introduce me, Nick announced that I’d be filling the “Cummings Professor of Behavioral Health” faculty position at ASU. The power of a single call.

How NOT to Achieve Clinical Excellence: The Sorry State of Continuing Professional Education

September 30, 2009 By scottdm 5 Comments

Greg Neimeyer, Ph.D., is causing quite a stir in continuing education circles. What has he done? In several scholarly publications, he’s reviewed the existing empirical literature and found that continuing professional education in heavioral health is not particularly, well, …educational. Indeed, in a soon-to-be published piece in the APA journal, Professional Psychology, he notes, “While the majority of studies report high levels of participants’ satisfaction with their CE experiences, little attention has been paid to assessing actual levels of learning, the translation of learning into practice, or the impact of CE on actual professional service delivery outcomes.” Neimeyer then goes on to cite a scholarly review published in 2002 by Daniels and Walter which pointed out that “a search [of the research literature] revealed no controlled studies of the impact of continuing education in the…behavioral health disciplines” (p. 368). Said another way, the near ubiguitous mandate that clinicians attend so many hours per year of approved “CE” events in order to further their knowledge and skill base has no empirical support.

Personally, my guess is that any study that might be done on CE in Behavioral Health would show little or no impact on performance anyway. Why? Studies in other fields (i.e., medicine, flight training) have long documented that traditional CE activities (i.e., attending conferences, lectures, reading articles) have no demonstrable effect. So, what does work? The same research that calls the efficacy of current CE activities into questions provide clear guidance: namely, brief, circumscribed, skill-based training, followed by observed practice, real-time feedback, and performance measurement. Such characteristics are, in fact, part and parcel of expert performance in any field. And yet, it is virutally non-existent in behavioral health.

Let me give you an example of a CE offering that arrived in my box just this week. The oversized, multi-color, tri-fold brochure boldly asserts a workshop on CBT featuring the “top evidence-based techniques.” Momentarily setting aside the absolute lack of evidence in support of such trainings, consider the promised content–and I’m not kidding: clinical applications of cognitive behavior therapy, motivational interviewing, cognitive therapy, mindfulness and acceptance based therapies, and behavior therapy. As if that were not enough, the outline for the training indicates that participants will learn 52 other bulleted points, including but not limited to: why CBT, integration of skills intro practice, identifying brain-based CBT strategies, the latest research on CBT, the stages of change, open-ended and reflective listening, behavioral activiation, acceptance and commitment, emotional regulation and distrss tolerance skills, the ABC technique to promote rational beliefs, homework assignments that test core beliefs, rescripting techniques for disturbing memories and images…and so on…AND ALL IN A SINGLE 6 HOUR DAY! You say you have no money? Your agency has suffered budget cuts? No worries, the ad states in giant print, as the same content is available via CD, web and podcast.

Such an agenda defies not only the evidence but strains credulity to the breaking point. Could anyone accomplish so much in so little time? Clinicians deserve and should demand more from the CE events they register for and, in many instances, are mandated to attend in order to maintain licensure and certification. The International Center for Clinical Excellence web platform will soon be launched. The mission of the site, as indicated in my blog post of August 25th, is to “support clinical excellence through creating virtual clinical networks, groups and clinical communities where clinicians can be supported in the key behavior changes required for developing clinical excellence.” Members of the site will use a variety of social networking and collaborative tools to learn skills, obtain real-time feedback, and measure their performance. Anyway, kudos to Dr. Greg Neimeyer for confronting the ugly truth about CE in behavioral health and saying it out loud!