Annual Counseling Program Evaluation 2020-21
MSU Counseling Program Objectives
Objective #1: Core education in counseling (Portfolio Assignments HDCO 558, 521, 522, 510, 523, 550, 551, 563)
A major objective of the counseling program is to provide students with training which meets the American Counseling Association's Council for the Accreditation of Counseling and Related Educational Programs (CACREP) counselor education accreditation criteria in the following core areas:
- Human and family growth and development
- Social and cultural foundations
- Helping relationships
- Lifestyle and career development
- Research and evaluation
- Professional orientation
Objective #2: Counselor’s personal/professional development (Benchmarks 2 and 3)
A second objective of the program is to promote counselors' personal, interpersonal, and professional development. The counseling program is committed to encouraging persons from diverse cultural backgrounds to become counselors. Because counseling requires high levels of professional maturity and interpersonal skills, the curriculum offers several experiential learning courses which are designed to foster students' personal development, relationship skills, and professional orientation. The course content includes self-exploration and skill acquisition regarding personal values, cultural heritage, professional issues, personal and professional relationships, and group dynamics. These experiences include opportunities for development of cohesive relations between students through self-disclosure, empathic listening, feedback, and role play.
Objective #3: Foundations of counseling (Portfolio Assignments HDCO 502, 503, 505, 525, 530, 568)
A third objective of the program is to assist students in the acquisition of knowledge in the foundations of counseling, including its history, philosophy, unique professional identity, professional organizations, training standards, credentialing mechanisms, ethical codes, and research and professional issues.
Students will be encouraged to join appropriate professional organizations.
Objective #4: Counseling in Montana's mental health and education systems (Benchmark #3; HDCO 505, 506)
A fourth objective of the option is to help students understand and prepare to fulfill mental health counseling roles in Montana's mental health care and education systems. The program is designed to meet licensure requirements in Montana for professional counselors and school counselors.
Objective #5: Counseling Skill Development (Benchmark #1)
The fifth objective of the program is that students acquire the comprehensive knowledge and skills needed for provision of counseling services across a variety of settings from schools to community mental health agencies. Studies in this areas include: 1) understanding the general principles which explain mental health; 2) becoming competent in appraisal methods including mental status exams, mental health history taking, testing, DSM-V diagnosis, and environmental assessment; learning models and techniques for promoting mental health; and treating disorders including mental health education, prevention, consultation, crisis intervention, psychotropic medication, and disorder specific individual, family, and group counseling.
Objective #6 Theoretical knowledge and applications (Portfolio Assignment HDCO 508, 571, 598, 554, 565, 569)
Competent counseling practice is informed and guided by theory. The program emphasizes thorough knowledge of clients' developmental, cultural, and social contexts. Theory-based approaches to appraisal and individual, family and group counseling are covered in-depth. During counseling practicum (100 hours), students are closely supervised by licensed mental health professionals about clients and counseling theories. Supervised practice continues and is expanded during internship (600 hours).
To assess these objectives the MSU Counseling program has developed an assessment plan that includes a series of benchmarks, as well as portfolio assignments.
Overview of MSU Counseling Program Assessment Plan
The following is an overview of the assessment plan and process of the MSU Counseling program. This assessment plan is put in place to assist with measuring and tracking program outcomes and assist faculty in identifying key assessments to measure whether students in the program are meeting/exceeding expectations.
The assessment plan for Montana State University-Bozeman starts with the interview and proceeds through a series of benchmarks and portfolio assignments designed to meet specific learning outcomes in each specialty area. The final overall assessment of student learning culminates with the final benchmark- the comprehensive exam. The assessment process comprehensively addresses both knowledge and skills/practices outcomes for Clinical Mental Health, Marriage, Couples & Family, and School Counseling programs. Knowledge Areas in the 2009 Standards are demonstrated using portfolio assignments and Skills/Practices are demonstrated in benchmark or professional dispositions. Students’ progress through a series of four dispositions in which Skills/Practice are evaluated, as well as Knowledge Areas reviewed.
Skills/Practices is measured in a series of four benchmarks for MFC, MH and SC students. These benchmarks are outlined below:
Benchmark 1: Counseling Skills (December mid- 1st year)
Benchmark 1 is determined during HDCO 521: counseling skills in which students submit a final tape. This benchmark uses that tape, as well as the overall course grade and feedback to determine the benchmark disposition of the student. Any disposition of a “3”, on a 5-point Likert scale, which represents the equivalent of a B- grade, will most likely result in a remediation plan to address areas of concern. While this is a passing grade; it suggests concern for counseling skill. However, it should be noted that at this point in a student’s training it can be difficult to determine if he or she lacks the skill or, perhaps, needs additional time to develop it, thus more leeway may be given regarding this benchmark, versus benchmarks at other points in the program. A remediation plan will be constructed in consultation with the student’s committee. A disposition of 4, the equivalent of a C+ or lower grade, will result in a student retaking the skills course. Passing this course demonstrates the potential of the student to meet future benchmarks and skills/practices in each specialty area. Please see Appendix A for an example of a remediation plan for a disposition of a 3 or 4.
Benchmark 2: Practicum (May – end of 1st year)
Benchmark 2 is determined during practicum in the spring semester. This benchmark is measured using the practicum supervisor’s evaluation completed at the end of the semester. It is expected that feedback given throughout the semester has been integrated into the student’s work with clients. Any benchmark disposition of 4 will most likely result in a remediation plan. A disposition of a 5 will result in a student re-taking practicum and possibly skills if it is determined that there is a serious skill deficit or that deficits evident in Benchmark 1 have not been remediated. If students pass benchmark 2, they will be allowed to proceed to internship. In addition to the review of students’ skills, students’ portfolios are reviewed, and tracking sheets collected at this meeting.
Benchmark 3: Internship (Qualitative Evaluation mid-fall semester & Benchmark meeting in December -2nd year)
Benchmark 3 is determined by the evaluation of the student by the internship site supervisor in combination with the observation of skills by the internship course instructor (usually the faculty program leader) as well as student performance in internship class, HDCO 598. Students will be given a disposition indicating what areas of improvement are needed, as well as areas that the student is doing well. In addition to the skill review, portfolios are reviewed, and tracking sheets collected.
Also, students are required during their internship course (HDCO 598) to complete a qualitative evaluation mid-semester of the feedback they have received throughout their program of study. This form is utilized to help students stay accountable for the feedback they have received throughout the program, as well as help them see areas that they have made improvements in.
Benchmark 4: Comprehensive Exam (March of 2nd year)
Benchmark 4 is the comprehensive exam. Students must successfully pass this exam, which is a written paper demonstrating skill in theory-based case conceptualization and treatment planning, to proceed to graduation. The exam is blindly graded by two faculty readers. If a student receives one pass and one fail, then a third faculty member is asked to read the exam. If the third reader grades the exam as a pass, the student passes comprehensive exam. If the student receives two fails the student must re-take the exam after a 3-month period, which may be during the summer semester but is determined at the committee’s discretion. If the student fails the exam a second time, he or she will not be allowed to graduate from the counseling program. Graduation hinges on successful completion of the comprehensive exam.
Checkout: (April of final year)
Students will complete a final checkout with their program leaders during the last week of internship class during their final year. During this checkout students’ portfolios will be given one final review. All tracking sheets will be collected, and students will be given their final evaluations from their site supervisors.
Knowledge for both core and specialty area outcomes are measured by portfolio assignments. Each course in the program has at least one assignment designated as a portfolio assignment. The matrices for each program document the standards, which courses those standards are being met in, and how the standards are being measured (portfolio assignments).
Knowledge areas in each program specialty are measured through carefully selected portfolio assignments that specifically meet those standards. Other course assignments may meet those standards; however, the portfolio assignment is an agreed upon assignment by all core faculty. Each portfolio assignment is clearly marked on all syllabi.
Students will collect portfolio assignments in a binder and keep track of the grade and corresponding assessment value in a tracking sheet located in the front of the binder. An example of this tracking sheet is in Appendix D. If a student receives a score of 5 indicative of “does not meet standard”, the student will need to complete remediation. This may include repeating the assignment until he/she receives a score indicative of “meets standard.” If a student receives a 4 “meets standard with concerns,” the student may have remediation. This determination is based upon student performance in previous courses and if this issue has been a pattern throughout the student’s program of study.
In addition, students will be asked to fill out an assessment of their progress during their meetings for Benchmarks two and three. Please see Appendix C for this student assessment form. During their meetings for benchmarks two and three and the final checkout, program faculty will review tracking sheets with students to ensure that students are keeping track of their progress. This data will be collected and used to determine how successful we are at meeting the knowledge area outcomes for CMHC, MCFC, and SC.
Counseling Program Yearly Outcome-Based Report
Application and Acceptance into Program
For the 2020/21 Academic Year, the program received 175 applications to the counseling program: 92 in Clinical Mental Health Counseling, 43 in Marriage, Couples, & Family Counseling, and 35 in School Counseling. The Clinical Mental Health program acceepted 12 applicants, the Marriage, Couples, & Family program accepted 8 applicants, and the School program accepted 8 applicants for a total of 28 applicants. As of August 2021, all 28 students are enrolled in the program.
The average score of portfolio assignments per program area were:
- CMHC: 1
- MCFC: 2
- SC: 1.29
For more detailed scores per portfolio assignment raw data from each portfolio assignment is available. Overall, what these averages suggest is that within program and between programs, our students are scoring well within the range of 1-2 on their portfolio assignments.
A=1, A-/B+ = 2, B= 3, B- =4, and C+ or lower = 5; students pass Benchmark 1 with a score of 1, 2 or 3. Anything lower and the student will have remediation of the assignment.
Benchmark/Professional Disposition Data
The following data is Benchmark/Professional Disposition average scores across program area. For all assignments, A=1, A-/B+ = 2, B= 3, B- =4, and C+ or lower = 5; students pass Benchmark 1 with a score of 1, 2 or 3. Anything lower and the student must retake the course.
Benchmark 1 comprises basic counseling skill demonstration on a final tape. Students must successfully pass skills to progress to Practicum (Benchmark 2). For the 2021 cohort of students the average benchmark score per program area was:
- CMHC students: 89
- MCFC students: 33
- SC students: 5
- Overall average across all three specialty areas: 24
Benchmark 2 is comprised of a demonstration of basic counseling skills with community clients as well as a demonstration of greater conceptual ability. Students must successfully pass practicum to progress to internship (Benchmark 3). For the 2021 cohort of students the average benchmark score per program area was:
- CMHC students: 66
- MCFC students: 83
- SC students: 33
Overall average across all three specialty areas: 1.61
Benchmark 3 is comprised of a demonstration of advanced counseling skills in a community agency or school setting. The benchmark is based upon university observation, case conceptualization grade and site supervisor evaluation. For the 2021 cohort of students the average benchmark score per program area was:
- CMHC students: 1
- MCFC students: 5
- SC students: 16
Overall average across all three specialty areas: 1.25
Benchmark 4 is the comprehensive exam distributed in the spring of the students’ second year. See narrative below.
Comprehensive Exam Data 2020-2021
Twenty-six students took the comprehensive counseling exam in February 2021. Of the 26 students, no exams went to a third reader. For 2020-21 we had a 100% pass rate for the comprehensive exam.
Internship Site Data AY 2020-2021
The Counseling Program added four new internship sites this last year: Three Rivers Medical Center, Intermountain Mental Health, the Eating Disorder Center of Montana, and rural school sites connected to our federal grant (Rural Mental Health Preparation/Practice Pathway [RMHP3]). Students report being satisfied with their internship site experiences. Areas of common concern are lack of opportunities for group and at times MCFC students had difficulty gaining their systems related hours. For several students meeting the program/CACREP direct hour requirements was challenging, due to being restricted to offering telehealth versus in person services, and subsequently less consistent engagement from clients. The sites that we utilize for internships are very strong in terms of the quality of onsite supervision, relevance to career goals, exposure to a wide range of clientele and clinical issues. They prepare students for employment immediately upon graduation. We consistently hear from employer’s that our students are ready to start.
To address the lack of group counseling opportunities, the GTAs in the program are given the opportunity to run small groups with the first-year students. A faculty member interested in working with the prison population started multiple groups at the jail that the students ran this year, even resorting to telephone based groups at the height of the pandemic over the fall and winter of 2020-21. Both the Eating Disorder Center and Three Rivers Medical Center will provide more group counseling opportunities. For MCFC students, if they are struggling with systems hours at their sites, they have the option to take on additional couples at our program’s Human Development Clinic.
Internship Site Supervision Data AY 2020-2021
Overall, students report being satisfied and supported by their site supervisors. There were no major areas of concern reported. There were challenges associated with covid and the quality of services available due to telehealth/social distancing constraints, but the issues were largely temporary and resolved as each site learned to adapt to the new normal. Students reported being supported and appropriately challenged by their site supervisors. Occasionally students reported discrepancies in professional identities between how they are being trained and how their site supervisors identified their role and scope. This was most salient for school counseling interns.
Vital Statistics for AY 2020-2021
In 2021 there were 25 graduates from Montana State University’s Counseling program: 12 graduates from the Clinical Mental Health Counseling program, 7 graduates from the Marriage, Couple, and Family Counseling program, and 6 graduates from the School Counseling program.Of the admitted students for the Clinical Mental Health program,100% graduated in the expected period, and 12/12 of the students reported employment by graduation. Of the admitted students from the Marriage, Couples, and Family program, 88% graduated in the expected period and 7/7 students had reported employment upon graduation. Of the admitted students for the School Counseling program, 86% graduated in the expected period and 5/6 students obtained employment by graduation. Data regarding the NCE examination is not yet available for 2021; however, our graduates are consistently successful at obtaining licensure both in- and out-of- state.
The Human Development clinic offered over 2000 hours of low-cost counseling services, and for 2020/21the counseling practicum and internship students offered over 10,000 of free mental health services to schools, community agencies and at the Human Development Clinic. Due to COVID, the Mu Tau Sigma, our local chapter of Chi Sigma Iota, did not engage in outreach projects.
Program Modifications based on Evaluation Data
Mental Health and Program Accessibility Issues
Based on communication with site supervisors and clinicians in our community and state, the program is considering ideas and options for how to meet the needs of the greater state of Montana. Our state is vast, mental health issues are significant, and there is a serious accessibility issue for Montana residents being able to receive mental health services, especially in the more rural areas. Our rural school grant (RMHP3) was developed to attend to these issues, and we are continuing to examine other ways to increase access for Montanans.
Relatedly, we are exploring avenues for making the program more accessible for graduate students who are not able to move to Bozeman or give up working while enrolled in the program. Areas of discussion for the future include: 1) how to incorporate a distance component into the program for applicants who are more place bound. The graduate coordinator and office personnel continue to receive phone calls regarding if the program is online. 2) There is an increase in the number of online programs offered in the state. Montana State’s counseling program continues to hold regional and national recognition as a quality program; therefore, in the spirit of our land grant mission we would like to explore how to offer the program in some type of distance capacity.
Informed by consistent feedback from students we have made two programmatic changes. First, we adapted our Crisis Counseling course from a summer-intensive to a full-semester course, based on student feedback and our own observations of the importance of properly training our students to be trauma and crisis informed. The first full semester course offering will begin in January 2022. Second, the Marriage Couple and Family Counseling track has decided to shift the Child and Adolescent Counseling class to an elective, due to some students being interested in systems or couples work, but not having a specific interest in working with minors.
The program conducted a supervisor training in the Summer of 2021 with site supervisors and community supervisors, to create more available clinical supervisors in the community for the program interns as well as graduates. We intend to offer this same training every 6-9 months, to increase the number, but more important the caliber of clinical supervisors in the community and state.
The counseling program fully implemented Tevera into the program, where all clinical paperwork for all three tracks is recorded and stored. The students’ have lifetime access to their files, and additionally can use the program to track their post-graduate hours and paperwork. Our Human Development Clinic moved from a word processor remote drive to using Titanium for all clinical paperwork (primarily scheduling & notetaking). This is a more organized and secure system that also allows for remote access for practicum supervisors to review student notes. Regarding covid precautions, this allows for less traffic onsite, particularly during the spring semester.
Addiction Specialty Area
The program has noted that more students are pursing the LAC certification. We have created a more overt pathway for students to pursue the LAC required coursework during their program. Dr. Ed Dunbar incorporated more addictions course work in the HDCO 568 MH Methods course.
Other Substantial Program Changes
The MSU Counseling program experienced five additional program changes:
- Adaptation to telehealth, while a response to COVID, this provided an opportunity to explore how to reach out to rural areas of Montana with the Human Development Clinic. TheClinic was able to maintain clients even in lieu of this switch to telehealth.
- Rebecca Koltz and Dr. Anna Elliott implemented their second of five year $2.3 million grant to increase access to mental health in rural schools in the state of Montana. Ten new students across the MSU and University of Montana counseling programs were placed in federally designated high-need rural schools.
- Mark Nelson retired as school counseling program advisor. Dr. Edward Dunbar is serving as interim program advisor for school students. Dr. Kara Hurt-Avila resigned from her position in the summer of 2021. Dr. Heidi McKinley stepped down as clinic director, while still maintaining her duties as a course instructor and MCFC program advisor. We were approved to search for two tenure-track positions in September 2021. We plan to have those positions filled in early 2022, for an August 2022 contract start.
- Bryan Lamb was hired as the Human Development Clinic director, as well as a core faculty, teaching one course per semester, and also taking on the faculty advising position for our Chi Sigma Iota Chapter, Mu Tau Sigma.
- We implemented the Counselor Competency Scale-Revised (CCS-R) as part of our ongoing assessment process for students. In addition to having students use the tool to self-evaluate, we used the dimensions being assessed as part of our Benchmark evaluation.