MSSE Program Assessment 2022

Academic Years Assessed: 2020-2021; 2022-2022

College: Letters & Science

Program: Master of Science in Science Education

Submitted by: John Graves

Program Assessment Plan

Data Sources

1. Fifteen randomly chosen student oral Capstone presentation artifacts were assessed by five MSSE graduates using a rubric created to evaluate the learning outcome of “communicating information orally in an organized and effective manner by sharing the results of their Capstone research projects.”
2. Fifteen randomly chosen student writing artifacts from MSSE education and science courses were assessed by five MSSE graduates using a rubric created to evaluate the learning outcome of “communicating in writing in an organized and effective ”
3. Twenty-two MSSE 575 Capstone papers were assessed using the MSSE 575 writing

Threshold Values

The completed assessments were consistent with the program’s assessment plan and were updated based on the feedback from the 2020 MSSE Program Assessment.

Data Collection & Analysis Methods

The data collected and analyzed spanned the 2020-2022 academic school years. Two Program Learner Outcomes were assessed:

• MSSE students will communicate orally in an organized and effective manner by sharing the results of their Capstone research projects.
• MSSE students will communicate in writing in an organized and effective

Rubrics were created to assess each of the two learner outcomes. Each rubric was based on a 1-4 scoring scale with 4 being the top score. Artifacts in each of the three areas of assessment were randomly selected as follows:

• Oral Random samples of Capstone Presentation recordings.
• Written Communication. Random writing samples from three education and one science content The Capstone Writing Rubrics of 22 Capstone papers were also assessed.

Five graduates of the MSSE Program were recruited to serve as evaluators. Each evaluator was assigned five artifacts in each assessed category. As a result, each artifact was evaluated by two separate evaluators and the two scores were averaged for each artifact. Then the scores for each of the artifacts in each category were averaged and reported as the percentage of students scoring in each of the rubric descriptors.

Learner Outcome Rubrics

 Data Source 1: Oral Communication Rubric

Data Source 2: Written Communication Rubric

Data Source 3: MSSE 575 Capstone Paper Rubric

Note: Rubric scores 31/31 = Exceptional; 29-30 = Approaching Exceptional; 28 = Sufficient; 26- 27 = Approaching Sufficient; 26 and below = Insufficient; 0 = No evidence

What Was Learned

The results of the Oral Communication Rubric indicated that 94% of the oral Capstone presentations assessed scored at or above the sufficient level, meeting the threshold of 75%, (n=29).

The results of the Writing Communication Rubric indicated that 77% of the artifacts assessed scored at or above the sufficient level, meeting the threshold of 75%, (n=22).

The results of the assessment of Capstone Paper Rubric indicated that 77% of the assessed papers scored at or above the sufficient level, meeting the threshold of 75%, (n=19).

Areas of strength of the program identified through the assessments include the oral presentation of the Capstone project. Over the years, we have refined the guidelines of the presentation.

Prior to the assessment year, we changed the length of the presentation from an hour to 20 minutes. In addition, all four Core Instructors set up online presentation practice sessions for students prior to coming to campus to present. Anecdotal observations were that students’ presentations were better organized, were succinct, and flowed better than presentations in past years.

We also believe that the professional paper written by MSSE students is a strength of our program. We contribute this to refining the MSSE Capstone Paper Rubric and including language like “effectively utilizes claims, evidence and reasoning” and “includes a clear synthesis of themes and patterns for each research question based on the collective quantitative and qualitative data across instruments during the treatment.”

How We Responded

The results of the assessments were shared with the Core Instructors at the Fall 2022 Retreat. We reviewed the results and discussed ways to address concerns. Regarding oral communication, the Core Instructors agreed to continue the model of keeping the presentation length to 20 minutes, providing online practice time before the Capstone Symposium, and leveraging the standardized rubric for class assignments as a way to maintain high achievement for students for this program learner outcome.

For the outcome of written communication, the faculty discussed that requiring a specific writing rubric across all 50 or so courses offered in our program would be problematic due to the diversity of science content courses offered. Therefore, as the Core Instructor team, we will focus on writing in the four core courses that lead up to the professional Capstone paper. To accomplish this goal, we will continue to:

• consistently use the Capstone Writing Rubric as a component of all written assessment in our core courses.
• develop writing resources aligned with the written communication needs of our students and post them in our online courses.

As a program, we will partner with the Writing Center in Fall ’22 to create opportunities for students to get writing support throughout the Capstone writing process. The Writing Center is equipped to work with MSSE students, all of whom are online, in both small groups and individually. We have established key points during the writing process when students will be encouraged to meet with a writing tutor.

Finally, the results of these assessments were shared with the MSSE Steering Committee.

Closing the Loop

In reviewing the 2020 Program Assessment feedback, we now feel the assessment of the research outcome conducted two years ago was insufficient. Discussions with both education and science faculty who serve as committee members for Capstone Projects are planned for the

Spring 2023 MSSE Faculty Retreat. It is anticipated that the outcome of this discussion will provide direction to better assess the research learner outcome in the future. We also hope to assess research being conducted on a smaller scale in individual science courses.

Download a printable copy of the 2022 MSSE Program Assessment