2019 Impact (CAEP Standard 4) and Outcome Measures

We are working with the Montana Council of Deans of Education’s (MCDE) Montana Education Preparation Providers (MEPP) Continuous Improvement Collaborative (CIC) to design a case study template per our statewide protocol agreement with CAEP. The Case Study template of the MEPP Statewide Protocol for Continuous Improvement is (draft as of 1/31/19):

  1. Description of Participants
    • Each EPP will engage in purposeful selection of participants (completers and employers) for the Case Study most relative to key areas of program inquiry.
  2. Description of Context
    • Context of the EPP
    • Context/rationale for chosen area(s) of inquiry
    • Context of completers’ employment
  3. MEPP Data Collection
    • MEPP Completer Survey data
    • MEPP Employer Survey data
    • Observation data – e.g., use of Danielson tool to demonstrate impact/performance
    • Interviews – individual and/or focus group data to demonstrate impact/performance
  4. EPP Specific Data Collection
    • Completer developed artifacts, documents, journals, blogs, assessments, student work, student outcomes, etc. to demonstrate impact/performance
    • Professional learning community participation to demonstrate impact/performance
    • Professional teacher work sample or professional portfolio to demonstrate impact/performance
    • Field experience performance instrument to demonstrate impact/performance

Indicator

Mean scores reported in the aggregate

1-3 scale
(% pass rate)

Standards Alignment

National InTASC / Montana State PEPPS

Assessment Type 2016-2017 Completers 2017-2018 Completers 2018-2019
Completers
Signature Assignment 1 - Indian Education for All Lesson Plan 2 Formative 2.59
(95.9%)
2.64
(95.5%)
2.81
(97.1%)
Signature Assignment 2 - Development 1 Formative 2.95
(100%)
2.90
(97.9%)
2.91
(100%)
Signature Assignment 3 - Assessment 6 Formative 2.92
(98.3%)
2.97
(100%)
 2.90
(98.9%)
Signature Assignment 4 -Relationships 10 Formative 2.96
(99.4%)
2.97
(100%)
 2.95
(100%)
Signature Assignment 5 -Communication/Instructional Media  9 Formative 2.67
(97.1%)
2.70
(97.4%)
2.73
(98.8%)
Signature Assignment 6 - Planning 7 Formative 2.78
(97.5%)
2.82
(95.8%)
2.76
(96.3%) 
Signature Assignment 7 - Pedagogy 8 Formative 2.69
(98.8%)
2.73
(98.1%)
 2.72
(98.9%)
Signature Assignment 8 -Reflection Formative 2.85
(98.8%)
2.88
(99.3%)
2.88
(99.4%)
Teacher Work Sample* 1-10 Summative N/A N/A N/A 
Praxis II Exam
Mean % of MT Passing Score/Licensable at 100% of MT Passing Score
4 Summative 107%
(86%)
110%
(90%)
 
Employer Survey** 1-10 Summative N/A 89% Very/Somewhat Satisfied
11% Very/Somewhat Unsatisfied
 
Completer Survey 1-10 Summative N/A 17% Very Effective
13% Not Effective
 

  * Benchmarks not available; in process of rubric conversion
** All Montana EPPs reported in the aggregate per MT statewide protocol agreement with CAEP.

Survey sent to Montana employers N=362

Responses included:

  • N=132 started (36.4%)
  • N=125 completed (34.5%)

Survey results are reported in the aggregate for all Montana EPP completers as per our statewide protocol agreement with CAEP.

Somewhat
+ Very
SATISFIED
Somewhat
+ Very
UNSATISFIED
Area of Professional Preparation
96% 4%   1. Design instruction based on learners’ development.
87% 13%   2. Differentiate instruction effectively for English Learners (EL).
94% 6%   3. Collaborate to meet the learning needs of all students. 
94% 6%   4. Actively engage students in learning.
82% 18%   5. Respond productively to negative behavior.
95% 5%   6. Purposefully use instructional technology.
95% 5%   7. Demonstrate accurate content knowledge.
89% 11%   8. Encourage critical thinking for problem solving.
80% 20%   9. Analyze assessment data to improve the effectiveness of instruction.
87% 13% 10. Plan instruction based on knowledge of students in their community context.
92% 8% 11. Implement instruction aligned with Montana State content standards.
90% 10% 12. Use a variety of instructional strategies.
96% 4% 13. Participate in ongoing professional development opportunities.
85% 15% 14. Reflect on how instructional choices affect students.
94% 6% 15. Respect beliefs, norms, and expectations of families.
86% 14% 16. Apply knowledge of the legal responsibilities of teachers.
87% 13% 17. Initiate professional conversation with supervisors.
90% 10% 18. Engage with colleagues in a professional manner.  
80% 20% 19. Accurately incorporate the 7 Essential Understandings of Indian Education for All.
89% 11% Average
20. Please share any additional comments regarding your satisfaction with the PREPARATION of your teacher for professional teaching responsibilities. (Redacted for anonymity.)

Survey sent to Montana State University completers: N=389 (2014-15; 2015-16; 2016-17)

  • N=104 started
  • N=90 completed (23% response rate)

Very
EFFECTIVE
Not
EFFECTIVE
Area of Professional Preparation
12% 6%   1. Design instruction based on learners’ development.
1% 43%   2. Differentiate instruction effectively for English Learners (EL).
12% 8%   3. Collaborate to meet the learning needs of all students. 
22% 4%   4. Actively engage students in learning.
7% 26%   5. Respond productively to negative behavior.
22% 8%   6. Purposefully use instructional technology.
33% 6%   7. Demonstrate accurate content knowledge.
14% 10%   8. Encourage critical thinking for problem solving.
13% 17%   9. Analyze assessment data to improve the effectiveness of instruction.
8% 13% 10. Plan instruction based on knowledge of students in their community context.
30% 6% 11. Implement instruction aligned with Montana State content standards.
27% 4% 12. Use a variety of instructional strategies.
28% 13% 13. Participate in ongoing professional development opportunities.
16% 6% 14. Reflect on how instructional choices affect students.
14% 13% 15. Respect beliefs, norms, and expectations of families.
10% 18% 16. Apply knowledge of the legal responsibilities of teachers.
11% 20% 17. Initiate professional conversation with supervisors.
19% 6% 18. Engage with colleagues in a professional manner.  
17% 18% 19. Accurately incorporate the 7 Essential Understandings of Indian Education for All.
13% 7% 20. How relevant was your preparation for the responsibility you confront on the job?
17% 13% Average

Level

Completers

Initial (undergraduate TEP and post-baccalaureate completers)

174

Advanced (added endorsement programs and K-12 Principal/K-12 Superintendent licensure)

100