The index/time stamps below are meant to accompany the National Science Olympiad test writing workshop hosted in January 2022. Please disregard the time stamps, as this is now available only as powerpoint slides. The index allows you to see the content of the presentation.

Note that some of the topics may apply to a contest delivered virtually during COVID-19 and would not apply to an in-person tournament,.

Test Writing Strategies presented by Dr. John Loehr, National Science Olympiad 

Download from the Test Writing Section on the National Science Olympiad event supervisors page

 Below is an index of the presentation, not the actual content

Presentation topics covered

Introductions 

Test-writing vocabulary 

  • Item –  
  • stimulus, prompt;  
  • Stem/question;  
  • Response/answer;  
  • Distractors 
  • Characteristics of the item 
  • Quantitative measures (more for SAT, etc.; not used for SO) 
  • Item type 
  • Multiple choice 
  • Constructed answer 
  • Short answer, essay, etc. 
  • biggest detriment of constructed response is consistency of grading 
  • Item difficulty – Depth of knowledge (DOK) 

 

ADA Accessibility 

 

Item Types 

  • Multiple choice 
  • MultiSelect (multiple correct answers, e.g., A and B) 
  • True/False and matching 

 

Constructed response 

  • Fill in the blank 
  • short answer 
  • essay 
  • labels/drawings 

 

Item Difficulty 

 

 •Bloom’s Taxonomy 

 •Webb’s Depth of Knowledge (DOK) 

 

Test writing strategies 

  • general principles; how Science Olympiad differs from teaching 

 

Overall test format 

  • Test length – number of items? 
  • How to differentiate a winner 
  • You need some really difficult questions that allow for partial credit to differential between the top teams 
  • Deciding on the mix of items – e.g., how many multiple choice 

 

Item pros & cons: multiple choice 

 

Item pros & cons: constructed response 

  • Why fill in the blank are not so good 

 

Item type and time 

  • Multiple choice – expected response time for various difficulty levels (e.g., level 1 difficult (recall) = less than 30 seconds. Level 3 (strategic thinking) = about 2-3 minutes 
  • Constructed response items – Level 1 (recall) = less than 30 seconds; extended thinking (level 4) could be 8 minutes or more 

 

Concrete example of strategic thinking vs. extended thinking question 

  • Strategic thinking is to envision or develop a solution, enabling brainstorming. Extended thinking requires someone to synthesize their answer.  

 

Formats and test writing 

  • New conditions are not as secure – limited supervision, etc. 

 

Rethinking test construction in online environment 

  • Longer tests, teams shouldn’t finish; more difficult items 

 

“Budget” for item distribution (e.g., level 1 – 20 to 30%) 

  • Hard enough to differentiate a winner but don’t want to crush souls 

 

Item Writing Best Practices 

  • Match expected task to item type 
  • Limit use of negative stems 
  • Performs negatively with some student demographics 
  • Consistently order distractors 
  • Avoid certain distractors (e.g., All of the above, A and B) 

 

• Match vocabulary to age level (very important) Use EDL book 

 

Working Online (This is more specific to Science Olympiad but also useful to teachers using an LMS) 

  • Which question types are available? 
  • Can you use images? What file types? Where can images to placed? 
  • Specifics of the platform/LMS 

 

Writing Good Distractors 

  • Use common misconceptions 
  • The best distractors are common misconceptions students who memorize but don't understand have 
  • Use routine calculation errors 
  • Use related ideas 
  • Consider flipping the script (but limited) ….correct EXCEPT; ….all BUT 
  • Keep a consistent number (e.g., multiple choice always have four) 

 

Where to find good prompts and stimuli? 

  • Science textbooks 
  • copyright 
  • Test prep materials and books – released AP tests 
  • Scientific articles and magazines 

 

Writing outside your content area – resources to check out 

 

Changes to Science Olympiad with COVID 

  • One binder each instead of one per team 
  • Identification events can be a challenge – try compare and contrast – e.g., two different pictures of a bird