One of the most unnerving things you can do as a teacher is to ask the students in your class to take out a piece of paper and list on one side the aspects of the course that are working well and, on the other side, list the aspects of the course that they wish could be changed. Whenever I do this-and I normally do some form of course assessment in all of the classes I teach-I always feel a sense of uneasiness that maybe I will learn some things about how my students feel about me and my approach to teaching that I would rather not know. Ultimately, however, this process never fails to reinforce that most of what I am doing is positive while pointing to areas in which real improvements can be made. Additionally, most students appreciate that asking for feedback is an act of faith and trust, which contributes to the sense of an educational partnership. No single act in the classroom sends the important message of caring more than a professor opening up to, and acting on, honest feedback from students.

Of course, this kind of assessment only provides a glimpse of the entire picture. It provides a general sense of how much students are enjoying the class and can give some insightful guidance as to which course elements students feel are most beneficial in helping them learn. What it does not provide is detailed information on how well the students are really learning. In fact, it is my position that if you do nothing to measure student understanding in a regular and systematic way through the course of teaching any topic, it is likely that major areas of student misunderstanding will go undetected until revealing themselves on the final exam, and possibly not even then.

Any process to gather data during a course for the purpose of improving learning-whether data about students' attitudes or content understanding-is called formative assessment. This is in contrast to summative assessment, which is data gathering for the purpose of making an evaluative final judgment such as assigning final grades. As a general rule, it is difficult to do both formative and summative assessment at the same time because students will not usually provide meaningful feedback to you if they think that their grade might depend in part on what they say. However, a few assessments can serve both purposes depending on how the data are used. For instance, a final exam is normally used to determine students' standing in the class, but a careful analysis of the results can reveal persistent student difficulties and suggest changes for the next semester. However, in this discussion, I limit formative assessment to only ungraded, usually anonymous, assessments that have no implications for students' grades. And, to simplify the discussion, I divide formative assessment into two broad categories: course assessment and learning assessment.

COURSE ASSESSMENT

I believe that end-of-course assessments, such as the widely used Aleamoni form, do provide data that are useful in guiding course improvement. The problem is that, from the students' perspective, the timing isn't very good: By the time the feedback is provided, it is too late to do anything about it. I contend that the single most important step we can take toward improving teaching is honestly soliciting student feedback early in our courses and responding to it. Note that I say "respond" and not simply, "acquiesce." I am not advocating making every change students suggest (No, I will not supply pizza every Friday!) but rather that it is important to consider all of their comments carefully and, where appropriate, make midcourse corrections to address their needs.

As with all robust assessments, there are three important steps in conducting meaningful early-semester course evaluation: data gathering, data interpretation, and data reporting (i.e., closing the loop).

Data Gathering

Possible data gathering methods range from the very informal, like that described in the opening paragraph (write down what you like and what you don't like), to the very formal, such as structured focus-group interviews, described later. Each approach has its strengths and weaknesses, but for those who have never tried to look critically at their own class, the method that you choose is not nearly as important as that fact that you are doing something. Although not exhaustive, the following list provides a range of possibilities that could be easily adapted to fit your needs.

§       Informal written response-This is the method already described in which you ask students to write their answers to a few simple questions, such as, "What part of this class is most important in helping you learn and what part is least important? Please explain," or "Do you think that we should continue using class time for collaborative group activities? Please explain."

§       Check-box questionnaire-A questionnaire of your own design is a quick, simple, and effective way to generate quantitative data on a wide range of topics. In addition to measuring students' attitudes toward various aspects of the course, questionnaires can also be used to learn about your students by asking demographic questions such as age, major, or hours of work each week. In collecting student attitude data, questions are normally phrased so that students respond on a 5-point scale. For example, "I am really struggling to keep up in this course: strongly agree (1), agree (2), neutral (3), disagree (4), strongly disagree (5)."

§       Group interview by colleague-In this technique, you invite a colleague to visit with your class, typically for about 20 minutes, while you leave the room. You assure the students that you have invited the colleague to do this (otherwise, it can be perceived as some form of administrative review, causing students, in their desire to protect you, believe it or not, to be less forthcoming) and that their identities will be protected. You should seek help from someone you trust and, preferably, from someone with experience. The interview can be highly structured with a list of specific questions or almost completely unstructured. Both approaches can be effective. Independent of the level of structure, it is important for the interviewer to ask follow-up questions and ask for examples to keep the discussion active. Even more critical, the interviewer must constantly monitor who is responding to ensure that the opinions of a small but vocal minority do not dominate. Although the interview can be tape-recorded, hand written notes are normally sufficient.

§       Focus group interviews-Focus groups are widely used in marketing research and in designing political campaigns. They normally involve a trained facilitator and a group of 6 to 10 participants; the interviews are normally audio- or videotaped for later analysis. Compared with individual interviews, focus group interviews have the advantages of being more efficient and of generating discussion as participants react to each other's comments. Actually, it is this student-to-student interaction that is the primary advantage of focus group interviews compared with large group discussions, which tend to be a series of one-on-one conversations with the interviewer rather than a true discussion among participants. If you choose to record the interviews, it is especially important to tell the students that you will not listen to the tapes until the following semester (and follow through on this promise).

§       Observation by a critical friend-Although this form of data collection does not involve students, using a critical friend to act as a confidential peer observer and provide you with a list of strengths and weaknesses will go a long way in helping you improve your teaching. If a confidential peer is not available, then watching a videotape of yourself will highlight a long list of things you didn't even know you did.

Data Interpretation

Having gathered the data, the next important question is, what are you going to do with it? The type of data you have gathered-qualitative or quantitative-will, in large part, determine the answer to this.

Qualitative data, like that gathered from student writing or interviews, have the advantage of providing depth. The biggest risk is that your overall assessment can be influenced too easily by a minority opinion-especially one that is particularly negative. (Who among us has not lost sleep over that one particularly negative course evaluation from a student even when many more were equally positive?)

It is for this reason that, in the case of analyzing written response data, simply reading the responses is not enough. Even if not overly detailed, I encourage some elementary organization of the data to look for trends. The simplest way to do this is to use different colored highlighter markers to indicate generally positive and negative comments worth reviewing (and scissors if both appear on the same page). Once you have read through all of the comments, or a representative sample in the case of a particularly large class, begin making piles of related comments based on a few broad categories (which you may alter during the process). Finally, review the comments within each category and write some brief notes in which you record not only the theme but also its prevalence. Remember, not every negative comment will require action-12 students claiming that the course is too difficult is not alarming if another 18 report that it is one of their easiest classes! You will be surprised at how little time this process actually requires using a simple tally sheet or a computer spreadsheet.

The other forms of qualitative data we have discussed-large group and focus group interviews-rely on the data collectors for analysis. At the very least, you should plan to sit down with the interviewer(s) to gain a general impression of what was learned and any recommendations for change. Beyond this, I would urge you to request a written evaluation of the interview with a list of recommendations both because it encourages a more thoughtful review of the data and because you might decide that the report can become part of your teaching portfolio. If interviews were recorded, you should review the tapes the following semester-to protect students and provide some distance for you.

With quantitative data, looking only at averages can sometimes be misleading and so I recommend that, for each question, you compute the frequency of each response in addition to the average and then create a means of displaying the data summary in graphs or tables for the class.

Independent of the type of data, the analysis must include your personal assessment of what you have learned and what you might do about it. Areas of difficulty in your courses uncovered through assessment can be typically divided broadly into four categories:

1.     Simple fixes-Sometimes, you discover that there are small procedural things you are doing that could be changed easily to match student wishes. For instance, you might learn that the top 12 inches of your whiteboard can't be seen from the back row, the solution to which is obvious.

2.     Fixes for next semester-It is not unusual to uncover something that would have been changed easily in the syllabus had you anticipated it but cannot be changed during the semester. This might be a simple technical matter like the timing of your office hours or a policy issue like the apportioning of grades for homework (which you might agree is inappropriate but you shouldn't change from what is on your syllabus during the term).

3.     Offsetting concerns-One of the truisms of teaching is that there are no "one size fits all" solutions. Sometimes, for every student who complains about a particular issue, another applauds it. Although this usually means that there is no need for you to change, you should tell the class of the results so that they are aware that their issue has been identified, but contrary what students tend to think, their views are not held universally.

4.     Issues without a remedy-It is sometimes the case that there are important issues, ones with which you sympathize, for which no solution is available. We never take this position lightly-sometimes, creative solutions can be found-but, occasionally, difficult issues simply must be acknowledged to the class and an explanation offered.

It is sometimes the case that there is no appropriate response to any of the issues raised that can be implemented immediately. However, it is still helpful to have something tangible to take back to the class as an immediate and genuine response to the effort they put into completing the assessment, and with a little creative effort, something substantive can usually be found. Neglecting to close the loop will result in students being unwilling to expend the effort required to provide meaningful feedback in the future.

Closing the Loop

It is a mantra of the assessment movement that assessment data, to have any value at all, must inform decision making and that those decisions, including the rationale for them, must be made public. The loop is closed when an action is taken and the assessment cycle begins again.

Although this is the shortest section of this discussion, it is unquestionably the most important. As already stated, students are already asked to provide a fair amount of feedback on end-of-semester evaluation forms and, as far as they are concerned, this information all too often disappears into a black hole. To involve students as partners in this process, it is imperative to report the results of the assessment and discuss what changes, if any, will be implemented in response. When changes cannot be made to address a particular issue, a rationale must be provided.

Return to regular view	Text-only	Updated: 11/01/06