This test was given to 700 students in an introductory physics course at the University of Michigan, most of whom were engineers who needed the course to fulfill a requirement. The course was composed of a lecture, given by a professor, and 18 recitation sections, given by teaching assistants, professors, and post-docs. The lecturer and I wrote the exam together. We generated two slightly different versions by altering the order of the questions, the numbers in the questions, and, for the multiple choice items, the order of the choices. The exam was given outside of class for one hour in the evening, which seemed to work very well. The mean score on this exam was about 65%, which determined the dividing line between a B- and a C+.
Even though we spent over a week preparing this exam, a few problems emerged only after the students took it. Quizzes given early in the class reminded students that their answers needed to have the correct units to receive full credit--a feature the exam did not have, so some students assumed they did not have to include units with their answers.
More importantly, the exam turned out to be too long--most students did not get to the last question, which was different on each version of the exam. On one version, the last question was both worth less points, and proved to be more difficult than the final question on the other, so different students didn't get to different questions. This meant that the two versions, which had identical questions placed in different orders, ended up being somewhat different because students could not finish. On the next test, we learned, and only asked five questions. Even though the questions were harder on that test, students had little trouble finishing it. I think students just need some time to think about the questions and become comfortable with them, so five difficult questions took less time than six straightforward ones.
If I had to make the exam shorter by eliminating one question, I might choose the first one. It was supposed to be a short, easy multiple choice question which would build student confidence. However, the two versions were worded slightly differently, which made one somewhat harder than the other. In general, multiple choice questions represent a compromise. They are much easier to grade, but the students don't like them--probably because no partial credit is given for them. I think multiple choice questions can play a part in a good physics exam if they test concepts rather than the ability to get a specific numerical answer. I do not like numerical multiple choice questions because a student who completes the entire question but makes a simple math error gets the same score as a student who left it blank. I also think that multiple choice questions should come in bunches, so each question individually does not represent much of the total grade, but all the questions together do amount to something.
The other kind of question on this exam is what I call a "long question" where the students must show all their work to receive full credit for the question. On this test, we broke all of the questions into smaller parts which made everyone except the exam's authors happier. Students like them because they all but guide students through the solution, and graders like them because it is much easier to assign points to a student solution when it is already broken into parts. However, questions in the real world do not come broken into little parts, and doing so on a test means that students' ability to answer real world questions is not being tested.
Another way to test students' knowledge of principles and ability to solve realistic problems is to include more information in the question than is necessary to answer it. Otherwise, students can look at the variables in the question and either find a formula with all of them in it or twiddle the numbers they are given to get the correct units on their answer. Unfortunately, some good students may erroneously conclude that they must have done something wrong if a number is given, but not needed in their solution. It was for this reason that we gave no extra information on our exam. I think a better approach would be to warn students that they may be able to answer the question without using all of the numbers given.
I think it is interesting that physics teachers want students to remember concepts and definitions, and to develop a good problem solving technique, but end up testing and lecturing in a way that doesn't necessarily reflect this. Textbooks are also written to emphasize problem solving and not general relationships, definitions, and broad concepts. Many students therefore have trouble with questions on concepts and definitions because they review their homework, which is taken from the book, to study for a test. One of the shortcomings of our test arose because it was based largely on the book we used, and therefore did not test concepts or definitions very well.
If I were to rewrite this exam, I would probably throw out a couple of questions and add several conceptual multiple choice questions. I would also turn the multi-part problems into single ones containing more information than the minimum necessary to solve it. I believe this would better emphasize problem-solving techniques and the ability to see what is important in a given problem. Finally, if I could figure out a reasonable way to do it, I might replace one question with a short essay question. Taken together, I think these ideas would produce a very different exam which would emphasize concepts and thought processes more than mathematical manipulations.
Look at the test itself.