Up next: Production of Lecture Videos and Online Quizzes
A huge amount of work went into creating this course, and we have therefore carefully assessed the effectiveness of the course by comparing test performance with the traditional version of the course, by using a pre-test/post-test approach to evaluating the discussion sections, and by polling the students to assess their perceptions of the course. Details are provided in a paper that we are currently preparing for submission to a scientific journal, but we will describe the highlights here.
Bottom line: Students learn the lecture material better in the hybrid course than in a traditional lecture course that covers the same material, and they also learn additional, deeper skills in the discussion sections that could not be realistically included in a large lecture course.
Comparison with Traditional PSC100
Prof. Luck taught PSC100 in a traditional lecture format for many years, and most of the midterm exam questions were identical in the traditional and hybrid versions of the course. This made it possible to directly compare learning of the lecture material in the traditional and hybrid versions of the course. When scores on these identical questions were compared, the scores were significantly higher in the hybrid version than in the traditional version. The entire distribution of scores was elevated: The number of scores in the A range was doubled, and the number of scores in the D and F range was cut in half. This is consistent with informal reports by the students. Students who struggle in other classes (often because of ADHD or learning disabilities) report that they can more readily learn in this hybrid format, and students who do well in all their classes report that they can really master all of the material in this hybrid format. A careful analysis of demographics indicates that differences in performance between the hybrid and traditional versions do not reflect differences in the types of students who sign up for these two versions.
Pre-Test and Post-Test
Scores on the pre-test and post-test indicate that the students learn a great deal in the discussion sections. Mean performance in a recent quarter (Spring 2015) was 45.1% on the pre-test and 76.4% on the post-test. This is an enormous increase, but it is clear that there is still room for improvement.
Formal Student Evaluations of the Course
Student evaluations are administered at the end of the quarter. In the standard set of questions, students in the Spring 2015 quarter gave a mean score of 4.6 (on a scale of 1-5) for questions asking them to judge the overall quality of the course and the educational value of the course. Custom questions were also included to assess the hybrid course format. Here are some of the key findings (see details in the following table).
- The students did not find the course to be difficult or have a higher-than-appropriate workload
- Students reported that they were better able to pay attention to, and learn from, the online lecture videos relative to traditional live lectures.
- The students found that the discussion sections were useful in achieving our stated goals.
- Given a choice in the future, they would prefer a hybrid-format course over a traditional course
| Spring 2015 Course Evaluation Responses (N=100) | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree |
| The proportion of lectures I watched was substantially higher in PSC100Y than it would be for live lectures in a traditional course. | 34% | 27% | 22% | 10% | 7% |
| When I watched the video lectures, my ability to pay attention to the lecture material was increased compared to traditional live lectures. | 49% | 27% | 14% | 6% | 4% |
| In terms of my overall ability to learn, the video lectures were better than traditional live lectures. | 56% | 30% | 8% | 2% | 4% |
| The discussion sections, journal articles, and worksheets helped me learn to read, analyze, and explain primary research articles in cognitive psychology. | 40% | 36% | 15% | 5% | 4% |
| The workload for this course was appropriate for a 100-level, 4-hour course. | 39% | 42% | 9% | 6% | 4% |
| Given a choice in the future, I would prefer a hybrid course (organized like PSC100Y) over a traditional large lecture course (for courses in Psychology). | 57% | 23% | 14% | 2% | 4% |
Here some of the written comments provided by the students:
- I think the hybrid course is the best way to go. I felt as if I learned the material so much better than if I took a traditional course. And with the help of quizzes, I apply the material I learn for answers and this actually teaches me instead of just trying to memorize information without really learning. I really enjoyed this class and hope more hybrid courses are offered in the future.
- This course taught me things beyond cognitive psychology. I also learned how to be an educated consumer of research and how to study more effectively for other classes.
- The hybrid style of the course allowed me to pay much more attention to the material and take necessary breaks during the lecture so that I did not feel overloaded with information at any given time. The constant graded quiz questions also kept me on track study-wise.
- This class was amazing! I would love to take more online classes during my time here at Davis because of this class..The one-on-one interaction in discussion was perfect for the large university setting. I felt respected, included, and I was called by name almost every class. I feel as though I learned the important material in this class whereas in a classroom setting I would have had trouble parsing out what was actually importance. My ability to pay attention was highly increased and I saw a huge increase in my test scores. I am thankful for the ability to rewind video and for all the types of quizzing that occurred. I was worried about how much I would have to remember, but it definitely became routine. It would be lovely to get email reminders, but that’s minor. I have also been reading psychology studies in my free time because of this class. I actually understand the majority of them now, which is great for graduate school and just for fun!
- I absolutely loved the layout and structure of this course. I was able to watch the videos on my own time and go back if i missed something. The review sessions before each exam were fantastic as well. I loved how whenever I would post a question on the Facebook page, my question was answered quickly and fully.
Important note: These evaluations were for the 4th time the course was taught. The evaluations were quite good the first time, but they have been going up as we refine the course. Also, we did extensive pilot-testing before the first time the course was taught, which was essential for getting things right.
Informal Student Comments
We receive amazing positive feedback from the students. They spontaneously tell us that this is the best course they have ever had and they really appreciate the effort that went into creating the hybrid version of the course. They send us emails months after they’ve completing the course telling us that what they learned about reading and evaluating journal articles is helpful in their subsequent classes.
Students also frequently tell us that they’ve taken hybrid courses previously (mainly in community colleges) and that these previous hybrid courses were terrible. When pressed further, they tell us that the online portions of these courses basically just consisted of recordings of the lectures from the traditional version of the class, and the live portions were unstructured discussions. It is clear that a hybrid course will be effective only if it takes advantage of the opportunities provided by online lecture delivery (e.g., frequent quizzing) and by small-group face-to-face classes (e.g., structured, interactive activities).