Issues in Measuring Time to Teach Information Systems Online Gregory W. Hislop hislop@drexel.edu College of Information Science and Technology, Drexel University Philadelphia, PA 19104, USA Heidi J. C. Ellis Heidi.Ellis@trincoll.edu Computer Science, Trinity College Hartford, CT 06106, USA Abstract During the past decade, the use of online education to deliver information systems courses and even entire degree programs has increased significantly. One result of this growth in distance education is that an increasing number of faculty members are being asked to teach an expanding number and variety of courses in an online format. The time required to teach an online course impacts scheduling, course delivery costs, and faculty incentive to teach online. Therefore, a precise assessment of the time required to teach online is necessary in order to accurately plan and motivate faculty. However, most studies on time to teach online are survey-based or anecdotal in nature. There is little research on faculty effort based on measurement of time, as the quantitative measurement of faculty effort is a difficult task. This paper discusses the issues involved in measuring the time required to teach a course online as compared to traditional, face-to-face delivery. It examines recent reported research results including results of the authors’ work in this area. It then discusses problems in accurately measuring the time to teach and possible solutions to these problems. This discussion is intended to support further studies in this important area of inquiry. Keywords: Online education, teaching online 1. INTRODUCTION Online education has been the topic of much discussion in recent years, and sometimes very heated dispute. In these conversations, the notion that online teaching takes considerably more time than traditional, face-to-face instruction is a commonly voiced opinion. It is an opinion often expressed without question, but it is an opinion backed up by very little hard data. Given the rapid growth in online education, this is clearly not an acceptable state of affairs. Faculty and administrators need to know the workload implications of online teaching assignments. Unfortunately, it is not easy to accurately assess faculty time to teach online. This paper discusses the current state of knowledge and opinion in this area, and outlines what might be done to improve understanding of this issue. 2. CURRENT KNOWLEDGE This section discusses what we really know about time required to teach online. Qualitative studies exist that show faculty estimate that teaching online takes considerably more time. There have been only a few quantitative studies that address this question, and they present a more mixed picture, with some showing an increase and some showing a decrease in online teaching time when compared to time required to teach a course using a more traditional approach. A variety of studies provide support for the notion that it takes more time to teach at a distance. These studies include the National Education Association (2000) phone survey of 402 faculty members on perceived time to teach online; McKenzie, Mims, Bennett and Waugh’s (2000) survey-based study of time required to teach online courses at the State University of West Georgia; a follow-on study by McKenzie, Waugh, Bennett & Mims, (2002) of instructor use of technology in online courses in 19 Georgia institutions of higher education; and Jacobsen, Wijngaards, Kremer, Shaw and Gaines, (1999) comparison of instructor and student opinions of a Requirements Engineering course taught in both an online and face to face format. All of these studies are based on qualitative, survey data rather than any quantitative measurement of the actual time it took to deliver the courses. In addition, these studies have a broader context since the studies looked at a variety of distance education modes including synchronous, asynchronous, and mixed mode delivery. For example, in the National Education Association (2000) study, telephone interviews with 402 faculty members at institutions of higher education were performed to determine faculty opinions on both synchronous and asynchronous modes of distance learning. NEA study results indicated that 53% of faculty felt that they spent more time developing and teaching distance education courses (NEA, 2000, pp. 7). Studies with a more narrow focus on measuring the time required to teach online courses include Hartman, Dziuban and Moskal’s (2000) survey of 36 faculty members at the University of Central Florida, and Schifter’s (2000) survey of full-time faculty at a Research I University. While still qualitative, these studies concentrate directly on online education and whether online courses take more time than traditional courses. Hartman et al surveyed 36 faculty members from five colleges at the University of Central Florida, 32 of whom were teaching fully online courses. Results of this study indicate that 90% of faculty perceived that online courses take more time than the same course offered in a traditional format (Hartman, Dziuban & Moskal, 2000, pp. 166). Ad hoc feedback gathered from interviews and focus groups conducted with faculty teaching online courses indicated higher time demand for online courses. Overall, faculty viewed online teaching as increasing their workload and increasing interactions with students. In another study of online asynchronous courses, Schifter (2000) surveyed full-time faculty and senior administrators at a Research I university. In this investigation, both faculty and administrators identified faculty workload as the third greatest inhibiting factor to teaching online. While most of the qualitative studies provide evidence that online courses take more time than traditional courses, one study reported mixed findings. An informal survey (Almeda & Rose, 2000) of nine instructors of writing at the University of California Extension’s online program investigated instructor perceptions and satisfaction with teaching online. Results indicated that four instructors felt that teaching online took more time than that required for traditional courses, while two instructors felt that online courses took the same amount of time as traditional courses, and the remaining three instructors felt that the workloads varied. There are fewer studies that offer any quantitative data on the time to teach online. These studies also present a mixed picture. One recent study provides an analysis of online teaching time in three online course sections of 25 students each (Lazarus, 2003). While there is no direct comparison to traditional courses, the total time spent by the instructor per class section per week ranged from 3.5 to 7 hours. The author observes that the time spent seems to be well within what might be expected for traditional delivery. Several studies provide a comparison of online and traditional classes. DiBiase (2000) examines time spent teaching two similar geography courses (one online graduate, one traditional undergraduate) and the data indicate that while the online course required more consistent interaction across the semester, the total time spent per student both in instruction and maintenance time was less for the online course. Visser (2000) tracked the time required to teach a distance version of a graduate-level course in public administration and compared the results to the average time required to teach traditional versions of comparable courses. Visser's distance course used a mix of asynchronous online instruction and two-way interactive television for course delivery. Visser compared the time required to develop and teach the distance course to the average time (of three courses) required to develop and teach a traditional course with similar content and delivery, adjusted for class size. The overall results of Visser’s study appear to indicate that online courses take more time to develop and teach than do traditional courses. Recent work by the authors of this paper also provides a comparison of online and traditional classes in the areas of information systems and software engineering (Hislop & Ellis, 2004b). In this study, seven pairs of class sections are analyzed. Each pair consists of two sections of the same course taught by the same instructor where one section was taught online and one section was taught using a more traditional face-to-face mode. Over the seven pairs of courses, total time logged for the online sections was 737 hours as opposed to 814 hours for the traditional sections. While the overall totals are interesting, it is important to note that the class sizes varied substantially. When class size is taken into account, the average time spent per student is nearly equivalent for online and traditional sections of the same course with averages of 6.26 hours per student per online section and 6.17 hours per student for the traditional sections. The difference of 0.095 hours per student is under six minutes per student, or less than three hours over the entire term for a class of 30 students. It is also important to note that the results were mixed for the individual pairs of courses. In three of the pairs, the online section took more time per student. In the other four pairs, the traditional section took more time. Table 1 presents the data for each of the 7 pairs. Note that these courses were taught on a quarter term schedule, so total hours represents ten class weeks plus an exam week. Total Hours Hours per Student Class Size Pair Online Traditional Online Traditional Online Traditional 1 97.9 101.0 4.3 4.4 23 28 2 99.3 91.2 3.5 3.3 28 28 3 78.6 67.7 4.6 4.0 17 26 4 57.7 87.5 4.4 6.7 13 20 5 187.3 126.0 17.0 11.5 11 26 6 132.5 183.2 5.3 7.3 25 28 7 83.7 157.1 4.7 6.0 18 26 Average 105.3 116.2 6.26 6.17 19 26 Table 1 - Hours per Class Section Additional investigation into the distribution of instructor time over the various instructional tasks involved in online education (Hislop & Ellis, 2004a) shows that instructors of online courses spend more time interacting with students when compared to instructors of traditional courses. However, the study also shows that instructors spend less time on other activities such as grading and materials preparation. In addition, the study found that instructors that were more efficient using the online mode of delivery (when compared to the face-to-face mode of delivery) spent significantly less time on grading, materials, preparation, and discussion activities than instructors who were more efficient using the traditional mode of teaching. Taken together, the full set of related studies that examine the time to teach online do not provide a conclusive description of what is actually happening. The qualitative studies clearly seem to favor the notion that online teaching takes more time. This matches what seems to be popular sentiment in many informal faculty discussions about online education. The more interesting point is that the quantitative studies do not seem to provide much support for the popular opinion that teaching online takes substantially more time. The picture is still mixed, but results from quantitative studies include examples of online teaching taking less time than teaching similar courses using a traditional approach, and some substantial evidence that the time difference between online and traditional instruction may not be large one way or the other. If the current results indicate anything with perfect clarity it is that this question needs more careful study. As a step in that direction, the next section outlines some of the issues and possible approaches to quantitative measurement for teaching online. 3. MEASUREMENT ISSUES It is not an easy task to get an accurate picture of teaching time in any mode of delivery. This section addresses some of the key aspects of this problem that should be addressed in an attempt to quantitatively measure teaching time for online and traditional course sections. Object of Study Online and Traditional Forms - While it may seem clear that the goal is to compare online delivery to traditional, face-to-face delivery, it turns out that in practice it is difficult to find these modes of delivery as pure forms. A first issue is whether to study hybrid or blended courses that have substantial online activities but also meet face-to-face at least some of the time. While a simple study might avoid these sections, they represent a common and rapidly growing type of course delivery. A second issue is that even if blended sections are excluded, the remaining sections may not be entirely pure forms. Many online courses include students who are on or near campus. While an online course section may have no formal face-to-face activities, these students may be interacting face-to-face with each other and with the instructor. Similarly, traditional course sections that have no designated online activities typically include email communication between instructor and students. Activity Description Administration Add/drop; class evaluations; other administrative activities Discussion Online discussion with the whole class via the online discussion area or broadcast email to the class Email Email to/from students (individuals or groups but not the entire class) Grading Online or off-line grading or review of student work Lecture Face-to-face lecturing or other group activities (e.g., scheduled class meetings) Materials Preparing or changing course materials during the term Other Work not fitting particular categories (Please name the activity) Phone Phone calls with students Preparation Preparing for class during the term (e.g., lecture note review or background reading) Talk Face-to-face informal discussion / meetings / talk with students (e.g. before and after class) Technology Technology problems or time learning about the online environment Table 2 Categories of Instructional Activity Categorization of Activity - A partial answer to the problem of online elements in traditional sections and face-to-face elements in online sections is to have instructors categorize their activities and record the time spent in these activities. While this complicates the data collection, it helps to separate the actual online and face-to-face activities, and provides essential additional data for the analysis. The DiBiase (2000) and Visser (2000) studies both provided some categorization of teaching time. Table 2 shows the categories used for class activities in the authors' study (Hislop and Ellis, 2004a). Development and Delivery - Instructor time to teach a particular course offering includes both the time to develop the course and the time to deliver it. A complete understanding of online education must encompass both areas of activity. However, if the intent is to compare online and traditional courses, this complete view of instructor workload is probably not possible due to the fact that development frequently cannot be measured in a comparable way for the two delivery modes. For purposes of comparison, measuring delivery time only is a worthwhile endeavor. The problem in measuring development time begins with the fact that, due to the recent increase in online education, many online courses are new offerings and therefore require a completely new development effort. While it is possible to measure the development effort of these new online course offerings, most courses taught using a traditional mode of delivery are existing courses and there is no record of time spent developing them. Finding a sufficient number of new courses taught using a traditional mode to make a reasonable study sample would likely be impossible. The development time measurement difficulty is further complicated by the fact that it is likely that new courses being developed for a traditional mode of delivery would likely also be simultaneously developed for online delivery. Even if there were enough instances of new development both online and face-to-face, it would be very difficult to control the interaction of the two development efforts (assuming the same people developed both modes) or to control for variation across developers (assuming different people developed each mode). It is also important to recognize that course development activities are very difficult to differentiate from some professional development activities normally undertaken by faculty. For example, it would be difficult to separate the background reading that specifically supports development of a course from general reading an instructor does to keep current in an area of technology related to the course. In addition, it would be very difficult to identify new course development in time to capture the preliminary work on the course as many instructors informally research course topics before formal course development begins. A further complicating factor is that the development of an online course typically is not development from scratch, but rather conversion of an existing, traditional course for online delivery. While course conversion may take substantial effort, it is not comparable to the initial development effort for a traditional course. Finally, it is important to recognize that development of a completely new online course may mean something very different than development of a traditional course. In developing a traditional course, an instructor may develop the basic concept, select a text or other materials to support that concept, and then proceed with detailed development using those materials. Development of some online courses follows a similar path, and these online and traditional courses would be comparable in this regard. However, other development efforts for online courses involve extensive development of course materials. This would be comparable to a traditional course that involved writing a text instead of choosing from existing texts. In this case, the development model is completely different from a typical traditional course development and the two processes are not comparable. While this point may seem obvious, it is mentioned here since discussions of online courses in the popular press often focus on development projects that involve extensive materials development. These projects are not likely to become commonplace, and are not good candidates for comparing to typical development of traditional courses. In summary, it seems that in a comparative study of online and traditional teaching, the time to develop the course should be excluded, and only time to deliver the course should be counted. This approach must be taken because a record of development time for traditional sections is unlikely to be available, and, because to the extent data is available for both modes, it is usually not comparable and it is almost certainly seriously confounded. An alternative approach to determining the time required to teach a course is to measure course delivery time only. Delivery can be more easily defined as occurring within a fixed time period such as all course activity from the first through the last day of the term. This approach has the great advantage of having clearly defined time measurement boundaries that are known beforehand. Note that this definition of delivery would capture some development activity since instructors often work on course materials during the term. However, the categorization discussed above could identify this type of activity, and it would simply be considered as one part of the overall delivery time. Data Collection Manual Logging - Although online courses are conducted via computers and communication networks, automated methods cannot capture the full picture of instructor activity teaching online, and there is no comparable automated record for traditional sections. The only solution is to convince instructors to manually log course time by type of activity. This has all the potential for problems of any manual data capture approach, but cannot be avoided. Motivation - Manual time logging requires active participation of the instructor over an extended period of time, typically a semester. The logging is somewhat tedious and requires constant attention, so complete logs are difficult to obtain. The authors participated in one study that produced no complete time logs and are aware of another study that failed for the same reason. Some of the quantitative studies discussed above succeeded because the authors used themselves as subjects. The seven course pair study included a monetary incentive for instructors to complete the time logs. Accuracy and Interpretation - Some of the problems of manually recorded data relate to interpretation of categories of activity. In the authors study, a small set of categories, shown in Table 2, was provided along with a definition and examples for each category. In general, this approach worked well, but there was still some confusion among participating instructors and some variation in how each person mapped their activities to the categories. While there is no simple solution to this problem, extra caution about defining terms and allowing participating instructors to annotate their entries can help to control this problem. Confounding Variables Perhaps the most serious issue in comparing online and traditional teaching time is the potential for confounding variables to influence the result. The possible confounding variables include factors related to the instructor, the course, the students, and the environment. Examples of these factors are: Instructor - experience teaching traditionally, experience teaching online, teaching style, cultural background, technology skill and attitude, interest and attitude toward teaching online, number of times teaching this course in each mode, and knowledge of subject matter. Course - nature of the course material, amount and quality of course development prior to delivery. Students - experience with learning online, technology skill and attitude, maturity, interest and attitude toward learning online. Environment - class size, quality and reliability of online technology or classroom facilities, and particular technologies selected. Whenever possible, these confounding variables should be eliminated. In cases where the variability in the feature or aspect of the study cannot be eliminated, effort should be taken to factor in the variable. For instance, instructor effort measurements can be normalized on a per student basis to provide a clearer picture of the effort required to teach an online course of a particular size. The issue of confounding variables is notable in the Visser study (2000). First, Visser notes that the distance course in the study was the first distance course that he had developed and taught and that he was, at that time, relatively new to teaching in general. Second, as the distance course in the study was a new offering, the development time for this course could be expected to be higher than for an existing course for which materials had already been prepared. Third, Visser clearly indicates that the mix of asynchronous and synchronous delivery mechanisms used in the distance course could have increased the time spent delivering the course. Study Construction In constructing a study to measure the time to required to teach online versus face to face, there are several steps that may be taken to help ensure correct results from the study including: * Instructors must be properly motivated to participate in the study. Possible motivating influences might include compensation or release time. * An effective logging mechanism must be constructed. In a study performed by the authors, manual logging was shown to be more effective and more accurate than automated forms of collecting instructor time data. It is important that a simple form be designed that is easy to use. A clear categorization and definition of instructor activities must also be provided. * Selection of established courses with experienced instructors will provide less variability in the results. * Definition of a clear start and end time of the study provide concrete boundaries for the study. * Orientation for instructors should be provided to ensure an unambiguous understanding of the activity categories and logging requirements. It is important to eliminate confounding variables wherever possible by including in the study instructors with similar teaching style, experience, attitude and background. In addition, by keeping the course material and number of students closely similar, variability in the results due to differences in class size and subject matter can be eliminated. Student background is less easy to control, but by restricting the study to a single degree program or a subset of courses in a degree program, it is hoped that student experience, maturity and attitude would be similar across the study group. Finally, combining the basic time measurement with demographic data on students and instructors and qualitative data (e.g., on attitudes toward online education or technology) would be essential to understanding the landscape of possible confounding variables. 4. CONCLUSION Online education has moved into the mainstream of higher education, and it seems quite likely that expansion of activities in this area will continue. This growth means that it is essential to gain a better understanding of the time it takes to teach an online course. The few studies that have quantitatively measured and compared the time required to teach online versus face-to-face have mixed results with some indicating that online education takes more time and others indicating that online education takes the same or less time than face-to-face instruction. A study performed by the authors of this paper indicate that online courses do not take significantly more time to deliver than their face-to-face counterparts. In addition, the increasing use of mixed modes of delivery highlights the importance of examining the question of time required to teach a course for completely online courses, for courses mixing online and face-to-face activities, and for traditional, face-to-face courses. Given the complexity of teaching time as an object of study, great care will be needed if generally useful results are to be obtained. In particular, the issues of study construction including clear identification of teaching modes and study boundaries; data collection including unambiguous definition of activities, motivating faculty and accurate logging; and confounding variables such as instructor experience, student background, and class characteristics must be addressed in order to obtain an accurate picture of the time required to teach online versus face-to-face. This paper has attempted to characterize our current state of knowledge in this area, and to identify key factors that must be considered to move this area of inquiry forward. 5. References DiBiase, D., 2000, Is distance teaching more work or less? The American Journal of Distance Education, (14)3. Hartman, J., Dziuban, C., & P. Moskal, 2000, Faculty satisfaction in ALNs: A dependent or independent variable? Journal of Asynchronous Learning Networks, (4)3. Hislop, G. W. & H. J. C. Ellis, 2004a, “An Analysis of Variation in Teaching Effort Across Tasks in Online and Traditional Courses,” Proceedings of the 6th International Conference on Enterprise Information Systems, Porto, Portugal. Hislop, G. W., & H. J. C. Ellis, 2004b, “A Study of Faculty Effort in Online Teaching,” The Internet and Higher Education, Elsevier/Pergamon, (7)1. Jacobsen, D. M., Wijngaards, N. J. E., Kremer, R., Shaw, M. L. G. & B. Gaines, 1999, The comparative evaluation of classroom and distance sections of an industrial software engineering graduate course. Proceedings of the 1999 International Conference on Mathematics/Science Education & Technology, Retrieved July 3, 2003, from http:// www.acs.ucalgary.ca/~dmjacobs/seng/mset99_paper.html Lazarus, B. D., 2003, Teaching courses online: How much time does it take? Journal of Asynchronous Learning Networks, (7)3. McKenzie, B. K., Mims, N. G., Bennett, E. & M. Waugh, 2000, Needs, concerns, and practices of online instructors. Online Journal of Distance Learning Education, (3)3. McKenzie, B. K., Waugh, M., Bennett, E. & N.G. Mims, 2002, Course preparation for online learning: What faculty should know. 2002 Society for Information Technology and Teacher Education International Conference, USA. National Education Association, 2000, A survey of traditional and distance learning higher education members. Retrieved June 5, 2003, from http://www.nea .org/he/aboutthe/dlstudy.pdf Schifter, C. C., 2000, Faculty participation in asynchronous learning networks: a case study of motivating and inhibiting factors. Journal of Asynchronous Learning Networks, (4)1. Visser, J. A., 2000, Faculty work in developing and teaching web-based distance courses: A case study of time and effort. The American Journal of Distance Education, (14)3.