Student Performance Online vs Onground:
A Statistical Analysis of IS Courses
Gary Ury
garyury@mail.nwmissouri.edu
Merry McDonald
merry@mail.nwmissouri.edu
Gary McDonald
gary@mail.nwmissouri.edu
Computer Science/Information Systems
Northwest Missouri State University
Maryville, Missouri  64468, USA
Brian Dorn
dorn@cc.gatech.edu
College of Computing
Georgia Institute of Technology
Atlanta, Georgia  30332, USA
ABSTRACT
In previous papers, the authors reported results of smaller-scale studies. The purpose of this study is to combine and expand those individual smaller scaled studies, to determine if significant differences exist between student performance in online and traditional classroom environments. The study includes more than 1300 observations spread across seven courses that are part of the computer science and information systems curriculum at Northwest Missouri State University. Student performance was compared by grade point average, ACT composite scores, number of credit hours completed, instructor, and delivery method. The only significant difference found was between student performance and delivery method in three high volume courses that serve multiple majors and minors. Online students in these three courses obtained a significantly lower average grade than onground students. In four other courses that service upper-level computer science majors no significant differences in performance were found. The varied results of the study could be a simple statement of fact. Different courses in different programs might have different performance results. It could be concluded that online students are simply satisfied with a little lower grade in particular courses or that traditional students perform better because of the availability of added resources planned and implemented for online curriculum. It could be concluded that faculty continue to deal with problems in effectively transferring traditional classroom learning to the online environment. It could also be speculated that the difference is any combination of the above conclusions.
Keywords: online learning, distance learning, course delivery methodology


1.  TERMINOLOGY
Online: A course delivery method that is provided in an asynchronous mode through Internet technologies.
Onground (face-to-face): A traditional classroom delivery model used in typical residence programs in higher education. This is a synchronous method of instruction where students attend regularly scheduled classes in campus or satellite (off-campus) meeting rooms.
2.  INTRODUCTION
Online offerings in all disciplines are proliferating at a rapid rate.  The University of Phoenix offers online degrees (University of Phoenix, 2005) and MIT’s OpenCourseWare program makes course materials available online to the general public (MIT’s OCW, 2005).  There are websites devoted to providing information to potential students regarding online degree programs at the associate, bachelor, and graduate level (ClassesUSA, 2005; Dx: the DistanceXchange, 2005). Limited resources and studies have been employed to truly examine the impact of online education on the student. Primarily, studies have fallen into the two general categories of pretest-posttest models and opinion surveys. Most of the studies have contained relatively small sample sizes, have been performed over short time periods, and measured a single teacher’s experience with the two delivery methods of traditional and online (Ury, 2005).
Recently more quantitative and longer-range studies have begun to appear, with mixed results.  A study of two master’s programs at the University of Paisley in Scotland showed significant differences in online and face-to-face learning, with online students outscoring face-to-face students (Stansfield, McLellan, & Connolly, 2004). University of Wisconsin - La Crosse researchers evaluated performance of students in an educational and media technology course required for preservice teacher education students, and found no significant difference in performance (Ali, 2004). At Michigan State University, researchers compared student performance in classroom and online courses in Principles of Microeconomics and found that online students fared significantly worse on the most complex material (Brown, 2002). A five-semester study of students in a required undergraduate business statistics course at Indiana State University showed no significant difference in performance among students who completed the course (McLaren, 2004).
At Northwest Missouri State University, faculty members have been involved in the delivery of online courses for several years. This paper discusses the results of a study comparing the success of students in online courses to students in traditional courses. All courses are in the computer science and information systems area, but the students are majors in a wide range of disciplines, and the courses range from freshman to senior/graduate level.
3.  PURPOSE OF THE STUDY
In a previous paper McDonald & McDonald (2004), reported results of a study that included only students in the Database Systems course.  That study involved over two years of data and 195 observations. Ttests comparing mean ACT scores, grade point averages, and total hours accumulated verified that students enrolled in online sections and those in onground sections had similar backgrounds, but that onground students outperformed online students by about half a letter grade. Regression analysis showed that grade point average was the most important predictor of success in Database Systems, but that the course delivery method (online vs. onground) added significantly to the predictive capacity of the model. Ury (2005), in an independent study, reported similar findings for a group of 575 Management Information System students over a four year period (Ury, 2005).
The purpose of this study was to combine the techniques used in the two previous independent studies adding new data for those courses and including additional courses within the Computer Science/Information Systems curriculum. The authors were interested in determining any patterns that might exist between the online and onground delivery modes across multiple courses at various program levels that included entry level and advanced courses. Specifically, did onground students consistently and significantly outperform online students?
4.  METHODOLOGY
Development of Online Courses
The first step in developing quality online course delivery models is planning. Online courses should include a welcome, course orientation, syllabus, calendar of events, and course resources. The same elements should be contained in a well planned traditional course (Gerson, 2000). The difference comes not in content or even content delivery, but in delivery methods. Online courses should use the technologies available such as hyperlinks to internal and external course resources, streaming video, audio, and page design. Content should be broken into chunks or concise learning objects that students can use to construct a body of knowledge (Online Course Development, 2005).
Williams (2003) states that most ‘stand-up trainers’ are well accomplished at developing curriculum with rich content, but may lack some of the technical and page design skills to effectively port that content over to the online delivery environment. This fact reinforces the opinions of Gerson (2000) who believes that institutions must provide leadership, guidance, and support for faculty members interested in delivering online courses. There is no one right model to this conversion. Williams states that “educational content is never identical. Design and delivery should follow the demands of your content.”
Northwest Missouri State University (2003) created the Center for Information Technology in Education (CITE) in 1999 to support and advise faculty interested in developing online courses. This program began with 87 students enrolled in four courses. By 2003 the program had grown to more than 50 courses serving 800+ students. The CSIS department has worked closely with the CITE office since 2000.
The CSIS department has been engaged in the design and implementation of an active learning environment for the past 10 years. This delivery model has caused the faculty to develop teaching strategies that create concise learning objects and then reinforce that learning with hands on activities. This work assisted the department faculty in more effectively converting onground courses to the online model.
Online and onground courses offered by the CSIS department contain identical learning objectives. Great effort is taken to assure that content is similar regardless of delivery mode. Many of the courses in this department must hold to a strict set of prerequisite requirements. Students entering the next course are expected to hold a certain minimum knowledge of previous courses. Other courses in the department are service courses for other majors and departments.
The courses offered by the CSIS department must maintain certain common standards that satisfy the future needs of the students regardless of delivery method. All materials, including PowerPoint slides, recorded lectures, narrated online demonstrations, worksheets, and examples are available to both online and onground students.  Both groups of students have similar assignments and exams. While these items may be consistent, it is important to note that the CSIS faculty still struggles with online methods of instruction that replace the audio/visual and spontaneous learning activities of the traditional classroom.
Online students may have additional short quizzes, worksheets, and threaded discussions submitted to the instructor for feedback and participation points, but this is not always the case in all courses. Occasionally, onground students may have some in-class activities that are not exactly duplicated for online students – a weekly quiz, or an in-class exercise, for example. All other materials are available on the course website to both onground and online students.  In some instances, online and onground students share the same course website. The students’ acquisition of knowledge may be checked in a little different manner for onground and online students in the short term, but the majority of assessment instruments and content are similar.
Major assignments and exams are comparable. In some cases, online students close to campus are required to come to a classroom for monitored exams while distant students are required to find an independent proctor close to their location. In other courses both online and onground students are given timed open book exams. Finally, some courses allow timed open book exams for online students and monitored exams for onground students. Across all courses studied, exams are constructed to be rigorous and thought provoking regardless of processes used to administer the exam.
Online students who live near campus may see the instructor during regularly scheduled office hours, just as onground students may; online students who are not close to the campus may visit with the professor via e-mail, a private chat room, or by telephone.
Scope of the Study
This study includes data from the following courses:
CSIS 140 Intro to Programming Visual Basic
CSIS 317 Management Information Systems
CSIS 325 Programming Languages
CSIS 349 Survey of Algorithms
CSIS 445 Computer Organization II
CSIS 460 Database Systems
CSIS 525 Theory and Implementation of Programming Languages
Programming Languages, Survey of Algorithms, Computer Organization II, and Theory and Implementation of Programming Languages are advanced courses taken almost solely by computer science majors. Database Systems is required for computer science majors and also taken by a significant number of students who are seeking a minor in Computer Science or who are majoring in a technical field, such as Management Information Systems or Geographic Information Systems. Introduction to Programming Visual Basic is taken primarily by freshman students in a variety of majors, including Interactive Digital Media and Geographic Information Systems, as an elementary (pre-CS1) programming course. Management Information Systems is required as a part of the business college core curriculum. MIS majors along with any other student majoring in a business field must take the MIS (317) course.
The first data from online courses was collected in summer 1999.  This study includes data through spring of 2004.  Eight different instructors taught courses involved in the study. The courses in the study range in level from freshman (100-numbered courses) to senior/graduate (500-numbered courses). More than 1300 observations are included in the study, with about 38% of those coming from online courses.  Table 1 shows the breakdown by course and by total between online and onground students for the data used in the study.
This study included a total of 1326 students, enrolled in seven different courses, taught by a variety of instructors with ranks ranging from adjunct to full professor, over a two to four year time period. The focus of this study was on the dependent variable of final course grade considering the independent variables of delivery method, course instructor, overall grade point average, ACT composite score, and total credit hours completed.
Table 1: Total Enrollment
Course
Onground
Online
140
130
74
317
581
137
325
21
20
349
20
20
445
9
7
460
177
80
525
26
24
Totals
964
362
5.  FINDINGS
No significant differences were found in any of the courses between online and onground students in the areas of ACT composite score, grade point average, or credit hours completed. This led the researchers to believe that the two study groups differentiated by delivery method were academically equal.  Management Information Systems (317) and Introduction to Programming Visual Basic (140) were taught by a variety of instructors with varying degrees of experience. No significant differences were found between instructors regardless of delivery method used.

Figure 1: Comparison of Average Grades for Online and Onground Students
Individual course results for the current study were mixed.  For some courses, there were no significant differences between the performance of online and onground students. In other courses, we observed that onground students outperformed online students, and in yet other courses the online students outperformed onground students.  Figure 1 shows the average grade for online and onground students in each of the courses studied. A grade of A is represented by 4.0, B by 3.0, C, by 2.0, D by 1.0, and F by 0.0.
Table 2 provides more detailed information regarding the T-tests. Means are broken down by course and delivery method.  P-values associated with the T-tests for equality of means are also shown. A confidence level of 99% was used to determine statistical significance.
Table 2: Summary of T-Test Results
Course Dataset
Onground Mean
Online Mean
p-val
140
3.17
2.73
0.010
317
3.17
2.92
0.004
325
3.14
3.30
0.514
349
3.30
3.20
0.676
445
3.44
3.86
0.196
460
2.71
2.16
0.000
525
3.35
3.42
0.761
All Courses
3.08
2.80
<.001
It is interesting to note that the four courses populated almost solely by junior and senior computer science majors (325, 349, 445, and 525) show little difference in means, and for three of the courses, the mean of the online students is higher than the mean of the onground students, but in some of these instances the population was relatively small. Database Systems (460), an advanced computer science course populated by students from many different majors, exhibited lower mean scores for online students.  The freshman level Introduction to Programming Visual Basic (140) and the sophomore/junior level Management Information Systems (317) also exhibited lower mean scores for online students. In Introduction to Programming, Management Information Systems, and Database Systems, online students can expect to make one-quarter to one-half a letter grade lower than their onground counterparts.
Inspecting the data semester by semester yields some erratic information.  Figure 2 shows that performance in online sections of CSIS 140 may be improving over time, except for the marked differential occurring during fall 2002 when online scores were at an all-time low.

Figure 2: Semester Comparisons for
CSIS 140

Figure 3: Semester Comparison for
CSIS 317

Figure 4: Semester Comparison for
CSIS 460
For CSIS 317 (Figure 3), we see a sawtooth pattern for both online and onground scores, with online scores occasionally equal to or exceeding onground scores.
Figure 4 shows the semester comparison for CSIS 460.  Except for fall 2002 and fall 2003, the average scores of online and onground students are within about a half a letter grade, but online scores are consistently lower than onground scores.
6.  CONCLUSION
This study shows that effective learning can take place in both the online and onground environments. Across all courses over time online students attained a mean score of 2.80, which represents a high C, and onground students established a mean score of 3.08 representing a low B. While this difference is statistically significant, it is important to note that lumping together diverse populations is not a statistically rigorous measure. It is presented here as a general guideline.
It could be concluded that the facts represented here are simply a representation of real world situations and trade-offs faced by students. The three courses that illustrated significant differences (140, 317, 460) served students from many different majors and minors. The four courses that demonstrated no significant difference (325, 349, 445, 525) between online and onground students were advanced programming courses required only of computer science majors.
It could be concluded in some cases that online students who are proven to be academically equal to traditional students are satisfied to accept a little lower grade in trade for the convenience of on line courses. We know from experience that many students take online courses for the wrong reason.  Some students taking a course outside their major area do not want to spend much time on it, and will take a online course, thinking that it will be easier because they do not have to attend class. Others will take an online class because the onground class is offered at 8:00 a.m. and they don’t like early morning classes.
It could be concluded that faculty do not have the skill set or tools for transferring traditional classroom learning to the online environment effectively. Online course delivery systems have a steep learning curve for some faculty. Streaming video, audio lectures, video conferencing software, and reorganizing course resources into small, stand alone capsules, referred to as learning objects, require new proficiencies. The faculty may not have the time, desire, or ability to learn the new skills required to effectively transfer traditional courses to the online environment and many schools can not afford 3rd party or professional development resources.
It could be concluded that online courses require much more pre-planning and anticipation than traditional classroom delivery. Online content has to be published in multiple formats that will be attractive and intuitive to the student. The instructor has to make sure the content is available and organized in a logical manner so the students can navigate to the necessary information. At Northwest most instructors allow traditional students access to all or part of the resources designed for online consumption. It is possible that the increased resources of a well planned online curriculum, when made available to onground students, promote increased learning.
Professional and/or personal commitments continue to force students into the online environment to complete degree programs. The demand for online delivery methods of education continues to grow. Teachers and students must develop the techniques and tools necessary to guarantee this method of education delivery does not give the appearance of a sub-standard education. Sustained improvement of online methods of instruction driven by continued research and new technologies is essential in this highly competitive field of education.
Further quantitative and qualitative research is necessary to determine causes for the findings reported in this study. This study has determined that in some courses online students have not performed as well as traditional students. It is now important to find out why. The online course delivery method has experienced an explosive evolution over the past five years, but there is still much to learn.
7.  REFERENCES
Ali, A., Elfessi, A. (2004) “Examining Students Performance and Attitudes Toward the Use of Information Technology in a Virtual and Conventional Setting.” Journal of Interactive Online Learning, 2(3), retrieved from http://www.ncolr.org/jiol /archives/2004/winter/05/.
Brown, B., Liedholm, C. (2002) “Can Web Courses Replace the Classroom in Principles of Microeconomics?” The American Economic Review, 92(2), 444-448.
ClassesUSA (2005) “Online degree programs.” Retrieved from http:// www.classesusa.com
dx: The DistanceXchange (2005) “Welcome to the distanceXchange...” retrieved from http://www.distancex.com
Gerson, S. (2000) “E-CLASS: Creating a Guide to Online Course Development for Distance Learning Faculty” Online Journal of Distance Learning Administration, 3(4). State University of West Georgia, Distance & Distributed Education Center, Carrollton, Georgia. http://www.westga.edu /~distance/ojdla/winter34/gerson34.html
McDonald, M., Dorn, B., McDonald, G. (2004). “A Statistical Analysis of Student Performance in Online Computer Science Courses.” SIGCSE Bulletin 36 (1), 71-74.
McLaren, C. (2004) “A Comparison of Student Persistence and Performance in Online and Classroom Business Statistics Experiences.” Decision Sciences 2(1), 1-10.
MIT’s OCW (2005) “Welcome to MIT's OpenCourseWare:” Massachusetts Institute of Technology, retrieved from http:// ocw.mit.edu/index.html 
Northwest Missouri State University (2003) “CITE Celebrates Anniversary of Online Courses.” New release retrieved from http://www.nwmissouri.edu/UniversityRelations/news/NEWSRELEASES/031216CITEanniversary.HTM
Online Course Development (2005) “Online Course Development for Dummies” Alternative Learning Division of Madison Area Technical College, Madison, Wisconsin retrieved from http://matcmadison.edu/ald /online/develop.htm
Stansfield, M., McLellan, E., Connolly, T.  (2004). “Enhancing Student Performance in Online Learning and Traditional Face-To-Face Class Delivery.” Journal of Information Technology Education, 3, 173-188.
University of Phoenix (2005) “Programs.” Retrieved from http://onl.uophx.edu/
Ury, G. (2005). "A Longitudinal Study Comparing Undergraduate Student Performance in Traditional Courses to the Performance in Online Course Delivery." The Information Systems Education Journal, 3(20), retrieved from http://isedj.org /3/20/
Williams, P. (2003) “How to Develop an Online Course” Stylusinc.com. retrieved from http://stylusinc.com/online_course /tutorial/process.htm