The Effective Use of Web-based Training and Assessment in a Computer Literacy Course Therese DonGiovanni O’Neil Therese.ONeil@iup.edu Indiana University of Pennsylvania Computer Science Department 319 Stright Hall Indiana, Pennsylvania 15705, USA Abstract With the onset of the millennium comes the diversity of students’ knowledge in the field of computer literacy. High schools are now graduating a more computer literate student. This poses a challenge to the basic computer literacy course instructor. How to meet the needs of all students is a common quandary. It has been my experience as an educator that the answer lies in the implementation of a web-based training and assessment software package. Fifty percent of our course in computer literacy delves into computer concepts. This author does not endorse web-based training and assessment in that area. When discussing computer ethics or cybercrimes, for example, the traditional classroom is a much better venue to entice student interaction and critical thinking. This paper will explain how using training and assessment software in the computer applications portion of the course will effectively meet the needs of the non-traditional student as well as the traditional student. It will explain the use of this software in the classroom. The author does not endorse one specific software tool, only the concept of using web-based training and assessment for teaching a skill subject. The software package used for this paper is Course Technology’s SAM 2003 Version 3.0. Keywords: assessment, online, testing, computer, literacy, training, software 1. INTRODUCTION The Internet is increasingly used as a tool in education. Internet-mediated teaching and assessment is increasingly being used to supplement traditional courses (Buchanan, 2000). As early as 1997 educators have been delving into the area of web-based instruction with Terry Burton’s Just-in-Time-Training (JITT). He believed that individual learner needs is extremely important, but there is a limited amount of time in which to accomplish this. A situation needs to be developed that would provide learning experiences that were easily customized to the needs of individuals. Burton contends that technology can provide the infrastructure needed to enable this type of situation. (Burton, 2000) Acadia University created an Online Testing Module (OTM) used from 1998 to 2000. The OTM allows faculty to prepare web-based tests. Students using this testing package for tutorial assessment have responded very favorably to the ease of access, and the ability to monitor their own progress (MacDougall, Place, Currie, 2000). This concept developed the theory that technology would provide the right environments in which students could take responsibility for learning (Burton, 2000). As early as 2006, the Texas Assessment of Knowledge and Skills exam will be on computers instead of paper. “In the eyes of state Rep. Ken Grusendorm, taking a test with a pencil and paper is so 20th century” (Embry, 2005). Oklahoma City is experimenting with online testing by administering a geography test via the Internet this spring. As many as 48,000 seventh-graders will take the geography test online in a practice run that officials hope to spread to other tests (Hoberock, 2005). In Washington Park, New Jersey, students are using a new educational program recently installed called Odyssey. This program is web-based and uses games and animation to teach third-graders in an individualized way. One teacher’s comment about the test was “I see them accomplishing math concepts that most students don’t do until sixth or seventh grade” (Kovac, 2005). It is with this notion that I analyze the possibility of web-based instruction used effectively in the computer literacy course. 2. THE PROBLEM Diversity in Learning Styles and Background Diversity in learning styles and backgrounds can cause problems for students, institutions and faculty when basic student skills are uneven (McQueen, Fleck 1999). Meta-analytic research suggests the optimal method of teaching is congruent with students’ learning styles (Dunn/Griggs, 2000). In addition, students come into the classroom with varied backgrounds in the field of computers. A recent high school graduate might say, ‘I’ve had Microsoft Word since the seventh grade’. The non-traditional student might say, ‘I’ve never used a computer before’. Then you might hear the proverbial, ‘I can’t even type’. Although a low percentage of students are in the latter category, placing them into the classroom with the more computer literate student causes a problem for the instructor to challenge one student while not frustrating another student. The solution is to find a common starting point that will meet the needs of all students. To do this, it is necessary to know at what level of literacy the student brings with him/her into the classroom. 3. TEACHING STRATEGY Determine Levels of Knowledge To identify the specific level of knowledge a particular student has in a specific application, an information sheet is developed and administered on the first day of class. Results of this survey are below contrasting two different semesters; Spring 2003 (Table 1) and Spring 2005 (Table 2) of the Indiana University of Pennsylvania Punxsutawney Campus Microbased Computer Literacy classes. Our Computer Literacy courses teach not only computer concepts, but also Email, Windows XP, Microsoft Word 2003, Microsoft Excel 2003, Microsoft Access 2003, and Microsoft PowerPoint 2003. Students were to circle their degree of knowledge in each of those areas, as well as Using the Internet and Using Instant Messenger. The scale was from 5 (highest degree of knowledge) to 1 (lowest degree of knowledge). There was also an area to indicate if the student was completely new to computers. Table 1 shows a chart of the results. The gray shading indicates the highest percentage in each category. Table 1: Survey Results for Spring, 2003 Subject 5 4 3 2 1 Electronic Mail 43% 30% 11% 5% 5% Windows XP 27% 35% 19% 8% 8% Word 2003 37% 30% 16% 7% 9% Excel 2003 9% 18% 24% 19% 28% Access 2003 6% 12% 18% 26% 33% PowerPoint 2003 16% 18% 18% 18% 32% Internet Usage 42% 40% 8% 5% 5% Instant Messenger 35% 26% 12% 14% 13% Table 2: Survey Results for Spring, 2005 Subject 5 4 3 2 1 Electronic Mail 35% 37% 15% 6% 4% Windows XP 16% 37% 24% 8% 11% Word 2003 23% 41% 27% 3% 4% Excel 2003 3% 15% 41% 15% 19% Access 2003 3% 8% 27% 28% 30% PowerPoint 2003 6% 27% 28% 20% 15% Internet Usage 41% 39% 8% 5% 5% Instant Messenger 34% 25% 11% 13% 12% Upon comparing the two semester’s results, we see that student’s usage of each subject has increased in a two-year span, with the exception of Electronic Mail, which was unchanged. Table 3 depicts this data. Tables were attained by averaging the highest results (5) and the next highest (4) categories. Table 3: Comparison Chart of Average Usage Subject Spring 2003 Spring 2005 Incr in use E Mail 73% 73% none Windows XP 53% 62% 9% Word 2003 64% 67% 3% Excel 2003 18% 27% 5% Access 2003 11% 18% 7% PowerPoint 2003 33% 34% 1% Internet Usage 80% 82% 2% Instant Messenger 59% 61% 2% The purpose of these charts is to establish the fact that students are coming into the computer literacy classroom with a higher level of knowledge than in the past. This data helps the instructor plan teaching strategy; in determining each particular student’s knowledge of a subject and develop a training program based upon their individual needs. Students can work through a computer-based activity at their own pace. Rather than 25 individuals working together on one activity, technology allows independent completion of work. Those who fall behind can receive an instructor’s individualized attention while others can begin to tackle tasks that are more complex (Adams & Burns 1999). Based upon the facts that students’ knowledge base of computer applications is diverse, this necessitates the need for the methodology of teaching the computer 101 classes. We need to change from the traditional classroom setting, where all material is presented to all students at one time, to a learner-centered environment, where more individualized instruction occurs. Students can work through a computer-based activity at their own pace. Rather than 25 individuals working together on one activity, technology allows independent completion of work. Those who fall behind can receive an instructor’s individualized attention while others can begin to tackle tasks that are more complex (Adams & Burns 1999). Classroom Environment Once the students’ base knowledge of the subjects is determined, the classroom environment can enhance learning. Many computer labs are not conducive to the greatest atmosphere in which to teach. The instructor often sees many students’ backs and there is a computer between you and the student. The back of a CRT monitor may obstruct the view of the student. A seating arrangement based upon the results of the survey assists not only learning the student’s names, but also placing the beginner students ‘at arms length’ from the instructor. This allows for more individualized assistance when demonstrating applications. Choose a Web-Based Software Package As stated previously, the students enter the classroom with more computer literacy skills, but their levels of knowledge differ. This makes it difficult to start at any one point; you will either bore the more advanced student, or lose the novice. Web-based training and assessment software provides the answer to this dilemma. There are various publishers marketing such software. Some of the more common publishers and products are: * Course Technology Publishing: SAM 2003 Assessment & Training http://www.course.com * Prentice-Hall Publishing Train & Assess IT Generation http://vig.prenhall.com * McGraw-Hill Publishing SimNet for Office 2003 Enterprise Office Suite, 1st Edition, http://www.mhhe.com 4. ADVANTAGES OF WEB-BASED TRAINING There are several advantages of using web-based training. Using Course Technology’s SAM as an example, the following explains how web-based training individualizes instruction, provides feedback to the student, and allows the instructor to track the progress of the student’s assignments. This is a benefit to the instructor as it allows the instructor to give more personalized instruction to each student. Individualized Training Training is available in the following platforms: Windows XP, Internet, MS Excel 2003, MS Word 2003, MS Access 2003, MS PowerPoint 2003. The instructor can assign tutorials for each section based upon the textbook used. Each publisher has personalized the training to the tutorials of their own textbooks. The student can train at his/her own level of knowledge. The training module has four stages: Prepare, Observe, Practice and Apply. It is the old platform of telling them what you are going to do, show them how to do it, have them practice it and lastly, have them apply it. This author has had favorable comments from students using the training software. The beginner students especially liked the training as they could work at their own pace as well as at their own level. Whenever a tutorial is assigned, the student can view the concepts taught in that tutorial to determine whether he/she needs to complete the training session or can go on to the next. Figure 1 shows an example of such a list: Figure 1: Task List The student, upon studying the list of concepts, can determine whether he/she needs to complete that specific module. The novice may need to begin with tutorial 1, conversely the more experienced student may choose to begin with tutorial 3. This allows the student to take responsibility for his/her own learning. Training Progress Students are able to chart their progress as identified in Figure 2. The training Progress screen has five sections: Skill Set, Activity, Task Completed, and Launch Training. This list is very detailed and informative, allowing the student to keep track of their progress throughout the tutorial. Figure 3 shows how the instructor can also track a students’ progress, giving a detailed listing of all students in the section, and the degree to which they have completed the assigned tutorials. This allows for charting the progress of a class. 5. ADVANTAGES OF WEB-BASED ASSESSMENT As well as ease of administration, computerized tests may have other advantages over their pencil and paper antecedents (Buchanan, 2000). Based upon a student’s learning style, students can tailor their use of the assessment. Some individuals learn faster than others. Web-based assessment enables students to work at their own pace. Instructor Preparation The ability to tailor-make exams is an immense advantage to the instructor. Instructors can choose from a wide variety of objective as well as hands-on application questions relative to the concepts taught. On the other hand, an instructor has the option of creating their own questions and adding them to the test bank. The use of scenarios in the SAM product allows the creation of multiple exams used for administration of more than one section of students. SAM 3.0 has 2,500 performance-based task scenarios for all applications mentioned above. Course Technology also has a SAM Computer Literacy Package that includes all of the above plus training in computer concepts. All SAM versions also include access to over 60,000 objective test-bank questions from various Course Technology textbooks. It is reiterated at this point that computer concepts is an area reserved for classroom discussion upon presenting a chapter. However, SAM evaluates the student’s knowledge of the topic. Assessing Tool By the end of a unit, the student is responsible for completing the training for the application taught. The exams given are also in SAM and correlate to the training. Figure 4 is an example of a level 1 PowerPoint question. The screen displays the name of the student, the name of the exam, the question and how many attempts they have to answer the question. The recommendation is to have three attempts for each question with or without feedback for the answer given. A timer is available for use at the instructor’s discretion. The question displayed is in a simulated environment of the product so choosing the Help function is not an option. Upon completing the exam, the student has instant feedback and can obtain a detailed Exam Study Guide (Figure 5). Displayed on the guide are areas of weakness, whether they completed the training for that concept, and how many questions in the training they have completed. It even permits them to launch the training for the questions and observe the procedure for completing a question correctly. The value of assessment which is used to provide feedback to students rather than to evaluate them for course grades, is well known. Students, having received feedback on their performance, may then take steps indicated by that feedback to remedy whatever weaknesses the assessment has exposed (Buchanan, 2000). 6. WEB-BASED EXAMS VS TRADITIONAL MULTIPLE-CHOICE EXAMS When testing a skill, i.e., Word, Excel, Access or PowerPoint, it has been this instructor’s conclusion that students perform better when using the assessment software over the traditional multiple-choice, true/false exam. Statistics from three different sections of computer 101 classes in the Spring of 2005 at the Punxsutawney Campus of Indiana University of Pennsylvania are as follows: As indicated in Table 4, students scored 10 percentage points or higher on the web-based assessment verses the traditional exam. Table 4: Comparison by Exam Type Microbased Computer Literacy Survey Traditional Exams vs Web-based Exams Average Results, Spring, 2005 Traditional Exams Web-based Exams Section Avg Section Avg 301 73% 301 83% 302 73% 302 84% 303 73% 303 84% Table 5: Comparison by Semester Fall, 2000 No Web-based training or assessment Spring, 2005 Web-based training and assessment Section Avg Section Avg 201 75% 301 83% 202 78% 302 84% 303 85% Table 5 shows the comparison of the effect of using web-based training or assessment software in two different semesters. Microsoft Word and Microsoft Excel exams were given in both semesters. The method of instruction in both semesters remained constant; instructor lecture and demonstration. However, Fall, 2000 students were assessed using an objective exam, and had no method of individualized training. Spring 2005 students were able to use the web-based training to better master their skills. It has been this author’s experience that students have a better mastery of the subject when working in a learning-centered environment such as a web-based training program. As you can see, the scores increased by at least 7 percentage points using the web-based training and assessment. According to another study, Web-based education systems resulted in significantly better learning as measured by an average of 10 more points on a 75-point quiz (Radhakrishnan, 1997). This is what web-based training and assessment offers the student. It gives the student the ability to be responsible for their learning. This is a refreshing change from the computer games that hypnotize and mesmerize, requiring no thought; just mechanical functions rather than a cognitive activity. Exemption Exam Another use of the web-based assessment package is the administration of a campus-wide exemption examination from the computer literacy course. Such an exam is offered at Indiana University of Pennsylvania twice a year during the first week of the semester, the drop/add week. This allows the more advanced student to test out of the course. 7. DISADVANTAGE OF WEB-BASED TRAINING AND ASSESSMENT A drawback of using web-based training and assessment is the integration of the Microsoft Office products. In addition, McQueen (1999) found that tutorials, while excellent, emphasize skills over critical thinking. Since e-Courses cannot cover all topics, instructors should still incorporate projects and examples that emphasize those curriculum goals not covered in the tutorials. One project used by this instructor is to create labels in Microsoft Access, create a form letter in Microsoft Word and use the Officelinks feature in Access to create a mail merge project. Another integration module might be to create a spreadsheet and use the paste special function to link that spreadsheet into a Word document. Other options could be to link Excel charts into PowerPoint, or import Excel data into Access. 8. CONCLUSION The Educational Testing Service plans to unveil a standardized test designed to measure the computer savvy and information literacy of college students. The test will evaluate how well students can judge the validity of online information, build a spreadsheet, compose e-mail messages summarizing a passage, and perform other tasks. The test is unique because it attempts to measure not only proficiency in using computer software, but also information-processing skills that are increasingly important in college and in many jobs (Young, 2004). We college instructors have a responsibility to graduate computer literate students. Computer literacy must go beyond knowing how to use the computer. The use of web-based training and assessment will help the student become proficient in computer applications. We need to educate a more self-sufficient, independent student thinker, capable of making online choices and becoming a responsible member of the information age. Web-based training and assessment can be a tool toward that goal. 9. REFERENCES Adams, S. Burns, M. (1999) Connecting Student Learning & Technology. Washington, D.C.’ Office of Educational Research and Improvement. (ERIC Document Reproduction Service No. ED428759). Buchanan, T, (2000) “The Efficacy of a World-Wide Web Mediated Formative Assessment,” Journal of Computer Assisted Learning, (2000) 16, 193-200. Burton, T, (1997) “Visual Literacy And Just-In-Time-Training: Enabling Learners Through Technology,” Selected readings from the Annual Conference of the International Visual Literacy Association, January, 1997. Cooper, S., (2004) “Computerized Practice Tests Boost Student Achievement.” T H E Journal, Volume 32, Issue 2 Sept 2004. Dunn, R., Griggs, S. (2000). “Practical Approaches to Using Learning Styles in Higher Education,” Westport, CT: Bergin & Garvey. Embry, J., (2005) “Lawmaker proposes tinkering with TAKS; online testing touted as more efficient way to gauge learning.” The Austin American-Statesman, News Section, February 10, 2005.Pg A1. Hoberock, B., (2005) “Student testing is going high-tech,” The Tulsa World, News; Tulsa; Education Section, January 21, 2005, Pg. A13. Kovac, A., L., (2005) “A Link to Learning; Program shows each pupil how to improve.” Herald News, Our Towns Section: March 11, 2005. Pg. B01. MacDougall, G, Place, C, Currie, D, (1998) “Web-Testing: A Form-Based Template for Creating Multimedia Enhanced Tests.” ED_MEDIA/ED-TELECOM 98 World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications. Proceedings (10th, Freiburg, Germany, June 20-25, 1998). McQueen, T. F., Fleck, Jr, (1999) “An Evaluation of Alternative Technology-Based Instructional Formats.” T.H.E. Journal, Volume. 26 Issue 11, (June 1999) p108. 8p. Radhakrishnan, S., Bailey, J. E., (1997) “Web-Based Educational Media: Issues and Empirical Test of Learning,” National Science Foundation, Arlington, VA, 1997-11-00. Young, J., R. (2004) “Testing Service to Unveil an Assessment of Computer and Information Literacy.” The Chronicle of Higher Education, Information Technology Section, November 12, 2004.. Pg. 33.