The Center for Computing Education Research (CCER): A Nexus for IS Institutional and Individual Assessment Lynn J. McKell mckell@byu.edu Brigham Young University Provo Utah 84602 USA John Reynolds john.reynolds@cis.gvsu.edu Grand Valley State University Allendale, Michigan 49401 USA Herbert E. Longenecker hlongenecker@usouthal.edu Jeffrey P. Landry jlandry@usouthal.edu Harold Pardue hpardue@jaguar1.usouthal.edu University of South Alabama Mobile, Alabama 36688 USA Abstract The IS2002 Model Curriculum is a well researched and promoted standard for IS education. Over the past four years a coalition of schools formed to create an assessment exam to be administered to students for measuring performance against objectives of the Model Curriculum. Analysis of examination results provides reports to institutions for curriculum evaluation and development. A Center for Computing Education Research is being formed to manage this process. The ICCP is offering a new professional “Information Systems Analyst” (ISA) certification for students performing at a prescribed level on the assessment exam. Keywords: Model Curriculum, Assessment, Certification, Program Improvement 1. INTRODUCTION The IS2002 Model Curriculum outlines topical areas appropriate for a university four year program in Information Systems. The IS2002 revision is the latest update to a sequence of curriculum recommendations which were first reported in 1972 under the sponsorship of the ACM. More recent updates have added the AITP and AIS as co-sponsoring organizations. The Institute for Certification of Computing Professionals (ICCP) and the Center for Computing Education Research (CCER) formed a coalition of 40 universities and colleges to develop an outcome assessment examination reflecting the objectives and learning units espoused in the IS 2002 Model Curriculum. Versions of this examination were beta tested in 2003 by over 500 students and in 2004 by over 800 students enrolled in IS related programs at coalition schools. An operational version of the examination has been administered during 2005 to students at participating institutions. The assessment examination serves as a common instrument for integrating two related goals: (1) to provide institutional feedback on programmatic preparation of graduates consistent with the IS 2002 Model Curriculum – facilitated by summarizing and analyzing individual performance data on the assessment examination; and (2) assessment of individual student performance for the purpose of credentialing entry level professionals based on their knowledge of information technology concepts and skills commonly taught in IT related programs. To accomplish the first of the above goals, the CCER was formed to provide the assessment examination and related programmatic evaluation services to institutions. To accomplish the second goal, the ICCP created a new vendor-neutral professional certification titled “Information Systems Analyst” (ISA) based on successful assessment examination performance and completion of an IT related undergraduate degree. This paper describes these developments and explains how institutions may participate in the programmatic evaluation and individual credentialing programs. 2. BACKGROUND In 1951 the National Machine Accountants Association (NMAA) was founded in Illinois. In 1962 the NMAA name was changed to the Data Processing Management Association (DPMA – now Association for Information Technology Professionals - AITP). Also in 1962 the DPMA established the “Certified Data Processor” (CDP) Credential. In 1973 the ACM and several other major professional computer societies joined the DPMA to organize the Institute for Certification of Computing Professionals (ICCP) to manage the computer professional credentialing process and related certifications. The ICCP governing board expanded the initial DPMA assessment to an integrated set of examinations, which, in various combinations with experience, education, and commitment to a code of ethics, formed the basis for credentialing individuals with various professional certifications including: 1. Certified Computing Professional (CCP) – ICCP flagship certification requiring experience and passing 3-4 examinations. 2. Associate Computing Professional (ACP) – entry level certification based on 2 examinations; no experience required. 3. Certified Data Management Professional (CDMP) – Practitioner and Mastery levels; Offered under sponsorship with the Data Management Association (DAMA). 4. Certified Business Intelligence Professional (CBIP) – Practitioner and Mastery levels; Offered under sponsorship with The Data Warehousing Institute (TDWI). 5. Information Systems Analyst (ISA) – entry level certification based on the IS2002 Model Curriculum Assessment Examination and receiving a related Bachelors Degree The ICCP certifications have been relatively unique in the profession because they are vendor neutral, bridging a range of knowledge from specific technical skills to strategic level technology management and problem solving abilities. The specific examinations and certification requirements have been rigorously managed by a certification council, with periodic review and updating ongoing to retain currency and relevance. Other vendor-neutral certifications differ from the ICCP credentials in the following ways: 1) The Certified Information Technology Professional (CITP) is an add-on credential which requires an individual first be a CPA; it also possible to receive the CITP on the basis of experience without an examination or education requirement. 2) The A+ certification is oriented towards basic hardware and software skills – not professional systems development. 3) The IC3 certificate is oriented towards basic computer end-user skills – not professional systems development. In contrast to the three above designations, ICCP credentials are oriented to a much higher (strategic) level of professional capability as demonstrated by a combination of experience, education and examination. Parallel to the above professional certification activity, educational institutions have developed computer and information technology related curricula. Coincidentally, in 1962 the first university level Department of Computer Science was founded at Purdue University. In 1968 a Management Information Systems graduate degree program was established at the University of Minnesota. Subsequently other CS/IS/IT academic programs have proliferated. Concern over educational standards has precipitated the development and publication of several model curricula, the most recent of which was released in 2002 under sponsorship by the ACM, AITP, and AIS (Association for Information Systems). [Gorgone, 2002] 3. PROGRAM ASSESSMENT The process of evaluating, assessing and accrediting CS/IS/IT related educational programs has been a topic of discussion for some decades. In 2001 a panel discussion at the annual Information Systems Educators Conference (ISECON) [McKell, 2001] focused on this topic by addressing the question whether or not existing ICCP certifications could be used as a complement to program accreditation by extrapolating compliance with the IS Model Curriculum guidelines from individual performance scores on ICCP examinations. This was interpreted to mean “Could ICCP examinations be used to assess the preparation of undergraduate students relative to the objectives of the IS Model Curriculum?” (At that time the latest version was the IS’97 Model Curriculum [Davis, 1997].) Participants on the panel included one of the authors of the Model Curriculum and a member of the ICCP Certification Council. From that discussion (and a subsequent ICCP Certification Council point by point mapping of the ISCore exam to the IS2002 model curriculum) emerged the general conclusion that existing ICCP examinations were not calibrated for measuring compliance with the IS Model Curriculum. Follow on discussions after the ISECON panel and the ICCP Certification Council analysis concluded that in order to adequately assess student performance relative to the Model Curriculum, a new examination instrument was needed – mapped specifically to the skills and learning units espoused in the IS 2002 Model Curriculum. This was accomplished when a coalition of undergraduate IS educators representing 17 institutions met in February 2003 in a workshop to craft the new examination. This workshop was sponsored by the ICCP and the University of South Alabama, and hosted at Mobile, Alabama. The development methodology basically followed procedures established and tested by the ICCP Certification Council. Teams of educators were organized based on individual specialties, and were assigned to develop a given minimum number of questions in the specified topical area based on the weighting for that area in the Model Curriculum. All questions were in the multiple choice format with 4 choices for answers (3 distracters and 1 correct answer). The goal was to produce a 100 question exam which could be machine graded, that would measure student performance near the time of graduation from a relevant undergraduate degree program patterned after the IS 2002 Model Curriculum. The workshop successfully created the specified examination – referred to as the IS CORE examination. This examination was beta tested in two primary venues resulting in useable data points from over 500 students. Beta testing included students attending the AITP National Collegiate Conference held at Purdue University, West Lafayette, Indiana in March, 2003. Approximately 175 of the 500+ student attendees took the exam. Additionally, the exam was administered to students at the 17 coalition schools who had participated in the exam development workshop. Through the beta testing the overall results verified that the examination was reasonably rigorous, though a few faulty questions were identified, which were subsequently corrected or replaced. Though this beta test was deemed successful, subsequent analysis led to the conclusion that the initial examination was not sufficiently robust to meet the objective of extrapolating program compliance with the IS 2002 Model Curriculum, the reason being that a 100 question examination did not sample enough of the combinations of outlined skills and learning units. Thoughtful review of the Model Curriculum suggested that 258 questions would be the optimal number for accomplishing the stated purposes. With this goal in mind a subsequent workshop in Summer 2003 and follow on work by email correspondence among the coalition schools resulted in producing the 258 question IS CORE exam for beta testing during the 2004 season. The 258 questions touch on the various combinations of learning units and skills weighted consistent with the content of Model Curriculum. The number of coalition schools expanded to over 30 with nearly 1,000 students taking the revised IS CORE beta examination. After analyzing the results and fine tuning some test items, an operational version of the exam was issued for use in Winter/Spring of 2005. It is noted that further fine tuning may occur as deemed necessary by coalition schools and by the ICCP Certification Council. Time allocation for the IS CORE examination was carefully considered. Assessment professionals advised that for an examination of this type at the college level there should be an allowance of approximately 40 seconds per question. A quick calculation shows that the 258 question examination should, therefore, be allocated 172 minutes. For convenience and administrative purposes this has been rounded to 3 hours (180 minutes; i.e. 42 seconds per question). Student performance data is analyzed to produce reports for each institution showing how well the institution is meeting model curriculum objectives in the areas of specific learning units. The reports also show aggregate performance measures of an institution’s students relative to the whole population. 4. CENTER FOR COMPUTING EDUCATION RESEARCH While analysis of student performance data on the IS CORE examination is a vital component in assessing the compliance of relevant undergraduate educational programs as measured against the IS 2002 Model Curriculum, there are other components which will be useful, including mapping an institution’s existing program to the skills and learning units of the Model Curriculum. With a mapping in place, IS CORE examination data can be used to pin point areas of strength and weakness in an institution’s program. The administration of assessment, analysis and reporting functions suggested the need for an organization to serve as the nexus and supplier of these services. In order to meet the above need a Center for Computing Education Research (CCER) is being organized within the ICCP Education Foundation (ICCP/EF) [ICCP/EF, 2005]. The ICCP/EF, formed two decades ago, is sympathetic to aims of the ICCP but focuses on education related issues, and operates at an arm’s length legally as a separate 501c3 entity, with a board of directors separate and distinct from the ICCP operations. As such the ICCP Education Foundation is well positioned to seek external grants and research funding in support of educational objectives, and to host the CCER which will administer a package of institutional assessment services based on the IS CORE examination. It is intended that over time the CCER will develop a rich database of institutional and national (ultimately international) statistics which may provide additional information for helping institutions assess the goodness and validity of their IS-related undergraduate programs, and will support a wide variety of education research consistent with the mission of the ICCP/EF and the CCER. The CCER contracts with institutions, not with individuals, to provide the assessment services and reports described above. Currently, institutional costs range from $1,000 to $2,700 per year based on number of students and institutional involvement in the examination maintenance/update process. Institutions can direct inquiries to the ICCP/EF [ICCP/EF, 2005]. 5. “ISA”–CERTIFICATION BASED ON THE IS CORE EXAMINATION With the IS CORE assessment examination in place, a reciprocal question was posed to appropriate CCER and ICCP officials. In reverse contrast to using ICCP examinations for Model Curriculum assessment, the new IS CORE examination begs the question: “Could the IS CORE examination be used as the basis for a relevant credential?” CCER and coalition representatives outlined the requirements and proposed a new certification to the ICCP. This proposal was reviewed favorably and approved by the ICCP Certification Council. A contract exists between the ICCP and ICCP/EF whereby the ICCP/EF will pass relevant individual exam response data to the ICCP who is now prepared to issue a new professional credential titled “Information Systems Analyst” (with the acronym, “ISA”). The qualification requirements for the Information Systems Analyst (ISA) certification are: 1. Receive a four year college degree in a CS/IS/IT related area. 2. Ascribe to the ICCP code of ethics. 3. Participate in a recertification program. 4. Pass the IS CORE examination at an acceptable level. Those who pass at the 50%-69% level would be designated as ISA-Practitioner level. Those who pass at the 70% or higher level would be designated as ISA-Mastery level. Notably absent in the above criteria is an experience requirement. The ISA certification is specifically intended as an entry level credential and, therefore, is issued irrespective of experience. In the ICCP scheme of certifications, the ISA would generally be considered at a higher level than the ACP, but not as high as the CCP, which does have an experience requirement and requires a higher level of performance on both general and specialization examinations, and acknowledges credit for academic degrees. A normal career path would very likely involve an ISA recipient later qualifying for the CCP certification after several years of relevant experience. In either event, professionalism strongly suggests that holders of a professional credential should maintain their competency over time by enrolling in a re-certification program promoting procedures for professionals to confirm their ongoing competency. These procedures may include educational experiences such as formal coursework or technical seminars and/or passing additional certification examinations. 6. EXAM TIMES COMPARISON As noted above, the IS CORE examination is allocated 3 hours (180) minutes. Other ICCP examinations are either 90 minutes (for the regular ICCP Core exam and various specialization examinations) or 60 minutes (for language examinations). While the ISA requires only the IS CORE examination (3 hours), other ICCP certifications require that the candidate pass several examinations – a minimum of two exams for the ACP (the regular ICCP Core and a language exam) for a total allowed examination time of 150 minutes; - and a minimum of 3 exams for the CCP (the regular ICCP Core and two specialization exams) for a total allowed examination time of 270 minutes. This comparison suggests further validation that the ISA is a more rigorous credential than the ACP, but less demanding than the CCP. 7. INTEGRATION The new IS CORE examination has been designed to meet two complementary objectives. First, it is used to provide data relevant to assessing educational programs and curricula in comparison to the IS 2002 Model Curriculum. Second, this same examination also serves as the primary individual skill and knowledge evaluation tool used to assess the qualification of IS program graduates for receiving the Information System Analyst (ISA) credential. Consequently, the IS CORE examination successfully integrates curriculum objectives with professional certification in a way rarely achieved in any academic and industry cooperative effort. Also, the vendor-neutral ISA credential (aligned with the IS2002 Model Curriculum and the industry Code of Ethics, and issued by the ICCP certifying agency) provides a mechanism for validating qualifications of IS program graduates seeking entry level employment. 8. A CALL FOR ACTION The globalization initiatives of outsourcing and off-shoring introduce new issues with respect to assessment of professional qualifications. With the emergence of educational standards in the form of the IS2002 Model Curriculum and Professional Standards in the form of the IS Core Examination and the ISA Certification, we are calling on colleagues worldwide to participate in the CCER Institutional Assessment Program and promote the related entry level vendor-neutral credential. It is only with widespread support that standards become meaningful to the profession at large and in the worlds economic and social fabric. We further encourage institutions to support efforts to upgrade and maintain the assessment instruments and processes and to use the collective results for program improvements. There are many ways that IS Institutional Programs can participate, including: 1. Join the CCER to administer the IS Core assessment examination. 2. Use the IS Core examination results as a factor in planning adjustments and enhancements to IS related curriculums. 3. Support qualified professors to participate in IS Core examination maintenance and update. 4. Promote the ISA Certification to students graduating from IS related programs. 5. Promote other experience-based credentials as a normal professional career path. 6. Promote on-going recertification as an individual standard for maintaining professional qualifications. 9. SUMMARY Building on the work of major professional societies in computer related areas, a Center for Computing Education Research (CCER) is being organized to provide a package of institutional IS Program assessment services. The main instrument used to support these services is a new IS CORE examination – an individual outcome assessment examination mapped to the IS 2002 Model Curriculum. These assessment results may be useful both for internal program development and for external accreditations. The examination also supports the individual ISA certification offered through the ICCP. Institutions offering undergraduate IS-related baccalaureate degrees may obtain further information from the Center for Computing Education Research [ICCP/EF, 2005]. Information on the ISA certification can be obtained from the Institute for Certification of Computing Professionals [ICCP, 2005]. 10. REFERENCES Davis, G.B., J.T. Gorgone, J.D. Cougar, D.L. Feinstein, H.E. Longenecker Jr., “IS ’97 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems,” ACM, New York, NY and AITP, Park Ridge, IL. 1997 Gorgone, J.T., G.B. Davis, J.S. Valacich, H. Topi, D.L. Feinstein, H.E. Longenecker Jr., “IS 2002 Model Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems (Draft),” ACM, New York, NY, AIS, and AITP (formerly DPMA), Park Ridge, IL, 2002. ICCP (Institute for Certification of Computing Professionals, 2350 E. Devon Ave., Suite 115, Des Plaines, IL 60018-4610 USA, phone 800-843-8227. 2005. ICCP/EF (ICCP Education Foundation/ -Center for Computing Education Research), PMB #226 310 Busse Highway, Park Ridge, IL 60068-3251. 2005. McKell, L.J. (moderator), “Individual Certification: A Complement to Program Accreditation.” Panel discussion, ISECON meeting sponsored by the AITP EDSIG and the AITP Education Foundation, Cincinnati, Nov., 2001.