An Evaluation of Alternative Technology-Based Instructional
Formats.
:
Diversity in background and learning styles of students can
be a problem in college or university computer literacy
courses. Students typically range from those who have never
used a mouse before to those who have already mastered
50 percent of the course's basic skills. The students who
have not mastered the mouse require special attention, while
the others are bored and may disrupt the class. Assessment
can aid in tailoring a curriculum designed to eliminate
deficiencies, but this approach can be expensive, due to the
increased labor involved. Three alternative solutions are
presented: vendor-developed courses delivered over the
Internet; Vendor-developed tutorials and tests; and
instructor developed/vendor assisted Web-based instruction.

Many forces influence what is taught at an institution and how knowledge and computer skills
are developed, presented, and delivered to students. Some of the pressures facing institutions
and instructors are shown in Figure 1.

Student Skills

Students have always brought their personal learning frameworks to colleges and
universities. As members of the Academy, we have encouraged and often applauded diversity
in learning styles and background. However, that same diversity of learning styles can cause
problems for students, institutions and faculty when basic student skills are uneven. This is
especially true in computer literacy courses. Some students may have great mastery of the
material from work or other venues, while others, often older students, may have little or no
familiarity with computer operations or terminology. Introductory skill courses such as those
designed around MS Office may include students who have never used a mouse as well as
those who have mastery of 50% or more of the required skills. Those who do not know how
to use the mouse require individual attention while those with extensive skills are bored and
often disrupt the class with attempts to demonstrate their knowledge.

Costs and Resources

Limited faculty resources and the high costs of computer equipment exacerbate these
classroom management problems. Alternatives discussed and used at many institutions
include larger sections or using junior faculty or teaching assistants. These solutions may
reduce costs but may also reduce quality. If quality is reduced, successive courses that rely
on the skill courses to develop student proficiency may find that extra class hours are
required to develop missing skills. In such cases, arguments are sure to arise whether the
material was covered (in the skill course), covered well, or conveniently forgotten by the
student.

Assessment

A partial solution to student knowledge level is assessment. Assessment, when properly
applied, can help an institution, instructor, and student determine if specified skill sets have
been mastered. Assessment can take place before enrollment in a course, during the course,
or as part of the subject mastery evaluation at the end of the course.

Part of the assessment process requires setting objectives and then assuring that exams and
related materials focus on, and link to, those objectives. A key to successful assessment is
developing objectives that are both measurable and acceptable. The objectives must be
acceptable to faculty, administration and other publics. Those publics include students,
parents, legislative bodies and accreditation agencies. External validation has become a
phrase that underlies assessment.

Proper assessment procedures also provide guidelines for pre-assessing students and then
tailoring a course or curriculum to deficiencies identified by assessment tools. Such a process
can be labor intensive and may conflict with institutional needs to control costs.

Other Academic Institutions

Courses and curricula are not developed in a vacuum. Students are becoming less place
bound and often seek educational alternatives. State mandates for "equivalent" education at
all state universities can mean a loss of institutional identity. Mature students may also "shop"
for the institution with the most convenient class and registration schedule. The rapid
deployment of distance learning, especially Internet-based courses and degree programs,
adds an additional competitive element to the educational options available to students.

Overview of Alternatives

The authors teach at Columbus State University, a regional institution with a mixture of
traditional and non-traditional students; most students are primarily non-residential and
part-time. Both authors have taught introductory classes in computer skills and have worked
with students possessing all levels of computer competencies.

This paper discusses three alternative approaches to traditional classroom instruction. The
guidelines and comments provided in each section and at the end of the paper are designed
to help other faculty faced with finding instructional solutions within environments similar to the
ones described above.

Three alternative approaches are currently in use at Columbus State University:

1. Vendor (NETg;
www.netg.com) developed Internet-delivered courses

2. Vendor (Course Technology;
www.course.com) developed tutorials and tests

3. Instructor developed and vendor-assisted Web-based instruction

The first two alternatives discussed are self-paced courses. While the last alternative could be
self-paced, this alternative has been used more as a support device rather than the primary
delivery vehicle. Several options are discussed within this alternative.

NETg

The depth and scope of the technology revolution has caused a dramatic increase in the
number of course delivery alternatives. Many colleges and universities are using the Internet
to deliver degrees, courses or course components. If colleges and universities are to remain
competitive in the higher education market, it is imperative that professors and administrators
consider this option. Developing and implementing alternate methods of course delivery, such
as distance learning via the Internet, helps meet student needs by providing flexible
scheduling and 24 by 7 access to course material. Internet courses can also reduce travel
time and minimize conflicts between job and home. However, the traditional model of a
University, where the professor lectures and the student listens, is not congruent with the
Internet model of content delivery. Therefore new course management tools are needed.

Columbus State University currently provides instruction for selected courses via the Internet.
These courses include Technological Solutions and components of certain courses, such as
the Introduction to Computer Information Systems. Software developed by the National
Education Training Group (NETg) was purchased for use in this endeavor.

NETg Components

NETg is a Web-based interactive, self-paced tutorial in a simulated environment. Courses
available include application packages such as Word 97 and programming languages such
as C++. Pre-assessment and end-of-unit "mastery" instruments are available for each learning
unit. The material presented in each tutorial is modified based upon the student's weaknesses
identified in the pre-assessment process. Course information is disseminated via a Semester
Class Schedule Booklet; hardcopy and Web page versions ( www.colstate.edu) are
available. Students also learn of course offerings via "word-of-mouth" from advisors and
other students. Students are often excited about the time independence provided by the NETg
courses and request course access before classes are scheduled to begin.

The Class Schedule Booklet clearly identifies each NETg course and provides additional
information including the class URL. Students contact the course professor either by e-mail or
telephone to request the proper procedures for accessing the course and logging on.
Student access accounts are created and passwords are distributed by e-mail or during the
course orientation, held the first week of the term.

Student Interaction with NETg

Students are able to complete each unit of course material at their own pace, receiving
immediate feedback to responses made through simulated conditions. Students may work
through the tutorials as many times as needed and repeat the mastery tests as many times
as they wish, for the purpose of improving their scores.

Student progress is tracked electronically. Therefore, progress and performance data are
accessible any time by the student or professor. Access is controlled via password. The
Course Manager software registers the student's mastery scores, as well as the number of
times the lesson was accessed and the amount of time spent on the lesson.

Unfortunately, NETg's reporting scheme can frustrate students. Following the mastery test, the
immediate score will be different from the one displayed in the Student Report. Test scores
are not updated until the subsequent logon.

Students with partial or complete knowledge of the programs claim that taking the
pre-assessment tests is a waste of time. Students with no prior knowledge dislike the constant
and negative feedback and low scores received from the pre-assessments. When students
find this annoying, they simply discontinue taking pre-assessments and work through the
entire tutorial.

Mid-term and final examinations are given online and students complete tasks in live, not
simulated, programs. Working in the active environment challenges many students, especially
those who have not practiced this type of application. In addition to the active environment,
the examinations are problem-oriented, not task-oriented. In view of this, professors should
provide problem-solving situations throughout the semester or quarter, preferably in a live
program project.

If a student withdraws from the NETg Internet course, a questionnaire is sent to identify the
reason(s) for withdrawing. The most often cited causes for course withdrawals were:

1) lack of adequate computer hardware at home

2) job or family responsibilities

3) underestimating the amount of time needed to complete the tutorials

Required Computer Skills Class

The College of Business at Columbus State University requires that all students be computer
literate and proficient in Windows 95, the MS Office 97 Suite, Internet and Web use, and
automated library software. For a number of terms, this course was taught using a traditional
lecture-discussion-lab format. In this format, instructors worked from a common foundation
syllabus, text and course objectives, but developed and administered projects and tests
independently of each other.

Earlier versions of the course used Symphony, WordPerfect, Office 4.2 and other software
packages. In those earlier terms, most students were unfamiliar with basic computer concepts
and skills. Hence, students entering the course generally had a uniform starting point: zero
skills. In recent years, students came to the class with a wide range of skills, frustrating
attempts to define a common starting point. Attempts were made to separate the course into its
various components and award credit based on skill development. This approach was a
scheduling and administrative nightmare soon abandoned. In addition, testing methods were
primitive, time consuming, and test materials were not as congruent with learning objectives
as desired by instructors. Simultaneously, program growth was draining resources as more
and more faculty were being allocated to the skills course and away from other tasks and
courses. It was in this environment that alternative methods and tools were sought.

A number of textbook vendors were contacted. Before contacting vendors, criteria were
established to evaluate alternatives. The abbreviated criteria list included:

1. The course would be self-paced using extensive tutorials that relied on a live rather than
a simulated environment. We believed that students should be taught using the environment
they would use in other classes, not an artificial environment or simulated desktop. In this
environment, the faculty would establish testing times, include additional projects, and provide
support and coaching while reducing the amount of time lecturing.

2. Testing would incorporate a live environment rather than a simulated environment and
testing modules would be of sufficient breadth and depth to permit testing at various time
periods without compromising test integrity. In addition, the system should have some form of
pre-assessment methodology.

3. Materials provided by the vendor would have to include the full MS Office 97 Suite and
Windows 95.

Course Technology e-Course

Course Technology's solution, e-Course, using the textbook, Microsoft Office 97 Certified with
Microsoft Windows 95[1] (ISBN 0-7600-7224-8) was selected. The textbook includes a
CD-ROM with tutorials for each of the MS Office 97 products and Windows 95. The
tutorials on the CD can be loaded on a local machine or run from a server. If loaded on a
local machine, the tutorials can either be installed with minimum disk space (which requires
that the tutorial CD be present in the CD-ROM drive) or full install, which places all files on the
hard drive. Similar functionality is available for server installations. We selected server
installation with minimum disk space requirements. This means that students must place their
tutorial CD in the drive to access and use the program. The startup screen is shown in Figure
2.

[Figure 2 ILLUSTRATION OMITTED]

Clearly, e-Course satisfied criteria one and three listed above. Criterion two proved to be
serious problem. An optional module with the text package is a product called SAMs (Skills
Assessment Manager). SAMs, as demonstrated by a vendor representative, included a live
testing environment. Unfortunately, what was demonstrated in mid 1998 was not fully
functional at the start of the fall term. We therefore turned to the alternative offered by Course
Technology: e-test.

Course Technology's e-Test uses a simulated environment that has the look and feel of a real
desktop environment. The testing module has adequate security features and tests are
immediately evaluated. Regrettably, those taking the test cannot use the Help Functions in the
software package and the instructor cannot turn off the exam once a student has started it.
Those wishing to limit the amount of time a student can spend taking an exam cannot do so.
Each exam however was usually completed within 40 minutes.

Course Technology promises to have SAMs operational in early 1999. This version of SAMs
is claimed to have additional features and improved testing options.

Evaluation

Initially, problems were encountered while installing e-Course on the Columbus State
University network. These problems were more related to the CSU system than to problems
with e-Course. Those selecting e-Course should work closely with computer center
personnel and Course Technology technical specialists to minimize problems.

The students seem enthusiastic about the self-paced approach. The course was structured so
that the first three weeks of class were an introduction to class procedures, computer use and
instructor-led exercises. This guaranteed that all students received the same instruction and
had an opportunity to ask questions. One of the essential tasks in the early part of the course
was how to logon and how to effectively use the university's e-mail system. Students were
informed that it was their responsibility to master the materials and check their e-mail on a
regular basis. They could then work elsewhere on their assignments or come to the lab,
where the instructor would be present.

The syllabus specified testing dates and due dates for instructor-developed projects. As the
course moved through the term, attendance at labs dwindled as students found that they
could work at home or places of employment. Those who attended the labs could be
characterized as those who either needed additional help or who wanted the structure of a
fixed time period. Anecdotal information indicates that the shift from attendance to
performance requirements encouraged many who did not already own a computer to either
purchase one or to use one belonging to a friend. Several indicated that their spouse was
also "taking" the course though not officially enrolled.

One of the primary difficulties with the new course structure was the shift for instructors from
lecturers in control to coaches; the tutorials now did the "teaching." A problem with
coaching is the need to be proficient on any topic in Office 97 at any moment, not just the
topic prepared for that day's lecture/presentation.

Students really seemed to appreciate the ability to proceed at their own pace and the
immediate feedback. e-Course tutorials include self-testing and tracking software. Since the
exams are closely linked to the tutorials, and the tutorials are focused on MS MOUS
certification, assessment has become more direct and objective. In addition, students can test
out of the class early as they master topics.

While the structure and content of the course has changed from the traditional lecture format,
it appears that students are achieving more. In addition, the drop rate is less and students
have taken the extra effort to make positive statements concerning both the course and the
text.

Lost CDs

One problem not anticipated were students forgetting to retrieve their CDs from the drive
when they left the labs. Unfortunately, not all CDs were recovered. Since this was a "test"
mode of the product with Course Technology, the company provided several extra CDs that
were loaned to students for the remainder of the term. Institutions adopting this text should
warn students to remove CDs or be prepared for similar problems. Students should use
nonalcohol-based pens to write their names on the CD. While this does not guarantee that
CDs will be returned, it does minimize ownership arguments.

Tracking Progress

Tracking software included with e-Course monitors a student's progress through the
tutorials. A "tracking" disk kept by the student records the amount of time spent on each
tutorial and the scores on self-assessment tests. When a student comes to an instructor for
additional help, the first step is to ask for the tracking disk. On many occasions, the response
from the student is, "I guess I have not been spending as much time on the tutorials as I
should." For those students who have spent time, the tracking disk can help determine those
areas that need remediation. Initially students were required to present a printout of the
tracking disk summary at test time. This encouraged them to complete the tutorials and gave
an overview of the relationship between test performance and the amount of time spent on
the tutorials. As was expected, there was a positive relationship between test results and
tutorial effort. Maintaining the printouts of each student's tracking disk became an
encumbrance and was discontinued after the first exam.

Projects

In addition to the concern with e-Test and SAMs, the material does not lend itself to projects
that integrate all the Office 97 functionality. Likewise, the tutorials, while excellent, emphasize
skills over critical thinking. Since e-Course cannot cover all topics, instructors should still
incorporate projects and examples that emphasize those curriculum goals not covered in the
tutorials. Projects should be assigned so students will learn various tools and techniques
via the Help Function. One project used in the course required students to create a form letter
in Word that retrieved data from an Access database. A query linking tables had to be created
in Access so that Word would retrieve the correct data. The mail merge in Word further limited
the data selection. The query from Access was exported to Excel where additional analyses
and charting were performed. One of the Excel charts was linked and embedded in a
PowerPoint presentation.

Evaluation

Unlike NETg, e-Course does not tailor the tutorials based on pre-assessment. Hence, all
students, regardless of their prior knowledge, are almost required to take the full tutorial. The
text and the CD do provide an overview of the tutorial contents so that a decision can be
made whether or not to take a tutorial, but that is a subjective rather than an objective
decision. Students can skip to the self-assessment at the end of the tutorial if desired.

e-Test is being supplanted by SAMs[2]. Either of these testing modes can be used to
determine whether a student has mastered the course material. The use of the tutorials and
testing modules has allowed us to devote more time to those students who need more
individualized instruction.

Those who might adopt this approach are cautioned to set a structure for testing and
mandatory class meetings. Without a structure for testing, students may either postpone all
tests to the end of term, or attempt to take all of them early in the term. Without procedures,
testing will subtract from the time available to coach students who need additional assistance.
All exams were closely proctored to assure that adjacent students did not benefit from the
success or failure of adjacent students. Questions in each test are not randomized, so
adjacent students take the same questions in the same order. While e-Test provides four tests
on each subject, we believed it too cumbersome to open and assign different tests to each
student. We also wished to reserve tests for those who had excused absences or who
wished to take an exam early.

After students had completed the first tutorial, one of the four exams was "unlocked" to allow
students to preview testing in a simulated environment. The score on this exam was not
recorded. Separate e-Test exams were administered for Word, Excel, Access, and
PowerPoint. Students were permitted to retake one of these exams under certain restrictions.
One restriction was that the last score, whether higher or lower, would be recorded.

Students were required to check their e-mail twice a week, however, this was an inefficient
and inconsistent method for communicating information to all students. Following this, part of
the scheduled testing day had to be reserved for general announcements. Next term,
mandatory attendance days will be included in the syllabus and e-mail usage will be included
in the grading scheme.

Internet-Based Instruction

NETg may represent one end-point in instructional use of technology: vendor supplied and
developed Internet-delivered courseware, e-Course is another end point since it is also
vendor developed and delivered courseware. The primary difference is the delivery method;
e-Course is directed toward campus-based instruction. In between these two alternatives are
options where technology is more of an assistant rather than the vehicle. This section
discusses some of those alternatives used by the authors.

CyberClass

CyberClass is a software package developed by the HyperGraphics Corporation (
www.cyberclass.com) to support Internet-enabled delivery of course material. Internet
enabled delivery includes delivery of all course materials at one extreme (which makes it
similar to NETg) or it can be used to provide additional course materials for more traditional,
instructor-led courses.

To create an online presence with CyberClass, the instructor logs onto the site (guest
registrations are available at the site) and creates the course. The course, which resides on
the HyperGraphics' server, can include bulletin boards, syllabus, assignments (readings, turn
in, online), links and testing. The template is easy to use and content can easily be modified.
Our first experience required about one hour to create a working Web site. Students (and the
instructor) need a password to access the created course site.

HTML Coding

Other course sites have been created using HTML coding and code generators such as
Adobe Pagemill and FrontPage. CyberClass provides an easy-to-use template for creating
common course elements and no coding experience is required. However, the structure and
formatting is less elegant than could be created through HTML or other page generators. In
many ways, CyberClass is similar to WebCT (Web Course Tools; University of British
Columbia; http://homebrew1.cs.ubc.ca/webct/webct.html). WebCT provides a template that
course developers modify to reflect course content and instructor style.

CyberClass has a number of publishing partners. Depending on the publisher and text, it is
possible to include vendor-developed tests on the Web site. Student reactions to the site
created by one of the authors received mixed reviews. Much of the initial criticism centered on
a desire to have a hard copy of the material listed on the site. Students clearly did not want to
continuously log on to review calendars and assignments. Part of the criticism was satisfied in
later classes by more carefully reviewing the appropriateness of material included on the site.
A calendar and grading information were distributed in hardcopy format. This seemed to
satisfy the need for transportable information and reduced the amount of printing from the site.

One of the options with CyberClass is online submission of class work. While this provides a
clear time-date stamp on all work, grading e-mail assignments was difficult and inconvenient.
No matter how good the monitor and e-mail editor, it still seems more efficient to work with a
hard copy that can be easily marked and returned. In addition, students seemed to expect that
homework would be graded instantaneously when submitted via e-mail. Students often called
and e-mailed asking if the assignments had been graded. While some inquiries are to be
expected in all classes, it appears that electronic submission engenders a higher inquiry rate.

For those unfamiliar with HTML coding, code generators or WebCT, CyberClass is an
excellent vehicle for supporting a class. For those who do not have space on a server, it may
be the only alternative. CyberClass combines ease of use and functionality in an
easy-to-learn package. Those wishing more robust or complete Internet classes should
investigate WebCT or similar packages.

Summary

All of the methodologies presented and evaluated in this paper have advantages and
disadvantages. None of the packages met all instructional needs. The same comment could
also be made for the use of chalk or transparencies. All of the products and methods
discussed do provide alternative instructional formats that meet specific pedagogical needs.
All involve tradeoffs. While e-Course has great tutorials but the tutorials cannot be
modified based on assessment. NETg has great pre-assessment, but ease of use and
modality are concerns.

The key to successful selection, adoption, and use of any of the tools discussed begins with a
clear assessment of institutional needs, student capabilities, resource requirements and
instructional objectives. The second key is continual monitoring of student use and success
rates. The third key is to maintain contact with both vendors and your local computer center to
assure that support is adequate and that new releases and patches are made available on a
timely basis.

[1] Course Technology states in the book, "If you take this entire e-Course, you will be
prepared to take exams for Word Proficient, Excel Proficient, Access Expert, and PowerPoint
Expert and become certified as a Microsoft Office User Specialist."

[2] Course Technology's e-test is available on the instructor resource disk provided by the
company. It can be installed on a single machine or implemented via network installation. We
selected the network installation. An instructor logs on to the network and unlocks one of four
exams available with each module. After a short time period, the instructor locks the exam.
Students are told to set the disk and print options. The print option prints the results and the
disk option saves the results to a floppy disk. The disk not only provides additional backup in
case of printer failure, but also provides additional testing detail. There is no charge for e-test.
SAMs is an added cost and has different security features. If an institution uses SAMs, the cost
of SAMs can be part of the text. SAMs includes as security certificate and other software to
help manage a course.

Tena McQueen, M. Ed., is an Associate Professor in the Department of Computer Science at
Columbus State University. In addition to teaching the microcomputer applications classes in
the traditional lecture/lab method, she uses the Internet to deliver one course per semester.
She is particular interested in studying student responses to traditional versus Internet
instruction.

Dr. Robert A. Fleck, Jr. is Distinguished Professor of Computer Information Systems in the
Abbott Turner College of Business at Columbus State University in Columbus Georgia. He
teaches database, networking and decision support as well as the introductory skills course.
Bob's research interests include metrics, systems design, and environmental issues.

COPYRIGHT 1999 T.H.E. Journal

COPYRIGHT 1999 Gale Group

Return to the Main Page