E-learning Tools Evaluation and Roadmap Development for an Electrical Utility

This paper describes a methodology based on 40 criterions as well as a mixture of MCDM methods to evaluate different technologies applied in modern training systems. The criterions are grouped in a three-dimensional (3D) model in accordance with their use and application in training processes. The proposed 3D model includes the Management (M), the Technological (T) and the Instructional (I) dimensions. Applying this methodology we can evaluate different training and learning technologies. Furthermore a technology roadmap is outlined using the obtained results in the evaluation. This roadmap will provide CFE (Comisión Federal de Electricidad which is the National Electricity Utility in Mexico) with elements to decide, enhance and upgrade its current training information systems or acquire new ones as well as showing what courses are worth implementing with e-learning technologies and when these courses should be programmed.


Introduction
The 3D model and the proposed methodology in this paper, not only are helpful to evaluate the applicability of each learning tool from a global point of view, but they are also useful to establish the utilization of each learning tool in every dimension (M, T, I), in every plane (MT, MI, TI) and in a 3-dimensional space (MTI). This provides different views of the same tool, which allows evaluating each tool from different perspectives. These perspectives help to determine whether or not a tool fulfills the requirements from an M-T-I point of view.
The management dimension evaluates aspects such as: Course Management, Student Tracking, Curriculum Management, etc. The instructional dimension includes Instructional Design, Didactic Planning, Content Production, Instructor Manual, Student Manual, and so on. Finally, the technological dimension involves the current software and hardware tools available as candidates for acquisition and use in the training processes of a company. These 3 conceptual dimensions outline a 3D space and three planes (see Figure 1). All of these represent different relationships among them. The elements of the management-technological plane or quadrant relate how technology is being applied in course administration and how much the cost of hardware and software is. The aspects of the management-instructional quadrant relate how we can administrate the content of courses and keep track of student's progress and performance of his/her training activities. The issues in the technological-instructional plane relate how software and hardware technology is used to develop content for courses. And finally the managementtechnological-instructional space indicates how the technology is being used to manage the learning process. Commercially available tools integrate different modules providing a complete learning tool. However, this integration increases the cost and complexity of each tool. In this paper we present evaluation results for some Learning Management Systems (LMSs), Learning Content Management Systems (LCMSs) and Content Management Systems (CMSs). The CMSs are complementary tools to obtain a more complete learning tool. The proposed methodology was used to evaluate three commercial platforms (Blackboard, IBM Lotus and PeopleSoft) and two in-house developed tools (SIC, and UTEC). The SIC and UTEC tools are systems used by CFE to manage its Human Capital to align workforce with business objectives.
The rest of the paper is organized as follows: section 2 summarizes a review of evaluation methodologies; section 3 includes some e-learning concepts related to the tools evaluated in this paper; section 4 describes the evaluation methodology used; section 5 illustrates how the methodology is applied to evaluate e-learning tools, also shows the evaluation outcomes and includes an e-learning roadmap slide, which is based on the evaluation outcomes; section 6 includes some conclusions and finally section 7 provides a list of references.

•
Weighting methods: These methods assign different weights to each criterion depending on their degree of importance with respect to the other criterions, some of these methods are: point allocation, hierarchical point allocation, swing weighting, tradeoff weighting, revision of weights, nonhierarchical weight assessment and hierarchical weight assessment.

•
Deterministic ranking methods: The options are ranked based on the evaluation carried out using previously obtained values, some of these methods are: additive linear value function, multiplicative value function, non linear value function, goal programming and ELECTRE. • Uncertainty ranking methods: The evaluation takes into account simulated values to rank the choices and make a decision, some of these methods are: linear utility function, non linear utility function, regret and stochastic dominance.
In this work we used the combination of three methods: hierarchical weight assessment, additive linear value function and goal programming in order to compare the results obtained with the application of different methods and based on our results compare the weighting methods and amalgamation rules in terms of their ease of use, appropriateness and validity.
The methodology used in this work is based on some concepts from the methodology developed in [17], which we applied in the evaluation of software tools to develop virtual reality systems. In this work we are evolving this methodology adding the evaluation of the alternatives using other two MCDM methods and finally we are using a proposed 3D model and other criterions to evaluate the software tools related with learning and content management systems.
It is worth pointing out here that in some degree the MCDM are general purpose methods, in the sense that depending on the kind of items to be evaluated, they demand the definition of a set of criteria (or parameters) that an item in turn must accomplish with. The set of criteria in turn personalises the methodology. However the sets of steps involved in a methodology might remain unaltered. That is, the criteria are different but the methodology is the same. Thus, using a MCDM methodology, we can evaluate different kinds of items such as LMSs or VR tools, different types of hardware, etc., as long as we establish an appropriate set of criteria in each case.

E-learning
E-learning consists of providing educational programs and learning systems by means of electronic devices, such as: computers, CDs, internet or intranet, multimedia among others technologies. The electronic education or elearning involves a group of applications and tools, such as: web based learning, computer based training, virtual classrooms and digital collaboration (group work). These tools have many advantages: higher productivity, simultaneous new product processes and delivery of applications to many participants. Participants can make their own learning path and perhaps the best e-learning contribution, which originates that the total cost per participant is lower than the expenses in a traditional instructor guided system. Nevertheless, design and development of elearning programs can initially be higher. Among e-learning hardware and software tools we can mention the following tools: web servers, CMS, LMS, LCMS, collaboration tools and video services [14], [15].
The following sections describe some of these e-learning tools, mainly those related to learning and content management systems.

Content Management Systems.
A Content Management System (CMS) is a combination of a large database, file systems and other related software modules which are used to store and later retrieve huge amounts of data. These systems are different from the databases in the sense that these can index and use text, audio clips, video clips, or images. Users of the content management system can find relevant content within a database searching by keywords, authors, date of creation, etc. Content Management Systems can also be used to create information portals which serve as the backbone of data management. They are usually based on a pre-written template that acts as a platform for each page in the site while those pages are being created [7].

Learning Management
Systems. An LMS is a high-level, strategic solution for planning, delivering, and managing all learning events within an organization, including virtual classrooms and instructor-guided courses. The primary solution consists of replacing isolated and fragmented learning programs with systematic means of assessing and raising competency and performance levels throughout the organization. Figure 2 shows the schematic operation of an LMS inside of a company.
For example, an LMS simplifies certification efforts, enables companies to align learning initiatives with strategic goals, and provides viable means of enterprise-level skills management. The focus of an LMS is to manage learners, keeping track of their progress and performance across all types of training activities. It performs heavy-duty 77 administrative tasks, such as reporting to human resources department and other systems but it is not generally used to create course content [8], [13], [14]. Learning Content Management Systems. An LCMS is a multi-user environment where learning developers can create, store, reuse, manage, and deliver digital learning content from a central object repository, see Figure 3. LCMSs motivate personalized learning for students and help corporations in reducing the distance between tool development and the LMSs. If corporations and educative institutions wish to take advantage of this technology, they will first have to assess their needs and incorporate e-learning solutions within a corporative strategy [3], [13], [14].

Evaluation Methodology
In the next four subsections the complete proposed methodology is described in several steps from criterions selection (a) to show the results obtained and conclusions (g).

Three-dimensional model
The model proposed in this work is used to analyze modern training systems. This model relates the three more important aspects involved in personnel training: Management, Technological and Instructional. Different analysis, evaluations or studies can be made with the 3D model proposed; therefore the evaluation of a specific technological capacity (a point in the technological dimension) will be a point in a 3D coordinate system. Every point in the model (see Figure1) represents the use, application or evaluation of the technological capacity with respect to the other two dimensions: instructional and administrative.
The use of the 3D model is a common practice in CFE training activities and it was a requirement to align the criteria to this model that CFE uses to rule training.

Criterions and weight definition
The methodology is based on 40 criterions used to evaluate different technologies applied in modern training systems. This evaluation methodology, personalized with an appropriate set of criteria, has been applied earlier in the evaluation of software and hardware tools, which are related with development of virtual reality systems [16], [17]. The criterions are grouped in the 3D model described above in accordance with their use and application in the training processes.
a) Criterions selection. In this step a group of criterions are defined. Each criterion evaluates the degree of accomplishment of a user's requirement for his/her training process. These criterions might even be useful as a guide to develop learning and content management systems.
b) Value assignment. This is achieved by assigning to every criterion a scale from 0 to 5 to identify different level of accomplishment of the feature associated with that criterion.
c) Weight assignment. Different weights using non-hierarchical and hierarchical weight assessment are assigned to each criterion such that the most important criterions for the final users are emphasized. The values assigned for these weights are in a scale between 1 and 2. Most important criterions are assigned 2 while 1 is assigned to less important ones. The purpose of the hierarchies is to make weighting easier; in theory, how the hierarchy is structured should no affect the final weight assigned for each criterion [11] and in experiments carried out in [6] and [20] non-hierarchical weights tend to be "flatter" (more equal), while hierarchical weights are "steeper" (have a greater variance). d) Tools selection. A selection of software tools must be carried out based on the relevance of the company or institution which has developed and supported the software package. Additionally the relevance of the software tool was considered in accordance with the number of users and recommendations found in most LMS and LCMS references.

Evaluation methods definition
The use of three amalgamation methods using the combination of MCDMs is to compare the weighting methods and value functions in terms of their ease of use, appropriateness and validity [2], [11]. e) Analysis and evaluation of each software tool. Based on the criterions, the weight assignment and three combination of methods to evaluate different options, the software tools were reviewed, analyzed and evaluated, grading them in accordance with each method applied: In this second method these values were scaled from 0 to 1 using the following expression: In this second method the hierarchical weight assessment was used The MAX in (1) and (2) indicates that higher values are better

MCDM 3. Goal programming and hierarchical weight assessment
Goal programming focuses on achievement of goals, as oppose to additive value functions, which emphasize trading off criteria. The results obtained at the end of the paper shown the same ranking of choices it does not matter the method used as opposed in [11] where: Also in this third method these values were scaled from 0 to 1 = i w Also in this third method the hierarchical weight assessment was used

Criterions definition and value assignment
In Figure 4, all defined criterions are shown. These criterions were taken from [5] and these were grouped for each dimension (the criterions that involve only one dimension), in each plane (the criterions that involve two dimensions) or in the space (the criterions that involve three dimensions). All the criterions were analyzed, modified and grouped in collaboration with the people responsible of the training processes at CFE.  Although subjective, it is worth clarifying that this grouping is based on experience and needs of CFE's decision maker and could be grouped in different ways, for instance some criterions can be included in a plane instead of in a dimension. For instance "student tracking" was classified as M (because student tracking usually is a management activity) but it could have been classified in the MT plane (because this kind of management is achieved using LMS technology), depending on evaluator interest on different aspects that a criterion might involve.
Nevertheless, the methodology's accuracy remains unaltered since each criterion is evaluated individually and its value is the same no matter where it is grouped.
In this paper, due to lack of space only one criterion is described in detail, as well as its value assignment. The rest of the criterions were analyzed in the same way and are described in detail in [18].

Student Tracking (Management dimension).
Student tracking is the system ability to track the usage of course materials by students, and to perform additional analysis and reporting, i.e. the usage by each student and by the whole group. Student Tracking tools include statistical analysis of student performance data and progress reports for individual students in the course. Progress reports generally consist of both activities and the time stamps when an activity occurred [5]. The tool has one of the features above 2 The tool has two of the features above 3 The tool has three of the features above 4 The tool has four of the features above 5 The tool has at least five of the features above

Results
The following subsections show the results obtained for the commercial systems: Blackboard, IBM Lotus and PeopleSoft and the CFE´s systems SIC and UTEC. In the first subsection the results obtained for each dimension and plane are shown in order to realize the accomplishment of each system from different perceptions: Management, Technological and Instructional perspectives. The second subsection shows the results obtained for each system applying the three amalgamation methods described above, and the third subsection shows how we applied the results to obtain a roadmap, which will help CFE to make decisions not only to choose the best LMS tools for training processes, but also to decide which are the hardware and software tools that CFE should use for course development and management.

Results obtained for e-learning systems from different perspectives
The evaluation was carried out by assessing the degree of fulfillment of features of each criterion. Each e-learning tool was tested and evaluated developing a training course prototype for valves maintenance [1] and in collaboration with software providers each feature of each criterion was reviewed. Due to lack of space, only the results for the technological-instructional plane are presented in a graphical way (see Figure 5) and the rest of groups are presented in a tabular form in Table 2.
It is very important to say that not all systems evaluated belong to the same type; however the methodology was applied under customer requirement in order to know how each system fulfills content and learning management in training processes.

Management-Instructional Plane
Blackboard 4 out of 5

Management-Technological-Instructional Space
Blackboard 62 out of 80

Results obtained applying three amalgamation MCDM methods
The results for the first MCDM method are depicted in Figure 6, which shows the ranking and global results for each software tool. These global results include all the criterions considered applying the additive value function without scaling the value function ( ) ij i x v and using non-hierarchical weight assessment.
It is important to note that PeopleSoft and IBM Lotus are considered to be LMSs while Blackboard is a tool only to manage content (CMS). SIC is a computational system to register and maintain information about employee training in CFE and UTEC is a web based learning system to teach personnel about different subjects related to activities in CFE. These last two systems were not designed to manage content and learning systems however, they have some LMS's features. Ease of Use and Appropriateness. Based on [11] the rating weights are easier to understand and use than tradeoff weighting, and that rating does a better job of helping users to understand the problem; therefore we used rating weights in this work, with rating weights we confirm the ease of use of this method because we did not have to do additional calculations to found the tradeoffs between criterions' weights. In the experiments and questionnaires made in [11] the majority of the users felt that goal programming method makes sense and is logically sound; improves consistency of judgment; and aids understanding of the problem in comparison with applying other MCDM methods or holistic assessment (direct evaluation based on users' expertise). The questionnaires also asked whether goal programming, additive value functions, or holistic judgment was preferred. The users said to prefer using goal programming because the goal setting exercise give additional insight. With the use of amalgamation methods using goal programming we corroborate in this paper that goal programming improves consistency of judgment and helps to better understand the problem, because we have to know a priori if we want a modest or ambitious goal.

Convergent Validity: Do Different Methods Yield the Same Options Ranks?
Hobbs and Meier in [11] answer two questions related to convergent validity. The first question is, how much do people agree on the ranks of the options? Their answer is: when a MCDM method is used, there is some but not complete agreement among the participants as to which options are best, and the use of goals results in greater disagreement. Further, MCDM methods result in more agreement among participants that does holistic judgment because MCDM methods do not have prejudices and consider all criteria at the same time, therefore such methods should yield greater convergence of opinion, In this paper the evaluation was carried out by several people applying the same methods, the same criterions and same weights in order to avoid disagreement in the results.
The second question was: do different methods yield different option ranks? The answer was that differences in option ranks between pairs of methods appear less important than the interpersonal differences and the choice of an MCDM method affect ranks less than who chooses the method, it agrees with [10] but disagrees with [9]. The results obtained in this work agrees with [11] because the results show high correlation between them, in fact the ranks obtained in the three amalgamation methods are the same it does not matter which weighting method and which deterministic method we applied (see figures 6, 7 and 8)

Roadmap
The core of our contribution is in the detection and classification of a set of concepts which are used to evaluate elearning systems and to identify the capabilities, properties or characteristics that structurally describe types of requirements and define the key elements to be included in an e-learning environment. These elements or criteria and the methodology are described in detail in [17], which are the basis to build a technological roadmap that is the guide for the CFE´s management personnel to make decisions not only about which are the best LMS tools for its training processes, but also which are the hardware and software tools that CFE should use for courses development and their management.
There are many proposals for modern training systems requirements; each one has its own elements. Some use the same concepts but the names are different, which makes it complex and laborious to compare. The approach presented here unifies the various terminologies, increases the knowledge about e-learning systems which helps to select the tool that is more appropriate to CFE's needs either if the tool is going to be in home developed or acquired from a provider. The roadmap proposed suggests to CFE what projects should develop in the next six years, based on the driving factors, the internal needs detected in CFE, and in the available technologies in the next years (see Figure 9). There are many aspects which an LMS might influence, for instance there is a cultural impact regarding training management, a computer is going to participate in teaching and management, availability of courses and tutors, etc. This is why CFE needs not only to buy and use an LMS, but it also has to work in all these aspects. For the time being and because CFE already had an LMS, which by the way, does not have the maximum score in the evaluation, it was recommended to use the one they already have to start the work. However and due to the cost of the LMS on one hand and the portals development experience of CFE on the other, it was also suggested to start an in house LMS development.

Conclusions and future work
In the application of MCDM methods to make a decision based on the results, we conclude as in [11] that it is recommendable to apply more than one approach because different methods offer different results to compare, in this case, goal programming and additive value functions are suggested and besides the results must be shown to decision makers who can mull over the differences or confirm the resemblances. In evaluating the results of different methods, the potential for biases should be kept in mind. The extra effort is not large; the potential benefits, in terms of enhanced confidence and a more reliable evaluation process, are worth.   The proposed model is being applied at CFE, a 70,000 employee company that generates, transmits, and distributes electricity through out the nation. One of its strategic objectives is to establish training programs to certify its employees in labor competences aligned with its mission and future vision.
Although the model can be used to analyze a broad variety of different e-learning technologies, the paper only address asynchronous web-based environments where learning content or courseware is served from a web server and delivered on demand to the learner's workstation. Learners can thus make progress by themselves. The courseware may be comprised of any combination of text, images, animation, sounds and movies. The courseware is interactive and is often combined with some type of assessment.
One of the main benefits for CFE after this evaluation is that personnel in charge of CFE's training management are now well informed about this technology. Furthermore, they have now a roadmap for modern training systems technology. Thus, they can make up an action plan and choose the best paths to follow, in order to integrate this technology into CFE's processes. For instance, the roadmap recommends beginning immediately the use of the tool already available in CFE (Lotus's LMS), which is not the system with the best score in the evaluation, but it is still acceptable. This would avoid spending more resources on new licenses. It is also recommended to develop simultaneously their own LMS which suits CFE's needs. This will avoid spending resources on annual license fees and they will have the source code of their own LMS. This will enable CFE to make changes and improvements under demand. This might be achieved in a period of 3 to 5 years.