Atenea Exams, an IMS LTI Application to Solve Scalability Problems: A Study Case

Featured


Introduction
Atenea is the Virtual Campus of the Universitat Politècnica de Catalunya (UPC) based on the free software Moodle. The Moodle-based version of Atenea has been in operation since 2004, when it replaced the previous Lotus Notes-based environment [1,2]. The current Atenea is fully integrated into the teaching, enrollment and academic management processes of the university, being the service most valued by the university's teaching and research staff in internal surveys, with an average rating of 3.9 out of 5 obtained in the internal survey among the UPC's faculty answered by 1545 participants with more than a 50% response rate.
The project to change the UPC virtual campus to Moodle was proposed and executed by the Institute of Education Sciences of the UPC (ICE) and UPCnet-the ICT service company that provides services and is owned by the university itself. The change meant making an unprecedented decision until 2004: to move a critical service for the university to Open Source technology with General Public License (GPL), without any guarantee or any external company support that was responsible for the system, its maintenance and its evolution.
This decision implied that UPC itself with its own resources had to develop the expert knowledge and ability to integrate Moodle into its processes and systems, leverage the platform, organize its processes around and support its users. Time has shown that the decision was right. As a result, UPCNet, the consulting and technology services company created by the UPC twenty years ago, opened a new line of business doing consulting and services related to Moodle; in 2019 UPCNet under the new brand IThinkUPC (https: //www.ithinkupc.com/) became a Moodle Partner.
At that time, the decision to adopt Moodle as a technology to implement Atenea, was strongly influenced by the report on the selection process of the Learning Management System (here in after LMS) based in Open Source technologies for the Universitat Jaume I, carried out Center for Education and New Technologies (CENT) which was visited by an UPC delegation [3][4][5]. Other factors in favor of the decision include:

•
Moodle is a platform based 100% on web technologies that in 2004 had become a standard.

•
Moodle design is focused on the needs of teachers, is designed by teachers and developed by a teacher learning community and based on pedagogical principles (constructivism and social constructionism) [6]. • Several UPC professors were already using Moodle for their teaching, with installations of Moodle hosted in PCs in their offices set up and operated by the professors themselves.

•
At the School of Informatics of the UPC, several final degree projects which aim was to contribute to the code base of the Moodle community, were being developed. Students doing these projects became internal talent of UPC and UPCNet, experts in the environment and active participants in the Moodle community.

•
Another important influence is the cultural change within the UPC with respect to the free software movement. UPC's Board of Management launched the Chair of Free Software to serve as a vehicle for all the initiatives undertaken by the University in this area. The Chair of Free Software, which was created in September 2004, aimed to raise awareness of Open Source software at UPC and in society at large [7].
During the first four months of the 2005-2006 academic year, a pilot test was carried out with 9 schools, involving 132 subjects, 216 groups and 2908 students. In January 2006, once the testing phase was over, the UPC approved the full implementation of Moodle as the technological platform for the Atenea virtual campus for the next academic year. The adoption of Moodle as a teaching support tool at Atenea was a success, with very few problems considering the level of adoption and size of the university.
At the second MoodleMoot España conference organized in the Canary Islands in 2005, the case of the adoption of Moodle by the UPC was presented. The case of Atenea as an instance of Moodle for more than 30,000 users was a success story that validated the viability of Moodle at the technological level, as a teaching and organizational tool, and contributed to other Spanish universities and education departments adopted Moodle as an LMS to support their university, primary and secondary education.
The UPC has been active in its collaboration within the Moodle community, doing research, providing services, proposing improvements and adding developments to the codebase. Until 2020 the Atenea platform has been operating without problems serving the university community.
In 2010, UPCnet started changing its internal server architectures in its Data Processing Center (DPC). After an initial phase of virtualization of the physical servers, the DPC started to operate a private cloud. A private cloud is a model of hosting where cloud computing services and IT infrastructure are hosted privately and dedicated to a single organization with isolated access. It can be hosted on-premise or by a third-party cloud hosting provider. With this change in architecture UPCNet gained the flexibility of the cloud computing model-being able to create, start, stop and pause virtual computing assets for the projects and the operations, and having the option of using seamlessly assets in public clouds (like Amazon Web Services-if the necessity arises).
Operating a private cloud on its DPC allows UPCNet to ensure GDPR compliance and other EU regulations in internal UPC projects like the Atenea Platform, and external projects of UPCNet's customers. For example, the UPCNet private cloud runs the Agora platform which serves independent Moodle instances for all the government primary and secondary education centers and some of the private owned ones-more than 2500 Moodle instances.
On March 14, 2020, the confinement of the population throughout Spain was decreed due to the emergency caused by the COVID-19 pandemic. For the UPC, as for the rest of the state's universities, this means moving all teaching to online format overnight. The daily creation of new resources and activities immediately doubled as did most activity indicators.
There was also a significant increase in workload. Internal server resources saw an 800% increase in platform stress. UPCNet technicians needed to address the workload issue by identifying bottlenecks and increasing resources dedicated to virtual machines. Some of these actions needed to be done urgently in a few weeks.
Once it was clear that the confinement would last until the end of the course, Atenea officials saw that the assessments should be done online in all faculties of the UPC. There are many ways to do online assessment, ranging from the delivery of exercises, the continuous assessment of activities and practices, presentations and oral exams by video conference, etc. One of the forms of assessment is the use of test-type exams, implemented in Moodle with the Quiz module.
The Moodle Quiz module allows you to define questionnaires in a very agile way. It allows you to work with various types of questions, create question banks organized by categories, create personalized exams for each student with random questions from established categories, randomize the order of answers, take self-assessment tests, make more than one attempt, and much more. This is a very complex module, with more than 100,000 lines of source code and making intensive use of the database engine.
At that time, the main question was whether Atenea would be able to bear the overload once large or very large courses (≥200 students) started doing questionnaires simultaneously. In previous simulations the UPCNet technicians discovered performance issues while doing some of the tests based on the Quiz module, and experienced some glitches during partial evaluations. So the main question UPC was forced to address was how to solve these performance issues when turning to online learning and assessment.
The main aim of this paper is to explain how UPC solved the presented problem using the Atenea based-Moodle virtual campus.
In the case of Atenea, we found out that the fact that the UPC has cultivated its expertise, knowledge a collaborative relationship with the Moodle.org community over the prevlious 15 years, has been a key aspect to solve the problem of massive online evaluation using the Quiz module. The UPC has been able to respond to the challenges of the COVID-19 crisis in part because it has this internal talent. By outsourcing critical services to SaaS models, certain complexities cannot be addressed and innovated upon.

Introduction to Moodle's Interoperability
Learning and teaching is one of those application areas where there are available different learning platforms, repositories, tools, types of content, etc. These systems should interoperate among them to provide better and more useful learning services to students and teachers, and to do so web services and interoperability specifications are needed.
In 2008 most LMS were not service oriented, so any attempt to integrate external applications with the LMS should be done ad hoc. This approach has important disadvantages such as the difficulty in maintaining and extending the new integrated system or the limited impact of these solutions [8].
Moodle is not an exception, in 2008 an extension of the Moodle design that allows to offer web services, in various protocols and forms of authentication was proposed. This extension of webservices facilitated the tasks of integration with academic management systems, and allowed the development of mobile apps that act as a client of Moodle, as shown in Moodbile [9][10][11][12].
In 2010, the version 2.0 of Moodle was published, it incorporated the Webservices extension described in Figure 1. The webservices extension provides services like authentication, authorization, enrollments, access to rosters, etc. Gradually the number of APIs in this webservices architecture allowed enough control over Moodle as to develop a fully operational Mobile client.
Interoperability specifications define ways to exchange information and/or interaction between systems, they can be focused on the exchange of a specific type of information or address interoperability in a global way [13,14]. According to Wilson one possible scenario of interoperability is based on the integration of external tools into the LMS [15]. Some examples of experiences that integrate tools with LMS and learning platforms can be found in [16][17][18][19][20][21][22].

IMS Learning Tools Interoperability
In 2008 the IMS Global Learning Consortium was working on a draft for a specification of its Learning Tools Interoperability (LTI) standard [23]. The purpose of this standard was to enable the execution of educational applications as a service as an activity within an LMS course. At that time, the standard was only a draft, it was considered to be way too complex and there was no reference implementation.
A simplified specification called SimpleLTI was proposed, a kind of minimally viable LTI product [24]. A reference implementation of SimpleLTI consumer was developed for Moodle by researchers of UPC, under the direction of Severance and Alier. SimpleLTI evolved into a more complete specification called BasicLTI, which implemented the webservices communications between the Provider and the Consumer. Gradually all major LMS vendors (Moodle, Sakai, Blackboard, Canvas, Desire2Learn) started to support BasicLTI.
In 2012 when IMS finally published the IMS LTI 1.0 specification 2012 Moodle incorporated in its official distribution a LTI consumer extension under the name "External Tool". Figure 2 shows the "External Tool" (in its Catalan translation "Eina externa") in the palette of available learning activities in Atenea.
Years later, a Moodle extension (LTI Producer) that allows a Moodle course or activity to act as an LTI service provider was added to the main distribution.

The LTI Protocol
IMS LTI enables the model of SaaS for learning activities within the context of the LMS. LTI enables to plug an external application into the LMS within a determinate context.
In the LTI protocol there are two relevant actors: the LTI Provider and the LMS acting as a Consumer. A service in the form of a Web application is provided (the Learning Activity or Learning Tool) to the LMS users so they can access this tool once some setup is done.
In an LMS, a learning tool is contextualized in a virtual classroom that may be a Moodle Course, the group or subgroup of students and the activity itself. For instance, a Discussion Forum in the Main Group of the Introduction to Cloud Computing Course. These three elements: course, group and activity define a specific activity and its context. When a user accesses the activity, he does so with an identification (Id) and a set of specific permissions and privileges granted by his Role (teacher, student, visitor, etc.), and this is managed by the LMS with internal services of authentication and authorization. In addition to that, the activity reports its outcomes to the LMS: logging information, learning outcomes and grades.
When a system acts as an LTI Provider, offering an online learning activity as a service via the web, it will receive the activity context (course, group, activity id), and the user's identification, authentication and role from the LTI Consumer (in our case the LMS) during the launch of the activity and via webservices communications between Provider and Consumer.
In 2010, the version 2.0 of Moodle was published, it incorporated the Webservices extension described in Figure 1. The webservices extension provides services like authentication, authorization, enrollments, access to rosters, etc. Gradually the number of APIs in this webservices architecture allowed enough control over Moodle as to develop a fully operational Mobile client. Interoperability specifications define ways to exchange information and/or interaction between systems, they can be focused on the exchange of a specific type of information or address interoperability in a global way [13,14]. According to Wilson one possible scenario of interoperability is based on the integration of external tools into the LMS [15]. Some examples of experiences that integrate tools with LMS and learning platforms can be found in [16][17][18][19][20][21][22].

IMS Learning Tools Interoperability
In 2008 the IMS Global Learning Consortium was working on a draft for a specification of its Learning Tools Interoperability (LTI) standard [23]. The purpose of this standard was to enable the execution of educational applications as a service as an activity within an LMS course. At that time, the standard was only a draft, it was considered to be way too complex and there was no reference implementation.
A simplified specification called SimpleLTI was proposed, a kind of minimally viable LTI product [24]. A reference implementation of SimpleLTI consumer was developed for Moodle by researchers of UPC, under the direction of Severance and Alier. SimpleLTI evolved into a more complete specification called BasicLTI, which implemented the webservices communications between the Provider and the Consumer. Gradually all major LMS vendors (Moodle, Sakai, Blackboard, Canvas, Desire2Learn) started to support Basi-cLTI.
In 2012 when IMS finally published the IMS LTI 1.0 specification 2012 Moodle incorporated in its official distribution a LTI consumer extension under the name "External Tool". Figure 2 shows the "External Tool" (in its Catalan translation "Eina externa") in the palette of available learning activities in Atenea. Years later, a Moodle extension (LTI Producer) that allows a Moodle course or activity to act as an LTI service provider was added to the main distribution.

The LTI Protocol
IMS LTI enables the model of SaaS for learning activities within the context of the LMS. LTI enables to plug an external application into the LMS within a determinate context.
In the LTI protocol there are two relevant actors: the LTI Provider and the LMS acting as a Consumer. A service in the form of a Web application is provided (the Learning Activity or Learning Tool) to the LMS users so they can access this tool once some setup is done.
In an LMS, a learning tool is contextualized in a virtual classroom that may be a Moodle Course, the group or subgroup of students and the activity itself. For instance, a Discussion Forum in the Main Group of the Introduction to Cloud Computing Course. These three elements: course, group and activity define a specific activity and its context. When a user accesses the activity, he does so with an identification (Id) and a set of specific permissions and privileges granted by his Role (teacher, student, visitor, etc.), and this is managed by the LMS with internal services of authentication and authorization. In addi- Let us suppose that we have a web app that implements a Code Dojo where students can do coding exercises and some kind of group interaction. The Code Dojo can act as LTI Provider, so it is set up to get the user's Id, context, and role from the LTI Launch when a student or teacher clicks on the LTI Activity (External Tool) in the LMS as illustrated in Figure 3. The Code Dojo can report back to the LMS information about the student's logs, learning outcomes and grades via webservices defined by the LTI Protocol.
Before the students can access the LTI learning activity a configuration of LTI activity in the LMS is required. An LMS activity-in a specific course and group-connects to a specific LTI service, this is set up with the External Tool in Figure 3, where the teacher can enter the proper configuration parameters (URL and Secret). Once the LMS LTI Consumer is set up, when the LMS users access this activity, they will be redirected to the LTI provider's website with the proper context, identification and authorizations to enjoy it.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 of 17 is set up, when the LMS users access this activity, they will be redirected to the LTI provider's website with the proper context, identification and authorizations to enjoy it. As shown in Figure 3, when the LMS user accesses an LTI Activity, in the LTI Launch the LMS communicates to the LTI Provider the execution context of the activity, an identifier for the user-which may or may not include the user's personal data depending on how it is configured the system-and the permissions of this user in the context of the activity (teacher, student, etc.). The LTI provider can send student activity records and assessment data back to the LMS.
There are some advantages of using an external LTI tool compared to modular Moodle extensions:

•
It does not pose a security or integrity risk to a Moodle instance. If LTI Activity drops, it does not affect the operation of the LMS. There is simply one activity that stops working. A security or performance issue in a module can compromise the operation of the entire LMS.

•
There is no need to update LTI activities when the LMS is updated. While the activity complies with the LTI protocol, it acts independently. • LTI Activities allow new forms of collaboration, a company can offer an LTI Activity as a service to customers around the world and regardless of the LMS they use (Moodle, Canvas, Sakai, Blackboard, etc.), as most LMSs currently support the LTI standard. However it poses a risk to the privacy and confidentiality of students. • Because Moodle can act as an LTI Provider, unofficial module functionalities can be incorporated into a large instance such as Atenea. You simply need to install them on a separate Moodle server and offer them as LTI activities.
In fact, you can also offer features offered by default in the Moodle distribution to solve scalability and performance issues, as we will see in the next section.

Solution and Results
The confinement of the population throughout Spain caused by the COVID-19 pandemic forced the UPC, and the rest of the state's universities, to move all teaching to online format overnight. The daily creation of new resources and activities in Atenea immediately doubled as did most activity indicators. There is also a significant increase in workload (see Figures 4 and 5). As shown in Figure 3, when the LMS user accesses an LTI Activity, in the LTI Launch the LMS communicates to the LTI Provider the execution context of the activity, an identifier for the user-which may or may not include the user's personal data depending on how it is configured the system-and the permissions of this user in the context of the activity (teacher, student, etc.). The LTI provider can send student activity records and assessment data back to the LMS.
There are some advantages of using an external LTI tool compared to modular Moodle extensions: • It does not pose a security or integrity risk to a Moodle instance. If LTI Activity drops, it does not affect the operation of the LMS. There is simply one activity that stops working. A security or performance issue in a module can compromise the operation of the entire LMS.

•
There is no need to update LTI activities when the LMS is updated. While the activity complies with the LTI protocol, it acts independently. • LTI Activities allow new forms of collaboration, a company can offer an LTI Activity as a service to customers around the world and regardless of the LMS they use (Moodle, Canvas, Sakai, Blackboard, etc.), as most LMSs currently support the LTI standard. However it poses a risk to the privacy and confidentiality of students. • Because Moodle can act as an LTI Provider, unofficial module functionalities can be incorporated into a large instance such as Atenea. You simply need to install them on a separate Moodle server and offer them as LTI activities.
In fact, you can also offer features offered by default in the Moodle distribution to solve scalability and performance issues, as we will see in the next section.

Solution and Results
The confinement of the population throughout Spain caused by the COVID-19 pandemic forced the UPC, and the rest of the state's universities, to move all teaching to online format overnight. The daily creation of new resources and activities in Atenea immediately doubled as did most activity indicators. There is also a significant increase in workload (see Figures 4 and 5). Appl. Sci. 2020, 10, x FOR PEER REVIEW 7 of 17  All of the indicators analyzed increased following the same patterns, and these increases compounded its impact on the demand of computing resources in the platform which increased more than 8-fold.

Training the Faculty
Given this situation, the Institute of Education Sciences (ICE) of the UPC was a key player, as the body responsible for faculty training. During this crisis period, the ICE provided a very fast and proactive response, working in forced marches, becoming a benchmark of expertise to make the abrupt transition to online learning. A number of specific actions were taken: • The creation of specific courses for the transition to online teaching. Some of which were designed, prepared and offered to faculty urgently in a few days.    All of the indicators analyzed increased following the same patterns, and these increases compounded its impact on the demand of computing resources in the platform which increased more than 8-fold.

Training the Faculty
Given this situation, the Institute of Education Sciences (ICE) of the UPC was a key player, as the body responsible for faculty training. During this crisis period, the ICE provided a very fast and proactive response, working in forced marches, becoming a benchmark of expertise to make the abrupt transition to online learning. A number of specific actions were taken: • The creation of specific courses for the transition to online teaching. Some of which were designed, prepared and offered to faculty urgently in a few days.  All of the indicators analyzed increased following the same patterns, and these increases compounded its impact on the demand of computing resources in the platform which increased more than 8-fold.

Training the Faculty
Given this situation, the Institute of Education Sciences (ICE) of the UPC was a key player, as the body responsible for faculty training. During this crisis period, the ICE provided a very fast and proactive response, working in forced marches, becoming a benchmark of expertise to make the abrupt transition to online learning. A number of specific actions were taken:

•
The creation of specific courses for the transition to online teaching. Some of which were designed, prepared and offered to faculty urgently in a few days. With regard to teacher training, it should be mentioned that from March to July 2020, 79 specific training courses are being developed for the transition to non-face-to-face teaching, with a total of 6190 applications for registration. This offer responds to a specific total of 4559 registrations, with an average of 57, 7 attendees per course. The academic and management part obtained a quality index valued at 4.53 out of 5, according to the surveys carried out after each course and on all participants (See Table 1). Table 1. Data on faculty training activity during the confinement course 2019-2020 2ond semester. "Before Covid-19" column covers the period from 2 February 2020 to 17 February 2020, "During Covid 19" covers the period 18 February 20 to 31 July 2020. The assessment is the result of surveys presented to all participants in the ICE's courses, with more than 60% of participation. The change in the way the Atenea platform is used is not limited to the faculty. Students are starting to use the platform more and in a different way. The maximum number of concurrent sessions goes from 4000 to 6000. Surprisingly, Figure 6 shows how the levels of access on weekends and during Easter holidays are equivalent to, or higher, than those of school days before confinement.

ICE'S RESOURCES
The next academic year Q1-2020-2021 started with some courses taught in the classrooms and some courses with a hybrid approach: teaching done part-time online and part-time in the classrooms. This situation changed in November 1-when we returned to online teaching due to the second wave of the COVID-19 pandemic. However, even if not all the teaching is done online in this semester we can observe that the usage of Atenea keeps on intensifying: the maximum number of concurrent sessions in Atenea are maintained and they even increase slightly (see Figure 7). Whether this suggests a permanent change in the way teachers and students use online tools is a question that will be answered after this pandemic end.
The way the faculty uses the LMS for the learning design in their courses also changed. A comparative analysis of the activity in the LMS between the course 2018-2019 Q2 and the course 2019-2020 Q2 reveals some interesting facts. The number of tasks (assignments) in the platform doubled, and half of the courses presented more than 40 tasks while in the last year the number was down to 7. The number of questionnaires per student and course quadrupled compared to the last course.

A solution to Cope with Scalability Issues in Atenea
As stated in the previous section, the Atenea platform is suddenly subject to a very significant increase in workload. Specifically, the UPC wants to assure that Atenea will be able to bear the overload when students in several large or very large courses, start the online questionnaires in order to do online assessment. This concern is caused by performance issues observed while doing some of the tests based on the Quiz module, in partial evaluations.

(MEAN)
The change in the way the Atenea platform is used is not limited to the faculty. Students are starting to use the platform more and in a different way. The maximum number of concurrent sessions goes from 4000 to 6000. Surprisingly, Figure 6 shows how the levels of access on weekends and during Easter holidays are equivalent to, or higher, than those of school days before confinement.  The next academic year Q1-2020-2021 started with some courses taught in the classrooms and some courses with a hybrid approach: teaching done part-time online and parttime in the classrooms. This situation changed in November 1-when we returned to online teaching due to the second wave of the COVID-19 pandemic. However, even if not all the teaching is done online in this semester we can observe that the usage of Atenea keeps on intensifying: the maximum number of concurrent sessions in Atenea are maintained and they even increase slightly (see Figure 7). Whether this suggests a permanent change in the way teachers and students use online tools is a question that will be answered after this pandemic end. The way the faculty uses the LMS for the learning design in their courses also changed. A comparative analysis of the activity in the LMS between the course 2018-2019 Q2 and the course 2019-2020 Q2 reveals some interesting facts. The number of tasks (assignments) in the platform doubled, and half of the courses presented more than 40 tasks while in the last year the number was down to 7. The number of questionnaires per student and course quadrupled compared to the last course. Before we explain the solution proposed to deal with this performance tests in Atenea, we need to discuss the current systems architecture of Atenea, why it was chosen and its evolution in the last 15 years.

Atenea: System Architecture
As mentioned above, with the adoption of Moodle as a teaching support platform, the UPC does an investment in the university's internal talent. Recently, Moodle.com, the company responsible for the development and distribution of Moodle, established its second headquarters in Barcelona, and has hired several people from UPC and UPC-Net [15].
As a result of the university's involvement with Moodle, several research and development projects are being carried out which have as a relevant result for this article, the incorporation of two new interoperability technologies into the Moodle system itself.
The Moodle-based Atenea virtual campus must serve as a training support platform at the UPC for [ The Atenea platform has considerable quality and performance requirements. It was estimated that the platform should handle at least 5000 simultaneous requests, and offer high availability. It is not acceptable for the platform to fall, for example, in the middle of a questionnaire-type exam. The Atenea platform needs to be available 24/7 throughout all year. The service only stops very occasionally for maintenance, usually on weekends and with notice to the faculty so it does not interfere with academic activity, like deadlines for task deliveries. Since its migration to the private cloud architecture these services interruptions are rare. Therefore, a study had to be conducted to ascertain scalability and reliability parameters. Moodle works in a simple LAMP environment (Linux, Apache, MySQL and PHP), which can be installed and run on a low-end PC from 2004. However, if one needs to support a large number of simultaneous connections, as many as 9000, there is a major scalability problem as there are several bottlenecks to be solved. Moodle, since its version 1.4, supported PostgreSQL, a more robust and scalable database than MySQL at the time. A first step in increasing Moodle performance was to move the database server to a dedicated, optimized machine. However, the adjustment was not enough for the scalability requirements, as 9000 possible concurrent transactions far exceed what a PHP application server can support. At that time two alternatives were presented [26]: • Multi-instance Moodle: implement Atenea with several instances of Moodle, separated by the organizational context (School, degree, campus). See Figure 8.

•
One single high-performance Moodle instance: work in a single instance, with code, database and file system shared by several servers working in parallel. See Figure 9.
• It allows to assemble machines with modules of educational application developed by third parties, for teaching innovation projects. Moodle has a modular architecture and allows you to extend its functionality with activity modules, course, topics and other extensions. Some of the projects developed internally at the UPC consist of activity modules for teaching innovation projects. These modules can be securely installed on a multi-instance system. However, this option presents several drawbacks: • At that time (2004), the UPC had not yet trained enough developers to work with Moodle in such endeavor, and the project to create a multi-instance management software is considered risky.

•
The processes of integration with academic management (user registration, course registration and grades) become much more complex in a multi-instance system. • Day to day, administration of a multi instance installation can be more complex and demanding of effort.

•
The developed software must be maintained and updated to work with each new version of Moodle. It is important to update frequently to the latest version for security updates.

•
At that moment, the university management was implementing a policy of centralizing services of all kinds, and it was not seen as a positive thing to move servers to different campuses.
The second option (one single high performance Moodle instance) is based on the implementation of a single instance of Moodle to serve all schools and colleges in the university. This instance runs on 9 servers.
Two of these machines act as load balancers. These load balancers route the requests of the web client to all the servers able to satisfy these requests in a way that maximizes the speed, the utilization of the capacity of the systems and guarantees that there is no server with an excess of work, something that can degrade system performance. If a server fails, the load balancer redirects traffic to the remaining online servers. The load balancer performs the following functions: • Distributes client requests or network load efficiently across multiple servers.

•
Ensures high availability and reliability by sending requests only to servers that are online.

•
Provides the flexibility to add or subtract servers as per demand.
One of the load balancers is active, balancing web traffic. While the other is in passive mode, to replace of the other balancer immediately in the event of a service outage.
Six machines act as front-end servers, executing Atenea's PHP code according to the requests they receive from the load balancer. These six front-ends (or Application Servers) run the same source code and work on the same network-mounted file system via Networked File System (NFS). This networked, fiber-optic shared file system is mounted in a high-performance disk cabinet, with cache, backup-restore, and hot disk replacement systems.
Finally, all front-ends share the same PostgreSQL database management system, which runs on an optimized server and manages the transactions of the six servers. It is important to remember that in 2004 when we talk about servers we are still talking about physical machines, not virtualized or cloud servers.
The advantages of this configuration were, at that time: • Load testing proves that this architecture worked and fully supports performance requirements.

•
The integration of the system with the academic management system (user registration, course registration and sending notes) is much simpler with a single instance of Moodle.

•
UPCnet already has the expert technical staff to configure and manage complex server architectures.

•
The policy of the university management is favorable to a single instance that integrates the service to the whole university. The disadvantages of this option are: The first option involves greater complexity in the software. The second option involves greater complexity in the configuration of systems. In the following paragraphs we are going to discuss the advantages and disadvantages of both options and the final selected architecture UPC adopted.
The first option (called multi-instance Moodle) is based on the creation of a first web application as a point of entry and user authentication. After logging in, users find a list of links to the courses where they are enrolled. When you click on a link, the user is directed to the server hosting the specific Moodle course. Instead of operating a single instance of high-performance Moodle, it works with multiple instances that receive a usage load within their performance parameters.
This option has the following advantages: • It works with Moodle instances with workloads that servers know they can take on. • At a time when the speed of Internet connection was limited in some of the UPC Campuses (the UPC has a Campus in 7 different locations in the province of Barcelona back in 2004), the machines that serve the Moodle instances can be located in each campus to provide a good service.

•
If a Moodle instance fails due to a server or software issue, this only affects the courses served by that instance, without affecting the rest of the university. • It allows to assemble machines with modules of educational application developed by third parties, for teaching innovation projects. Moodle has a modular architecture and allows you to extend its functionality with activity modules, course, topics and other extensions. Some of the projects developed internally at the UPC consist of activity modules for teaching innovation projects. These modules can be securely installed on a multi-instance system.
However, this option presents several drawbacks: • At that time (2004), the UPC had not yet trained enough developers to work with Moodle in such endeavor, and the project to create a multi-instance management software is considered risky.

•
The processes of integration with academic management (user registration, course registration and grades) become much more complex in a multi-instance system. • Day to day, administration of a multi instance installation can be more complex and demanding of effort.

•
The developed software must be maintained and updated to work with each new version of Moodle. It is important to update frequently to the latest version for security updates.

•
At that moment, the university management was implementing a policy of centralizing services of all kinds, and it was not seen as a positive thing to move servers to different campuses.
The second option (one single high performance Moodle instance) is based on the implementation of a single instance of Moodle to serve all schools and colleges in the university. This instance runs on 9 servers.
Two of these machines act as load balancers. These load balancers route the requests of the web client to all the servers able to satisfy these requests in a way that maximizes the speed, the utilization of the capacity of the systems and guarantees that there is no server with an excess of work, something that can degrade system performance. If a server fails, the load balancer redirects traffic to the remaining online servers.
The load balancer performs the following functions: • Distributes client requests or network load efficiently across multiple servers.

•
Ensures high availability and reliability by sending requests only to servers that are online.

•
Provides the flexibility to add or subtract servers as per demand.
One of the load balancers is active, balancing web traffic. While the other is in passive mode, to replace of the other balancer immediately in the event of a service outage.
Six machines act as front-end servers, executing Atenea's PHP code according to the requests they receive from the load balancer. These six front-ends (or Application Servers) run the same source code and work on the same network-mounted file system via Networked File System (NFS). This networked, fiber-optic shared file system is mounted in a high-performance disk cabinet, with cache, backup-restore, and hot disk replacement systems.
Finally, all front-ends share the same PostgreSQL database management system, which runs on an optimized server and manages the transactions of the six servers.
It is important to remember that in 2004 when we talk about servers we are still talking about physical machines, not virtualized or cloud servers.
The advantages of this configuration were, at that time: • Load testing proves that this architecture worked and fully supports performance requirements.

•
The integration of the system with the academic management system (user registration, course registration and sending notes) is much simpler with a single instance of Moodle. • UPCnet already has the expert technical staff to configure and manage complex server architectures.

•
The policy of the university management is favorable to a single instance that integrates the service to the whole university. The architecture chosen in a single instance turns out to be a good option and together with a good execution by UPCNet, the Moodle-based Atenea platform has been providing services for 16 years without major incidents.
In the autumn term of the 2019-2020 academic year, the Atenea system supports approximately: • 60,000 daily user sessions • Peaks of 6000 simultaneous sessions (with historical peaks of 9000 concurrent sessions)

Virtualization and Private Cloud
From 2005, the Data Processing Center (CPD) of the UPC began to incorporate virtualization systems. By 2009, the entire Atenea service was moved to virtualized servers in the UPC Private Cloud, in virtualization clusters. A virtualization Cluster is a group of hosts. When a host is added to a cluster, the host's resources become part of the cluster's resources. The cluster manages the resources of all hosts within it. Clusters enable high availability and distributed resource scheduling [27,28].
This move to a virtualized system does not mean a change of architecture. The scheme of 2 load balancers, 6 front-ends, fiber optic connected disk cabinet and a server for the database engine is still maintained. However, virtualization allows certain advantages over using physical servers:

•
Better server recovery capability in case of crash. The virtualization cluster manages the recovery of virtual servers in the event of a crash on the physical machine where the virtual machine is running. • Possibility of implementing high availability strategies, forcing the presence of redundant services in different clusters. For example, not putting all front-ends or load balancers in the same cluster. • Possibility to dynamically distribute physical resources (processors, memory, etc.) to one service or another depending on the workload. In this way, load peaks can be addressed, and resources redirected to other services when the system has less load. It turns out that the workload at Atenea varies greatly depending on the calendar (enrollment periods, course, homework assignments, exams, holidays), day of the week, time, or even time of day: when there is an exam all students start the quiz at the same time.

Atenea Exams
Once we have explained the architecture of Atenea we are going back to one of the problems we faced when turning to online learning: the possible overload because of a massive use of the Quiz Moodle module.
A possible solution would have been to create a duplicate of Atenea to conduct the quizzes, this would have involved creating duplicates of all courses, with all student registrations, which presents a lot of technical problems regarding data integrity. It would also generate different entry points to courses and places to take an exam, creating a worse user experience for the students. Another problem is the synchronization of student grades. If we were to have two Ateneas: where can the student find her or his grades?
The solution chosen was to deploy a high-performance version of the Moodle Quizz Module As A Service, to plug-and-play within the university's LMS without drawing on its internal resources, solving the possible overload problem. This would be implemented by a new Moodle acting as an LTI Provider, so that the teachers create the questionnaires there and connect them with the Atenea courses where they teach their subjects as an External Activity. This LTI service would be supported by a new high-performance standalone Moodle instance that could support a hypothetical maximum load of 9000 simultaneous connections. This new instance was called Atenea Exams, making clear its purpose.
Atenea Exams was mounted following the same system architecture as Atenea: 2 load balancers, 6 front-end, a dedicated server for the database. All virtualized in virtualization clusters in the private cloud. The only difference is that the drive bay used is a new highperformance drive with all the high-speed SSDs, which has already been purchased but not deployed into production.
Atenea Exams is not set as a copy of Atenea the LMS, is just a LTI Provider that offers the Moodle Quiz Activity and the Task Activity like a SaaS. The professor creates the Quiz in his course in Atenea Exams, publishes it as a LTI Activity using the Moodle LTI Provider feature and sets it up in Atenea using the Moodle External Tool (LTI Consumer) activity module.
In Figure 10 we present an overview of the interaction between Atenea Exams and Atenea. Atenea Exams is a platform set up only for the purpose of publishing quizzes as LTI Applications, Atenea is the LMS already in production. Let us analyze how it works: (1) The professor logs in Atenea Exams, there she finds a course with the same name and code as the one she is teaching in Atenea. This course is empty. There she can create a quiz, either from scratch or importing it from a backup file from her Atenea course. (2) Using the Moodle LTI Provider plugin the professor publishes the quiz as a LTI application. (3) The professor logs onto Atenea. (4) She goes to her course and creates an External Tool activity as shown in Fig 2. and binds the activity to the quiz she has created in step 2. (5) The students log in Atenea, access their course and click on the External Tool activity. (6) The activity redirects the student to Atenea Exams, where the student is automatically logged in by the LTI protocol, and the student can take her test.
With LTI external activities the students find the questionnaires transparently, as they would do with another quiz in the Atenea course. And the grades are set back in the Atenea gradebook. With LTI external activities the students find the questionnaires transparently, as they would do with another quiz in the Atenea course. And the grades are set back in the Atenea gradebook.
However, the complication of creating questionnaires in Atenea Exams, publishing them as an LTI activity and incorporating them as an external activity in the Atenea course is completely passed on to teachers. This is not trivial, if we consider the situation: more than learning how to do online lectures for a course where the workload is already very high, the move to teleworking (often working with each teacher's particular equipment and internet connections) and in confinement situation, competing for spaces, resources, and quiet moments to work with other family members, often also confined students.
This involved a significant effort of communication and faculty training. Training materials and courses had to be put together urgently, including web documentation and a training video explaining to teachers how to create questionnaires and integrate them into their courses [29]. Three training sessions were also organized by videoconference, in which approximately a hundred participants in each, and lasted for almost two hours.
The results of the experiment was satisfactory. With the combined use of the Atenea and Atenea Exams platforms, the evaluation process could be conducted throughout the university with minimal incidents.

Conclusions
A university's Virtual Campus is a critical asset. This study case shows that using Open Source technology for critical information systems is a good practice. The organization cultivates the expertise at all levels necessary to face a challenge or a crisis. Having the operational, technical, internal training capacity, analysis and proposal of solutions to problems, and decision making.
In the case of Moodle, the UPC had cultivated this expertise over the previous 15 years plus a collaborative relationship with the Moodle.org community.
The massive usage of the Quiz Module in the Atenea platform entailed the risk of recurring failures of the single instance of the LMS for the university. The consequences of such failures would be dire in a critical moment for the institution. The solution provided was the use set of interoperability technologies, which were coincidentally contributed to the Moodle community by researchers from UPC, to create a SaaS plug and play version of the Moodle Quizz that spared the use of internal resources from the LMS platform and preserving its stability. However, the complication of creating questionnaires in Atenea Exams, publishing them as an LTI activity and incorporating them as an external activity in the Atenea course is completely passed on to teachers. This is not trivial, if we consider the situation: more than learning how to do online lectures for a course where the workload is already very high, the move to teleworking (often working with each teacher's particular equipment and internet connections) and in confinement situation, competing for spaces, resources, and quiet moments to work with other family members, often also confined students.
This involved a significant effort of communication and faculty training. Training materials and courses had to be put together urgently, including web documentation and a training video explaining to teachers how to create questionnaires and integrate them into their courses [29]. Three training sessions were also organized by videoconference, in which approximately a hundred participants in each, and lasted for almost two hours.
The results of the experiment was satisfactory. With the combined use of the Atenea and Atenea Exams platforms, the evaluation process could be conducted throughout the university with minimal incidents.

Conclusions
A university's Virtual Campus is a critical asset. This study case shows that using Open Source technology for critical information systems is a good practice. The organization cultivates the expertise at all levels necessary to face a challenge or a crisis. Having the operational, technical, internal training capacity, analysis and proposal of solutions to problems, and decision making.
In the case of Moodle, the UPC had cultivated this expertise over the previous 15 years plus a collaborative relationship with the Moodle.org community.
The massive usage of the Quiz Module in the Atenea platform entailed the risk of recurring failures of the single instance of the LMS for the university. The consequences of such failures would be dire in a critical moment for the institution. The solution provided was the use set of interoperability technologies, which were coincidentally contributed to the Moodle community by researchers from UPC, to create a SaaS plug and play version of the Moodle Quizz that spared the use of internal resources from the LMS platform and preserving its stability.
The UPC was been able to respond to the challenges of the COVID-19 crisis because it has this internal talent. When critical and strategic services are outsourced certain complexities cannot be addressed. The response must come from an external entity with other priorities and no obligation to respond beyond the agreed terms of quality of service (QoS). Since the UPC and UPCnet operate their own private cloud, these problems fall to the roof of the UPC, which has the ability to solve them on its own, maintaining the option of using Public Cloud infrastructures if necessary.
On the other hand, some of the services needed to make the transition to face-to-face teaching are the tools in the Google Suite that were already deployed at the UPC just before the emergency. This indicates that the line that separates the services that a university must self-procure and those that it must outsource depend on the criteria of each institution.
However, the authors believe that the essential services of Virtual Campus and academic management must be operated by the institution itself using Open Source technologies, and the capacity and talent to innovate in this space must be retainedleaving interoperability mechanisms such as IMS LTI to incorporate third-party services as appropriate.

Institutional Review Board Statement:
The study did not require ethical approval. No personal information was used in this study. All data was anonymized and aggregated.

Informed Consent Statement:
According to the GDPR all the students using the Atenea platform have been informed and given consent to the treatment of their data.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author. The data are not publicly available due to compliance with GDPR.