Evaluating GraphQL and REST API Services Performance in a Massive and Intensive Accessible Information System

Currently, most middleware application developers have two choices when designing or implementing Application Programming Interface (API) services; i.e., they can either stick with Representational State Transfer (REST) or explore the emerging GraphQL technology. Although REST is widely regarded as the standard method for API development, GraphQL is believed to be revolutionary in overcoming the main drawbacks of REST, especially data-fetching issues. Nevertheless, doubts still remain, as there are no investigations with convincing results in evaluating the performance of the two services. This paper proposes a new research methodology to evaluate the performance of REST and GraphQL API services with two main ideas as novelties. The first novel method is the evaluation of the two services is performed on the real ongoing operation of the management information system, where massive and intensive query transactions take place on a complex database with many relationships. The second is the fair and independent performance evaluation results obtained by distributing client requests and synchronizing the service responses on the two virtually separated parallel execution paths for each API service, respectively. The performance evaluation was investigated using basic measures of QoS (Quality of Services), i.e., response time, throughput, CPU load, and memory usage. We use the term efficiency in comparing the evaluation results to capture differences in their performance measures. The statistical hypothesis parameters test using the two-tails paired t-test, and boxplot visualization was also given to confirm the significance of the comparison results. The results showed REST is still faster up to 50.50% in response time and 37.16% for throughput, while GraphQL is very efficient in resource utilization, i.e., 37.26% for CPU load and 39.74% for memory utilization. Therefore, GraphQL is the right choice when data requirements change frequently, and resource utilization is the most important consideration. REST is used when some data are frequently accessed and called by multiple requests.


Introduction
For the past two decades, Representational State Transfer (REST) has been considered the standard architecture for designing and implementing Application Programming Interfaces (APIs) as back-end applications or server-side web services. However, it is believed that the practical use of REST in the API world will soon change due to the potential emergence of GraphQL. Most middleware application developers today are at a crossroads whether they should stick with REST or face the challenges of the emergent GraphQL technology when building and implementing APIs. GraphQL is considered a revolutionary technology because of its simplicity of use and its ability to reduce workload problems on the server; hence, it can overcome the main drawbacks of REST in data retrieval [1][2][3][4]. However, there are still a lot of doubts and talk about GraphQL when compared to its predecessor REST. This doubt arises because there have been no convincing experimental investigation results in evaluating the performance of REST and GraphQL API services that support Quality of Service (QoS).
The REST API is built on several endpoints to determine data retrieval since those endpoints are called frequently. Multiple endpoints will increase the number of client-server calls for data retrieval, which may be more efficient than using a single endpoint. Therefore, the REST response speed is considered adequate but requires a lot of computational resources such as process loading and memory consumption [5][6][7]. On the other hand, a GraphQL API built with a single endpoint will be a little slow at running complex database queries that have multiple relationships. However, this problem was addressed by exposing the graph structure for the data service. The advantage of retrieving data on a single endpoint can result in efficient server resource workloads such as CPU load and memory usage. Simply put, it is nothing more than a query language for building API services. Although GraphQL query language is not tied to any particular database or storage engine, it is supported by existing code and data. GraphQL positions on the server side which executes the query using the system type defined for the data. This API technology was developed by Facebook in 2012 and immediately boomed among web application developers when the technology was published in 2015 [8][9][10][11].
The performance comparison of REST and GraphQL has been reported in some experimental papers. However, the results are mostly from simulations or execution of simple application development, therefore the results are either inconclusive or inconsistent. Experiments have tested basic operations in network services, i.e., displaying, adding, editing, and deleting data [12]. The performance tests are measured using the JMeter tool. The results showed no difference in performance in adding, editing and deleting data for both services. Differences in performance for display operations are observed under heavy load conditions. GraphQL has better performance in downloading small amount of data, while REST indicated higher performance in downloading large amount of data [12]. Contradictory results were obtained in [13], migration to GraphQL resulted in improved performance in terms of average number of requests per second and data transfer rate for two-thirds of the applications tested. However, it was found that the performance of the services after migration for GraphQL was below REST. Both REST and GraphQL architectures showed similar performance for more trivial workloads [13]. Another comparative analysis on real website samples using REST and GraphQL architectures in terms of different Internet connection qualities, code complexity and required number of requests is presented in [14]. For different Internet connection speeds, they analyzed the number of different records in the database and the total time required to show users the desired content on the website. Their results showed that GraphQL provides a significant reduction in the total time and number of HTTP requests compared to REST technology, which is very important for the user experience and the quality of web applications [14]. More contrasting result based on a decision model showed more differences between REST and GraphQL technologies in [15]. In their conclusion, REST results in a much more flexible and GraphQL solution that is easy to use but more restrictive for developers. GraphQL also requires an implementation that sets some restrictions on which languages may be used whereas REST does not [15].
In this paper, we propose a research methodology to evaluate the performance of REST and GraphQL API services with two main ideas. The first idea is the performance evaluation of the two services is performed on the real ongoing operation of the management information system, where massive and intensive query transactions take place on a complex database with many relationships. The data source for this investigation is primary data from transaction queries on a Management Information System of the Hasanuddin University Research and Community Service Institute, called SIM -LP2M [10]. Therefore, the results of the performance evaluations, derived from real client transaction activities that cannot be repeated, not simulations, must be carefully and accurately conducted in such a way that does not interfere with the operation of the running system. The second idea is the fair and independent performance evaluation results obtained for both API services. In this method, the request synchronisation node plays an important role in distributing client requests and synchronising service responses to two parallel execution paths for each API service. The execution path resides in a separate virtual machine connected to a single database engine hosts two identical and isolated schemas. For each performance measurement result, the findings of the performance evaluation are given and discussed. This research is expected to contribute to researchers and developers working on information systems or middleware applications that use REST services and the GraphQL API.
The remainder of this paper is organized as follows. Section 2 describes the materials and methods used in the performance evaluation study of two API service technologies. The experimental design used in this work, including the evaluated architecture and implementation of REST and GraphQL, is described in Section 3. Then, Section 4 presents the experimental performance evaluation results and discusses the comparison of these results. Section 5 concludes the paper.

Evaluated System Architecture
The performance investigation for both REST and GraphQL API services is evaluated in two independent monolithic information system architectures. The architecture of these two client-server services will only handle requests made to the endpoint as per the REST and GraphQL specifications. The two services are separated via request sync when a client triggers a series of actions or transactions using the SIM-LP2M system. The synced transaction requests from clients are then distributed in parallel to each API gateway for the respective REST and GraphQL services. The API gateway serves as an entry point to provide secure and unified access as well as manage incoming and outgoing traffic to each API service. Without a gateway, transactions would just be a segmented collection of related APIs and certainly less effective than a unified API, which can also provide identity validation services for performance evaluation. The API services are bounded to a single database engine that hosts two identical and isolated schemas virtually for both REST and GraphQL. Thus, we can evaluate the performance of both services fairly and independently. The connection between the server and the client is made using the Hypertext Transfer Protocol (HTTP) [16]. The illustration of the evaluated system architecture is depicted in Figure 1. The SIM-LP2M is deployed in 3 entities, i.e., web/mobile applications or clients, web services (server side applications), and database (persistent storage resource). Clients of the system consist of 7 actors, i.e., Rector of the university, Chairman of LP2M, Members of the National Research Council of the university, office, staffs, administrators, reviewers and researchers. Each actor has their own privileged tasks and some restrictions in accessing and modifying data. Restriction access to the data attributes in the database is implemented according to the designed RACI (Responsible, Accountable, Consulted and Informed) matrix. This matrix is also used as the basis in designing the class diagrams for database tables and methods/procedures for each user [16].

Data Sources
The primary data source originates from requests directed to the API gateways and services that generate queries to the database as a result of client activities on the system. When there are a series of actions or transactions to the database such as CREATE, READ, UPDATE, and DELETE (CRUD), they are first synchronized and then distributed in parallel to the REST and GraphQL API gateways respectively. Therefore, data related to response time will be counted from the difference between request and response times recorded in the API gateway. Meanwhile, other data related to throughput, CPU load and memory utilization are measured on API service activities. The implementation of CRUD actions from the front-end application to the API gateways are expressed in HTML methods, i.e., CREATE (PUT/POST), READ (GET), UPDATE (PUT/POST/ PATCH), and DELETE (DELETE). To ensure data validity despite errors and failures, ACID properties (i.e., atomicity, consistency, isolation, and durability) are applied to database transactions.
The performance data are evaluated from the real transaction activities in the SIM-LP2M system that is quite complex and very diverse. These activities include the research proposal submissions, administrative desk evaluations, evaluation of research presentations, budget justification, evaluation of proposal revisions, evaluation of the proposal score, reviewer selection, decisions of grant recipients, monitoring and evaluation (research progress and final results), reporting activities, etc. Along with these activities, the SIM-LP2M system intensively and massively accesses the database server with various actors and tasks. Since the performance data are primary and they are derived from the real operational of client transaction activities, they are not simulations, cannot be repeated, and must be carefully and accurately conducted in such a way that does not interfere with the operation of the running system.

Application Programming Interface (API)
The context of Application Programming Interfaces (APIs) in the field of software engineering and information systems is very different and refers to the interface elements of an application that can be called or executed at various levels of abstraction in a system. To further narrow the scope, this paper limits the API in context as a backend-side functionality that queries specific data on a persistent storage resource via an interface defined as API web services. The web services concept appears to bridge existing information systems (middleware) without disputing the difference in the platforms used by each source [3,6,15,17]. In building web services, the technologies most often applied are SOAP, REST, and most recently GraphQL, a new approach that leverages services in the query language released in 2015 by Facebook.

Representational State Transfer (REST)
REST is still the popular API design architecture used to implement web services that use the HTTP protocol for server-client communication in web applications. Creating an API with this concept is simple, easy, and flexible to accept since it is not bound by a specific transfer protocol. REST works by providing a path or endpoint to access a resource (data source) and the endpoint will be used whenever a client wants to access the resource. The server sends a representation of the requested resource state to the client that mostly in JSON or XML format. REST API typically gathers data by accessing multiple endpoints. For instance, let a client want to fetch the data of a researcher that includes the id, research titles, and members of a research project. Firstly, these could be /data/<id> endpoints to fetch the initial user data. Secondly, there will be a /data/<id>/research endpoint that returns all research titles for a user. The third endpoint will then be /data/<id>/members that returns a list of members per user. In essence, the REST APIs server is stateless, which means the client is responsible for storing all application state related client side information. The client needs to send state information to the server whenever needed. The client and server can operate independently at REST, which means the state information can be overridden and all data retrieved from the server can be cached on the client side in such a way that the overall performance can be improved. The REST APIs are based on 5 different HTTP methods, i.e., GET to retrieve data, POST to update data on the server, PUT to update property values, PATCH to modify a resource and DELETE to delete the resource. The main REST design principles are addressability, uniform interface, and statelessness. REST overcomes acceptance by specifying an endpoint in the directory structure via various URIs to extract data [5][6][7]17,18].

GraphQL
GraphQL is a query language for an API. It is not tied to any specific database or storage engine, but it is instead backed by your existing code and data. In essence, GraphQL has only one endpoint, which is HTTP POST request. In GraphQL, we will simply send a single query to the GraphQL server that includes the concrete data requirements. The server then responds with a JSON object where these requirements are fulfilled. For instance, let us use the same activity explained in the previous REST subsection, where the client wants to fetch the researcher data that includes the id, research title, and members of the research project. Then the client only specifies exactly the data it needs in a query and sends a request using the HTTP POST method. The specified query request sent to GraphQL using the POST method can be as shown in Figure 2. In GraphQL, the client sends customized POST request to fetch the only needed data. In other words, clients have the power to define what type of data to fetch from the database. It helps when the client needs a flexible response format to avoid extra massive data transfer. It is also supported in various popular programming languages such as Python, C#, JavaScript, etc. Two things are the most important part of GraphQL, i.e., Schema and Resolver function. GraphQL requires 3 important parts that must be defined firstly, i.e.: Mutation as a place to update, input, and delete data. Query for calling data, and Type to describe the type of field/data [8,10,11,19,20].

Architectural Difference of REST and GraphQL
The main difference between REST and GraphQL APIs technology is the availability of the number of endpoints when they are handling requests. GraphQL tries to collect every request to one place, whereas the REST is built such that each resource is handled by some specific endpoints. The REST configuration creates a complicated situation whenever there is a change on the back-end, it will cause an adjustment to the front-end to query the exact endpoint for the desired resource. On the other hand, GraphQL will take some time when the request is handled at a single endpoint to find the right resource [1,3,12,13,16]. Figure 3 illustrates the difference between REST and GraphQL architecture.

Performance Measurements
Web services have proven to be easier to implement and superior to other similar technologies available today. Web service performance concerns not only how fast requests can be processed and served, but also other performance in terms of load, stress, and service durability. Service speed requirements are mostly determined using throughput and response time measurements. Meanwhile, web service workload performance is more easily demonstrated through CPU load and memory usage. Obviously, the response time and throughput depend on the workload the web server is experiencing at the time [12,14,16,21]. Therefore, this paper evaluates 4 basic performance measures that clearly distinguish the two technologies based on their natural purpose. The four performances are response time, throughput, CPU load, and memory utilization. They are the most common metrics that can be tracked as good baseline data points and most needed in achieving QoS. Later in the results and discussion section, the efficiency of the results of the performance evaluation of the two technologies will be evaluated because it is a comparative measure for the discussion of the two technologies. We use the following Equation 1 to compute the efficiency of performance measures.
where and ℎ are the results of the REST and GraphQL performance measurements, respectively.

Data Model
The performance evaluation is investigated by a multiple experimental method that represents the differences between the two service technologies. In this case, the same query operations are performed on the two technologies for retrieving, loading, modifying, storing, and deleting data on the server side, which are manipulated from the client side. Comparison of the key performances in differentiating the two technologies is evaluated by calculating response time, throughput, CPU loads, and memory utilization. Before performing a data query, all attributes or variables in the database must be declared in a data model file as illustrated in Figure 4. Performance analysis is carried out using the Insomnia API testing application. Testing was carried out 20 times on samples repeatedly for both REST and GraphQL, respectively.

REST Implementation
Query of the REST data call is done on the controller and the data format is the same when using GraphQL, i.e., in a JSON format. REST uses Data Manipulation Language (DML) in the SQL programming language. DML is a collection of query commands that are used to manipulate data in a database, which is used to append, update, or delete data in the database. An example of request-response cycle of data query in REST is described in Figure 5.

GraphQL Implementation
GraphQL is a new concept in building open source query language APIs developed by Facebook. GraphQL executes server-side queries and only returns data specified by the system type to the related web service. Although GraphQL is a query language, it is indirectly related to the database. In other words, GraphQL is not limited to certain databases whether SQL or NoSQL. GraphQL is only a translator (query language) and runtime, thus it does not depend on any server-side programming language and database. The variables and fields available for the query are specified in the schema located on the server side. Specific queries can be built based on GraphQL having predetermined services available for queries allowing for a single endpoint rather than multiple endpoints. Figure 6 shows an example of request query and JSON response cycle using GraphQL.

Response Time
Response time is a measure of service performance that refers to the amount of time it takes for a service to respond when it receives a request. It is important to know the average and peak response times to services. Tracking both of these allows seeing how the service fluctuates with more or less load. Average response time is a good metric for gathering data in system performance tests. It provides an overview of the general user experience and provides insight into regression in case of changes. It is also important as the ideal baseline response time such that any lags that should be investigated or considered critical can be identified. Peak response time allows seeing the performance of the slowest requests. This creates a look that is different from the overall average. With peak response time, we can find more specific queries that might be problematic and determine what worst situation a user is in. Viewing the slowest queries can make it easier to track specific operations with latency, whereas the average response times are more general and give an idea of the entire system.
We evaluated 20 groups of on-going transactions in the system (hereinafter referred to as trial tests) each of which resulted in an average request-response time difference of 100 trial tests with various HTTP methods. There is no time set for when the trial test group is started because the focus is on the issue of how much time difference can be obtained between the time the request starts to be sent (which is recorded in the request timestamp) and the response timestamp of the corresponding request received. Each group of trial tests is applied in a distributed manner to the REST and GraphQL services respectively in parallel execution. The request and its corresponding response timestamps are recorded on each API gateway, respectively. The response time is obtained by computing the timestamp difference between the timestamp request message i was issued, ts(reqi), and the timestamp of the corresponding response message i is received, ts(respi), in the API gateway. We use the following Equation 1 to determine the average response time in the j th trial test group (RTj).

Throughput
We evaluated the throughput as the cumulative number of requests in REST and GraphQL transactions with data fetching performance in every interval time of 10 milliseconds, respectively, using Equation 2 as follows.
where is the throughput of the services in the time interval of i and # is the number of messages handled in the range of time t, | -| ≤ 10, = 10, 20, … , 200. The throughput results are given in Table 2.   Figure 8.

CPU Load
We evaluated CPU load without any other loads, services, and processes in the same setup machine in each API service server separately. The evaluation is conducted at the same time when evaluating the performance of the throughput. CPU loads of the services were recorded in every 10 ms and the results are shown in Table 3. In the first 10 ms of our experiment resulted that the REST used 82.5% CPU as peak usage and remained slightly fluctuating in heavy usage and has an average 75.50% within the range 70% -82.45%. Meanwhile, GraphQL is consistently steady with average 47.37% within the range 44.43% -50% and standard deviation 1.27%. CPU usage graph is depicted in Figure 9. The evaluation of memory usage is computed at the same time with the evaluation of throughput and CPU load. Memory consumption of the services is stored in every 10 ms and the results are shown in Table 4.

RESTful
GraphQL very stable and low in consuming memory as depicted in Figure 10. On the other hand, memory usage at REST tends to fluctuate periodically with alternating peaks and valleys but still under heavy usage in the range 55 -80 MB.

Discussion
This subsection discusses the performance comparison of REST and GraphQL based on the recapitulation of performance results as given in Table 5. The discussion is based on efficiency and stability of the performance variation measures (standard deviation). Furthermore, the reasons for the results of the performance comparison will be given based on the suitability of the nature and purpose of the technology. Recapitulation of the experimental results in Table 5 showed that REST is superior to GraphQL in performance with respect to speed. The response time performance at REST is 50.50% more efficient with a response variation of 32.99 ms. Furthermore, REST throughput performance is also 37.16% more efficient, however, GraphQL is more stable in handling requests with a very small variation of 103.01 requests in each 10 ms. This is because experiments executing REST requests are run per method on multi-endpoints running on Hypertext Transfer Protocol version 2 (HTTP/2). Furthermore, HTTP/2 has a multiplexing feature that allows multiple requests or methods at each endpoint to be executed simultaneously across their entire connection. Thus, the response time is faster and the number of requests/methods handled (throughput) per request is greater. However, this causes the amount of throughput to vary as the method response has a different amount of data transferred at each endpoint. Unfortunately, in the worst case, this can lead to under-fetching or over-fetching of data as each response will return the full set of data according to the request field regardless of the balance of data counts. Of course, this mechanism can cause high fluctuations in CPU load and memory usage. Another reason for this can also be due to the variable structure of the database table attributes. That is why memory utilization fluctuates.
GraphQL is superior to REST in terms of performance regarding resource consumption and utilization. For CPU load performance, GraphQL is 37.26% more efficient and stable in utilizing processing resources with a variation of only 1.27%. Furthermore, GraphQL memory consumption is also superior with an efficiency of 39.74% and is very

Time since start (ms)
RESTful GraphQL stable with a variation of only 1.03 MB. The stability of GraphQL in consuming memory and utilizing CPU usage occurs because the query has selected data fields or properties for specific requests (or contains complete data requirements). This results in a much smaller and more efficient utilization of computing resources with certain fixed data. In addition, the request is only made once at a single endpoint and the client can specify flexible response formats to avoid very large data transfers (over-fetching) or sparse data response (under-fetching), as can happen with the REST API service.

Conclusions
We have designed and implemented a performance evaluation method for REST and GraphQL web services based on the two main ideas proposed. There are some differences in developing applications that implement REST and GraphQL web service technologies in monolithic information systems or web-based backend applications. In GraphQL, 3 important parts need to be defined before using Data Manipulation Language (DML), i.e., mutations, queries and types, while REST uses DML directly in SQL programming language. The performance evaluation in our experiment revealed some differences between the two API services. In terms of service speed, REST is superior by 51% in response time and 37% in query throughput because its queries can use DML directly and are more stable in terms of access time and data size. GraphQL is dynamic because changes can be made as needed. In the context of resource utilization, GraphQL is the best choice for building web applications or information systems that implement web service technologies. GraphQL is 37% superior in CPU load and 40% superior in memory utilization because its process structure can manage queries on single endpoints. REST uses multiple endpoints to improve performance when retrieving data. We can conclude that the use of REST and GraphQL services depends on the requirements of the system and the particular application. GraphQL is a good choice when data requirements change frequently and resource usage is a concern, while REST is used when some data is frequently retrieved on demand and is therefore very suitable for monolithic information systems.

Supplementary Materials: Not applicable.
Author Contributions: All authors have contributed equally.
Funding: Not applicable.

Data Availability Statement:
The data used to support the findings of this study are available from the corresponding author upon request.