Analytics Framework for Comparing National Performance Achievements in International Mathematical Olympiads

Abstract

This study measures and analyzes the performances of the participating nations at the International Mathematical Olympiad (IMO). A country’s success at the IMO is evaluated by the number of gold, silver, and bronze medals won. To measure and compare the performances of the countries, the data envelopment analysis approach is employed and all countries that have won at least one medal have been included in our dataset. As measures for the inputs, two macro-level determinants that can support the effort made by a country are considered: population size and gross domestic product (GDP) per capita. Our findings indicate a mean level of technical efficiency for the nations analyzed. The results suggest that, on average, the inefficient countries in the sample could achieve improved results by increasing the number of outputs produced while maintaining the same level of effort spent.

1. Introduction

This study analyzes the performances of countries at the International Mathematical Olympiad that was held in remote format and administered from Saint Petersburg (the Russian Federation) in 2021. The International Mathematical Olympiad (IMO) is the oldest international annual science Olympiad, and 619 students from 107 countries participated in this competition. The IMO is the world championship mathematics competition for high school students and is held annually in different countries. In this study, the macro-level determinants of the Olympians’ performances at the IMO are investigated in order to measure the performances of the participating countries. A nation’s success at the IMO can be judged by various measures, including by the number of gold, silver, and bronze medals [1].

In order to measure the performances of the nations in terms of their mathletes’ achievements, the data for the year 2021, published by IMO.org, were utilized and appropriately modified as required. The current research concentrated on the country-specific characteristics that can support the performances of the mathletes. The results from this study provide information concerning the technical efficiency of the nations analyzed. The study indicates that by solving our model, one can obtain the efficiency scores for the participating nations investigated. Our approach draws on studies of the performances of nations at international sporting events, where researchers employ various social economic measures as inputs to measure and compare various outputs, but mainly the medals won [2,3,4].

The results of our analysis show an average level of technical efficiency for the nations analyzed and indicate that further training for the targeting of realistic objectives for future events can become a plan to pursue. The method can help nations to compare themselves with their peers and to discover the gap in their outputs and can thus be utilized as a tool that shows that some countries may not be performing as they should. In our investigated sample, there are nations with sufficient inputs which are efficient in their outputs, whereas the less efficient ones could be considered undersized in terms of the outputs achieved.

The purpose of the present study, however, is not only to assess the performances in mathematics of the students and young people from the various countries from some analytics point of view, but also to discuss the common features existing in mega-events and to confirm the degree of effect that the size and wealth of the participating nations has on the outcomes of these events. Although one can easily agree that no direct relevance exists between the International Mathematical Olympiads and the athletic mega-events in terms of the number of participants and the intensity of the investments involved, one can easily recognize, however, that performance in such major international competitions is still related to the economic and social prosperity of the individual participating nations and that there is also a strong relationship with the traditions and history of the participating nations. Some researchers also relate the IMO’s results to the efficiency of the individual national educational systems [5]. Moreover, as prior research has proved [6], for athletic mega-events one can provide both the organizers and the participants with valuable advice on how to manage and organize these events more effectively. This can be achieved by creating the optimal processes and environment; by implementing systems and structures; by developing an engaging and inclusive culture; and by providing appropriate support. All these factors are especially important for young people. Finally, assessing the mathematical capabilities of a nation, with respect to the performances of the other nations in a major international event, provides a very useful tool for education policy makers for adjusting national educational policies and/or adopting, perhaps, the successful practices of other nations. To the best of the authors’ knowledge, the present work is the first to assess the performances of countries in mathematical Olympiads using data envelopment analysis (DEA).

The remaining sections of this article are organized as follows. In the next section, the literature related to measuring performance in Olympic events using DEA is presented. In Section 3, the methodology employed to implement the assessment of the participating countries in the 2021 Mathematical Olympiad is discussed, and an analysis of the results is then presented. In the following section, the data utilized in this study are thoroughly presented and discussed. Finally, in the last section, the conclusions are drawn and recommendations for future work are provided for those who manage and participate in these non-athletic events.

2. Literature Review

All organizations use resources (materials, labor, etc.) to produce goods and services. This transformation adds value to the outputs produced, and managers are always interested in how efficiently this transformation is achieved. Measuring performance efficiency is of major importance to organization leaders. Numerous quantitative approaches, both parametric and non-parametric, have been applied to evaluate performance. Parametric methods such as stochastic frontier analysis (SFA) measure efficiency using random error terms [4], while non-parametric methods such as data envelopment analysis are based on mathematical programming techniques.

DEA is a powerful method to evaluate the relative performance of distinct decision making units (DMU) using various input and output measures [7]. Over the last three decades, DEA has consistently progressed and improved its modelling approaches to cater for complex real-world problems [8,9,10,11,12,13,14,15].

Several studies to measure the efficiency of Olympic athletic events or other athletic mega-events have been presented recently. However, although considerable research has been devoted to Olympic athletic events, only limited attention has been paid to Olympic non-athletic events, particularly the mathematical Olympiads.

In the DEA-based studies of Olympic events, the participating countries are mainly evaluated according to the number of gold, silver, and bronze medals won. Furthermore, the evaluation takes into account the specific conditions for each participating country, such as population and GNP, as input measures.

DEA has been applied to assess the performances in the Olympic Games in several studies [16,17,18,19,20,21,22], where the relative efficiency of the participating countries that won at least one medal in relation to their available resources was analyzed. In [16], the authors used an input-oriented model, where the number of athletes was considered as an input, and the number of gold, silver, and bronze medals won was considered as the output parameter. Moreover, DEA has been applied for the evaluation of soccer leagues (in, e.g., [23,24,25]). Moreover, the performance of Major League Baseball was also evaluated through DEA models [26], while ref. [27] focused on the analysis of the efficiency of NBA teams with the use of DEA. To tackle more realistic problems, researchers (in, e.g., [17,28]) applied weight restrictions on the outputs by introducing the assurance region approach for all participating nations. As mentioned before, most of the prior research looked at a variety of measuring parameters. However, there are two measures that are common in almost all studies: population and GDP. The reasons for the inclusion of these two measures are almost obvious: more populous countries have a greater pool of talent to draw on and thus have enhanced chances of success at the Olympics. Moreover, wealthier nations can afford increased investments in sports projects and thus would be more likely to win medals at the Olympics.

There are also few studies that evaluate the utilization of non-DEA methods for analyzing the performances of nations. For instance, in [29] the authors used a weighted directed graph to build a dominance network, which in turn can be used to characterize the performance differences between the countries participating in the Olympic Games. In [30], the researchers examined two different performance-ranking approaches based on weighted mean values and volume-based sensitivity analysis. Most recently, the researchers also experimented with optimization methods such as particle swarm optimization (PSO) [31] or machine learning (ML) and data mining (DM) techniques in order to evaluate the performances of athletes and teams [32,33]. These methods have been applied mainly to sports events (such as basketball), for which a variety of data concerning athlete and team performances can be collected via automated tracking devices due to technological advances.

3. Methodology

Data envelopment analysis (DEA) is a method for measuring the relative performance of distinct units where the presence of multiple consumed resources and end results achieved makes it difficult to compare these units in a direct way. In data envelopment analysis, those units whose efficiency is estimated are generally referred to as decision making units (DMUs). DMUs consume resources to generate a set of outputs of varying levels. The resources used and the end results of a decision making unit can be numerous and measurable in different units. In the context of a system of such units, all the decision making units are considered to consume the same resources and produce the same end results (i.e., the metric units are alike) and only their input and output levels differ.

The typical performance measure for efficiency, i.e.,

efficiency = output/input

is not sufficient to evaluate the DMUs because of the presence of multiple inputs and outputs associated with different sources, activities, and environmental factors. Thus, a common measure of relative performance is defined as the ratio of the weighted sum of the outputs to the weighted sum of the inputs. The sum of the weights of all the input/output criteria must be equal to the unit. The above relative performance measure requires the definition of the weight factors.

Charnes, Cooper, and Rhodes [6] proposed one of the most basic DEA models, appropriately termed the CCR model. Given that there are n DMUs and associated numerical data for each of the m inputs and s outputs for all the DMUs, the fractional mathematical programming problem that is solved in order to obtain values for the input weight (v_i) (i = 1, ..., m) and the output weight (u_r) (r = 1, …, s) variables is the following [34]:

$$\begin{array}{c}max z=\frac{{\displaystyle \sum _{r=1}^s{u}_r{y}_{r{j}_0}}}{{\displaystyle \sum _{i=1}^m{v}_i{x}_{i{j}_0}}}\end{array}$$

(1)

subject to

$$\begin{array}{c}\frac{{\displaystyle \sum _{r=1}^s{u}_r{y}_{r{j}_{}}}}{{\displaystyle \sum _{i=1}^m{v}_i{x}_{ij}}}\le 1\hspace{1em}\left(j=1,...,n\right)\end{array}$$

(2)

u_r ≥ 0,  (r = 1, …, s),

(3)

v_i ≥ 0,  (i = 1, …, m),

(4)

In the above model, index j refers to the DMU being evaluated. Objective function (1) maximizes the ratio of the output to the input of the DMU under evaluation by calculating the appropriate weights v_i and u_r. Constraints (2) ensure that this ratio does not exceed 1 for every DMU. This implies that the objective function value lies between 0.0 and 1.0; the latter value denotes that the DMU under examination is efficient. The above non-linear program is linearized, and the solution of its linear equivalent produces the efficiency score of the DMU being evaluated.

However, a common problem in several real-world DEA applications is that all the inputs and outputs may not be equally important to the decision maker. This study reports an application where weight restrictions are imposed in a DEA model to overcome the above problem. A weight-restricted DEA model is employed by introducing a discrimination criterion between several efficient units. This criterion is based on the fact that the input and/or output weights can be restricted since the decision maker knows the policies, i.e., the importance of the inputs and/or outputs. Introducing weight restrictions in DEA may decrease the efficiency scores and decrease the number of efficient DMUs. The following constraints are thus added to the CCR model:

$${\alpha }_r\le \frac{u_r}{u_{r+1}}\le {\beta }_r,\hspace{1em}(r=1,...,s)$$

(5)

where u_r is the weight attached to the r_th output and corresponds to the variables of the CCR model. The Greek letters α_r and β_r are user-specified constants to reflect the value judgements related to the relative importance of the output factors [35].

For the validation of the results, two basic parameters, which describe the method, must be defined:

• Input/output orientation of the model: the model is configured so as to determine how much the input/output consumed/produced could decrease/increase if resources are used efficiently;
• Constant/variable returns to scale (CRS/VRS) assumption: CRS reflects the fact that output will change by the same proportion by which the inputs are changed (e.g., a doubling of all inputs will result in double the outputs); VRS reflects the fact that production technology may exhibit increasing, constant, or decreasing returns to scale.

In the present study, an output-oriented DEA model that assesses the efficiency of the countries is utilized in order to maximize the outputs/medals obtained given the inputs of each of the countries. The most general assumption that can be made is that the returns to scale parameter is variable; thus, a variable returns to scale (VRS) and weight restriction DEA model is formulated and solved as our data also have some special features. As mentioned above, weight restrictions are incorporated in our model to allow for the integration of preferences in terms of the relative importance levels of the various inputs and outputs. In our case, the weight restrictions are incorporated to account for the relative importance of gold, bronze, and silver medals.

4. Data

Perhaps the most important step in conducting a DEA study is to determine the input and output indicators to be used in order to evaluate the efficiency of the decision making units (DMUs) in the sample. Several studies have attempted to ascertain the factors affecting the number of Olympic medals won by countries. Most studies have found that population and GDP were positively related to the number of Olympic medals awarded. Although some of the previous literature on the participants at the International Mathematical Olympiad focuses on the theories related to individual talent formation, this study focusses on country characteristics. National teams from countries with a higher level of material wealth and larger populations win on average more medals at the Olympic Games than the teams from smaller and poorer countries.

Models (1)–(5), as presented above, are applied to the competition of the International Mathematical Olympiad that was held in a remote format and administered from Saint Petersburg, the Russian Federation, in 2021. The DMUs consisted of the 80 participating countries in the 2021 IMO competition that won at least one medal. The participants are ranked based on their individual scores. Medals are awarded to the highest ranked participants. The input and output indicators for the studied 2021 IMO studied as follows.

Inputs:

• Participating country’s population;
• GDP per capita.

Outputs:

• Total score accumulated resulting in the award of gold medals;
• Total score accumulated resulting in the award of silver medals;
• Total score accumulated resulting in the award of bronze medals.

A total of 52 gold medals were awarded to the participants who had accumulated a score ≥ 24 points. One hundred and three silver medals were awarded to the participants who had accumulated a score ≥ 19 points. Moreover, 148 bronze medals were awarded to the participants who had accumulated a score ≥ 12 points. In relation to the weight restriction constraints, the coefficients α_r and β_r were set equal to 1 and 2.5 for the gold to silver medal output ratio (r = 1). For r = 2 in the constraints (5) above, α_r and β_r were set equal to 1 and 1.5 for the silver to bronze medal output ratio This latter judgment is based on the guidance given by the fact that the numbers of gold, silver, and bronze medals awarded are approximately in the ratio of 1:2:3 and on the results announced by the IMO organization concerning the actual number of medals in each category awarded in the IMO 2021 event.

The datasets of the population and of the GDP per capita of the participating countries are both collected from the official website of the World Bank. Olympic medal counts and the corresponding recorded scores were obtained from the official website of the IMO organization [Appendix A].

The descriptive statistics characterizing the inputs and outputs of the 80 participating countries that won at least one medal appear in

Table 1. Descriptive statistics of inputs and outputs for 80 participating countries.

Features	GDP per Capita	Population	Gold Medal Score	Silver Medal Score	Bronze Medal Score
Mean	23,545.71726	73,388,572.98	19.66	27.59	26.59
Standard Deviation	22,712.86862	222,441,422.5	39.71	27.14	18.26
Min	897.087902	620,173	0.00	0.00	0.00
Max	93,457.4404	1,412,360,000	208.00	100.00	79.00

, which contains the average level, the standard variance, the minimum level, and the maximum level. The large standard deviation of the inputs in Table 1 demonstrates that the input data vary significantly. The latter evidence indicates that the appropriate model to use in the current case study is the VRS DEA model.

5. Results and Discussion

The data envelopment analysis provides a collection of useful information on the efficiency scores.

Table 2. The efficiencies of 80 countries based upon CCR VRS weight restricted model.

Countries	Efficiency Scores	Countries	Efficiency Scores	Countries	Efficiency Scores
Argentina	0.29482	India	1	Puerto Rico	0.117316
Armenia	1	Indonesia	0.717203	Korea, Rep.	1
Australia	0.724873	Iraq	0.101411	Moldova	1
Austria	0.20955	Iran, Islamic Rep.	1	Romania	0.660937
Azerbaijan	0.201923	Israel	1	Russian Federation	1
Bangladesh	0.425016	Italy	0.804034	Saudi Arabia	0.43902
Belarus	0.77323	Japan	0.597117	Serbia	0.648156
Belgium	0.312652	Kazakhstan	0.830758	Singapore	0.923645
Bosnia and Herzegovina	0.578776	Latvia	0.504322	Slovak Republic	0.540708
Brazil	0.530838	Lithuania	0.124326	Slovenia	0.321556
Bulgaria	0.898728	Macao SAR, China	1	South Africa	0.277054
Canada	0.9616	Malaysia	0.329936	Spain	0.077867
Colombia	0.249902	Mexico	0.566773	Sweden	0.250324
Costa Rica	0.105926	Mongolia	1	Switzerland	0.304334
Croatia	0.863586	Montenegro	1	Syrian Arab Republic	0.474759
Czech Republic	0.848428	Netherlands	0.185744	Taiwan	0.878607
Denmark	0.104658	New Zealand	0.21858	Tajikistan	1
Ecuador	0.188862	Nicaragua	1	Thailand	0.76463
El Salvador	0.109485	North Macedonia	0.657155	Tunisia	0.288751
Estonia	0.619754	Norway	0.270472	Turkiye	0.584455
Finland	0.090023	Panama	0.13681	Turkmenistan	0.301805
France	0.575077	Paraguay	0.120877	Ukraine	1
Georgia	0.498647	China	1	United Kingdom	0.701677
Germany	0.677041	Peru	0.708673	United States	0.88371
Greece	0.189314	Philippines	0.915146	Uzbekistan	0.550591
Hong Kong SAR	0.811838	Poland	0.875827	Vietnam	1
Hungary	0.740305	Portugal	0.342095

summarizes the findings from the VRS analysis and the efficiency score that each participating country achieved, while Figure 1 illustrates a histogram of the efficiency scores.

As mentioned before, the CCR VRS model is used with the weight restriction coefficients α_r and β_r, which are equal to 1 and 2.5 and 1.5, respectively, for all outputs, r (r = 1, …, s), with the aim of discriminating between the values of the three categories of models.

In Table 2 above, out of 80 countries participating, only 15 countries are shown to be efficient. Moreover, the average efficiency score is equal to 0.5823. It is interesting to note, however, that when the DEA model is solved without the weight restrictions imposed on the three outputs, then the number of efficient countries is increased from 15 to 30 while the average efficiency score is calculated to be equal to 0.7308. The reason for this is that the feasible region of the output multiplier constraints in the model gets smaller. It is assumed in this study that one gold medal is worth at least 2.5 silver medals, and a silver medal is worth at least 1.5 bronze medals. Even though it seems quite reasonable to assume that coefficient β_r is equal to 2.5 or 1.5, depending on parameter r, a sensitivity analysis is usually performed in similar applications for different values of β_r. However, for the purposes of this study a sensitivity analysis seems to be unnecessary as the ratios between the numbers of different medal categories are defined by the IMO organization for all IMO events.

These results show that our proposed model provides a comprehensive ranking system tool for assessing the performances of the participating countries. The results provided by our approach can be compared to the country results published by the official site of the 62nd IMO 2021, where the countries were ranked based on the total score achieved by the individual contestants in all the problems for each country. In

Table 3. IMO2021org., CCR-VRS model, and CCR-VRS weight restricted model rankings.

Countries	IMO 2021 Ranking	CCR VRS	CCR VRS Weight Restricted	Countries	IMO 2021 Ranking	CCR VRS	CCR VRS Weight Restricted
Argentina	46	61	59	Mongolia	11	1	15
Armenia	37	1	3	Montenegro	79	1	7
Australia	18	48	31	Netherlands	47	69	70
Austria	64	66	66	New Zealand	69	64	65
Azerbaijan	51	62	67	Nicaragua	81	1	14
Bangladesh	43	46	52	North Macedonia	45	35	37
Belarus	24	1	28	Norway	51	63	62
Belgium	43	54	56	Panama	72	72	71
Bosnia and Herzegovina	40	1	42	Paraguay	87	71	73
Brazil	35	42	47	China	1	1	10
Bulgaria	18	1	19	Peru	30	1	33
Canada	5	1	16	Philippines	23	1	18
Colombia	61	67	64	Poland	13	1	22
Costa Rica	84	77	76	Portugal	54	60	53
Croatia	21	1	23	Puerto Rico	79	76	74
Czech Republic	16	34	24	Korea, Rep.	3	1	6
Denmark	67	78	77	Moldova	51	1	1
Ecuador	73	65	69	Romania	27	37	36
El Salvador	71	74	75	Russian Federation	2	1	11
Estonia	50	40	39	Saudi Arabia	38	51	51
Finland	69	80	79	Serbia	31	50	38
France	27	43	43	Singapore	15	1	17
Georgia	41	41	49	Slovak Republic	39	47	46
Germany	12	45	35	Slovenia	64	59	55
Greece	64	70	68	South Africa	60	52	61
Hong Kong SAR	21	31	26	Spain	63	79	80
Hungary	32	1	30	Sweden	57	68	63
India	26	1	9	Switzerland	48	56	57
Indonesia	33	1	32	Syrian Arab Republic	68	53	50
Iraq	89	73	78	Taiwan	9	1	21
Iran, Islamic Rep.	29	1	5	Tajikistan	59	1	2
Israel	7	1	12	Thailand	16	1	29
Italy	7	1	27	Tunisia	56	57	60
Japan	25	38	40	Turkiye	35	1	41
Kazakhstan	20	32	25	Turkmenistan	58	55	58
Latvia	48	44	48	Ukraine	6	1	4
Lithuania	76	75	72	United Kingdom	9	39	34
Macao SAR, China	54	1	13	United States	4	33	20
Malaysia	41	58	54	Uzbekistan	61	49	45
Mexico	34	36	44	Vietnam	14	1	8

, the 80 participating countries are listed. In the second column, the ranking order associated with each country, as published by the IMO organization, is presented. In the third column of Table 3, the ranking of the 80 participating countries based solely on the number of gold, silver, and bronze medals won, without taking into account the relative importance of the medals according to their weight restrictions, is presented for comparison purposes. The ranking order associated with each country, as derived from the CCR VRS weight-restricted model discussed in Section 3 above, is also presented in column 4 of Table 3 and graphically in Figure 2 and Figure 3.

In Table 3, the 62nd IMO 2021 ranking has been appropriately adjusted to include the 303 gold, silver, and bronze medals awarded by the IMO organization only. The 98 honorable mentions that were awarded to the participants of the 62nd IMO 2021 have not been taken into account in this ranking. Any participant who receives a score of 7 on any one question but who does not receive a medal is awarded an honorable mention. In our study, honorable mentions were not taken into account for the assessment of the participating countries, nor were any of the scores achieved by the participants that were not awarded a medal or an honorable mention.

As can be ascertained from the results in Table 3, some countries have been completely differently ranked by the IMO organization and the DEA approach. It is interesting to note for example that Moldova was ranked 51st out of 80 countries according to the 62nd IMO 2021 ranking while it was ranked first according to the DEA model rankings. The reason for this is that both the input measures for Moldova are well below the corresponding averages presented in Table 1, and thus, it can be proved that this DMU is efficient. The results show that our proposed approach is more general and reasonable. This is mainly due to the fact that in our model we also consider the value judgment for the weight relationships among the three outputs. In addition, our approach also provides a guide for performance improvement.

6. Conclusions and Recommendations

This study contributes to the measurement of the performances of the participating countries in an International Mathematical Olympiad competition by using a comprehensive model that evaluates the efforts of the participating teams more accurately. Although IMOs do not gain great international visibility when compared to some athletic events, the concern about the results of the countries in such events has been increasing.

With the aim of analyzing the results and improving the performance of the participating countries, the efficiency of the countries participating in the 62nd IMO 2021 was analyzed by implementing the DEA method. The input and output indicators for the 62nd 2021 IMO that were considered were the population and GDP per capita of the participating countries, as related to the three outputs utilized. The three outputs were the total scores accumulated that resulted in the award of gold medals, silver medals, and bronze medals, respectively. In our model, a variable returns to scale model was used with the output orientation.

A common issue occurring in real-world DEA applications is that often all the input and output measures may not be equally important to the decision makers. This study reports an application whereby, in a DEA model, weight restrictions are imposed on the three outputs of the model to overcome the above problem. A weight-restricted DEA model is thus constructed by introducing a discrimination criterion between the several efficient units.

Regarding the results obtained from this study, out of the 80 countries participating in this event, only 15 countries are shown to be efficient, with some countries being completely differently ranked by the IMO organization and our DEA model. The efficient DMUs that define a target are peers or references that can help to improve the whole efficiency of the competition. These measures describe the countries’ potential to generate a number of participants (mathletes) in the IMOs and are often considered as the most important factors explaining both sport and cognitive performance.

Future work for this research can focus more on the measures utilized and aim to investigate potential resources and outputs that would cover a wider range of measures that are more representative of the efforts put in and the results obtained. Moreover, the aim is to provide future researchers with some testimony on how these non-athletic mega-events can provide useful insights into the management of the mega-events by comparing and contrasting similarities and differences. Finally, regarding the comparison of the rankings produced by the present study and other international surveys, we strongly believe that the results of this study can add to the information about international performances in order to drive up education standards to a higher level.