Relative E ﬃ ciency of Canadian Banks: A Three-Stage Network Bootstrap DEA

: In this study, we focus on how banks can enhance their e ﬃ ciency in the utilization of resources to ensure their economic sustainability. We propose a novel three-stage (production, investment, and revenue generation) network Data Envelopment Analysis (DEA) with bootstrapping to evaluate the performance of the six big Canadian banks for the period 2000–2017, amid the 2007 ﬁnancial crisis and the increasing competition level due to new technologies. We identify the best practices in each stage that can be used as benchmarks by other banks to improve their economic sustainability. Our results indicate that the 2007 ﬁnancial crisis resulted in lower e ﬃ ciencies in the performance of Canadian banks. This decline was not substantial for the production and investment stages when the revenue generation stage received the greatest hit. In addition, we observed that the individual banks did not have consistent performance in the di ﬀ erent stages. Finally, we compared our model with the black box DEA model and concluded that the network DEA provides more insightful and accurate results in terms of banks’ e ﬃ ciencies.


Introduction
Finanancial services firms are an integral and important part of the economic systems of every country. They include banks (the largest proportion), securities brokers, insurance, mutual fund, and credit card companies (Melnick et al. 2012).
In Canada, the banking system plays a major role in the economy. By the numbers, in 2017, the Canadian banking system represented 86 banks, 5907 branches, 18,640 ATM, 275,825 employees in Canada, and 119,885 employees overseas. In the same year, banks accounted for 3.3% of the Gross Domestic Product (GDP) of the country. The top six banks paid 12.2 billion of CAD$ in taxes in Canada and 18.2 billion of CAD$ overseas (Canadian Bankers Association 2018). In addition to their direct contributions, banks have a more remarkable impact on the GDP through their financing activities. For example, in 2017, more than 1 million of financing relationships between banks and Small and Medium-sized Enterprises (SME) have been reported (Canadian Bankers Association 2018).
Canadian banks faced various challenges since the beginning of the millennium. First, technology is increasingly important in the process of creating value for their customers. Customers increasingly favor the use of technological outlets for their transactions. Hence, in 2017, they have completed 643 million of them through ATMs, 574 million through online banking, and 398 million through mobile banking. There are, therefore, strong competitive forces requiring that banks constantly innovate their customer relationship management using information technology (Canadian Bankers Association 2014). On the other hand, technology has significantly lowered the barriers to enter the industry and allowed several new competitors to enter the market, especially online banks.

Literature Review
The efficiency measurement of the banking systems, as a pillar of the modern economy, has long been of interest to researches. Accordingly, several approaches, such as accounting-based measurements (Fixler and Zieschang 1993) and the stochastic frontier approach (Weill 2004), have been implemented to assess the efficiency of the banking sector.
DEA is a non-parametric approach that allows for measuring the efficiency of DMUs considering several inputs and outputs. Various advantages of DEA have led to its wide applications in different industries (Emrouznejad and Yang 2018), including the banking sector. The first application of DEA in the banking sector can be traced back to the study of Sherman and Gold (1985). Since then, a considerable number of studies have examined the efficiency of the banking sector using DEA. See the studies of Kaffash and Marra (2017), Paradi and Zhu (2013), and Fethi and Pasiouras (2010) for recent reviews in the applications of DEA in the banking sector.

Network DEA in the Banking Sector
The banking process consists of several interconnected subprocesses, and classic black box DEA models, due to their holistic approach, cannot be used to analyze these subprocess. Therefore, researchers have implemented various extensions of DEA, such as network DEA, to investigate the performance of different aspects of banking systems. In the network DEA, the internal structure of DMUs is segmented into connected sub-DMUs, and the efficiency is calculated for each sub-DMU (Färe et al. 2007).
Some of the studies focused on two-stage DEA models in which deposits is the output of the first stage and the input of the second stage and investigated how this approach affects the measurement of efficiencies. Holod and Lewis (2011) showed that having deposits as the connecting factor of the first and second stages substantially impacts the efficiencies. Fukuyama and Matousek (2011) concluded a similar result by evaluating the efficiencies of the Turkish banking industry. By evaluating the efficiencies of Chinese banks, Wang et al. (2014a) also found a discrepancy between the results of the network DEA and the traditional back-box DEA.
A portion of the pertinent literature is devoted to exploring the role of risk in the efficiency of banks. Fukuyama and Matousek (2017) incorporated non-performing loans as a measure of risk in their model. They applied their model to the Japanese banking sector and indicated that the studied banks have gradually reduced their risk between 2000 to 2013. Similarly, Ebrahimnejad et al. (2014) considered non-performing loans as a measure of risk and evaluated the performance of the branches of a local bank in the US. Matthews (2013) interviewed 24 banking executives to identify the risks that the banking industry is facing and, accordingly, developed some indexes to measure the risk. Furthermore, they incorporated these indexes in their DEA model and concluded that, in this approach, return on investment can be considered as a proxy for the measured efficiencies.
Several studies concentrated on the evaluation of the marketability and profitability of banks. Seiford and Zhu (1999) argued that most of the studies in assessing bank efficiency focus on the operational aspect of the banks and ignore their marketing side. Hence, they divided the banking process into profitability and marketability stages and applied their model to analyze the efficiency of 55 banks in the United States. Their results indicated that large banks have lower performance in terms of marketability. Avkiran (2009) divided the bank production system into three parallel sub-DMUs: profit centre loans and overdrafts, mortgaged real estate loans, and discounted commercial bills. It was demonstrated that the results of the black box DEA model are not necessarily aligned with the network DEA model. Yang and Liu (2012) investigated the productivity and profitability of Taiwanese banks using fuzzy logic and concluded that profitability plays a greater role in the overall performance of the banks in comparison to productivity. Wang et al. (2014b) extended the previous study by considering the value creation process and similarly applied fuzzy logic to simplify the proposed model. They showed that intellectual properties have a significant impact on the value of the banks. Lin and Chiu (2013), in addition to profitability, included corporate and consumer banking to their model. The results showed that mergers and acquisition activities could enhance the performance and profitability of banks. By focusing on cost and productivity efficiencies, Wanke and Barros (2014) also concluded that mergers and acquisitions could reduce the costs and increase profitability.
Some researchers extended the models to the dynamic case. In this situation, some of the outputs in one period are the inputs for the next period. Akther et al. (2013) developed a two-stage dynamic network DEA model to evaluate the efficiency of the Bangladesh banking system. They considered non-performing loans generated in one period as an input for the next period. It was demonstrated that the results of their model are not aligned with the black box DEA model. Fukuyama and Weber (2015) extended the previous study by considering carry-over assets as an intermediate product between the periods. They applied their model to the Japanese banks and concluded that the performance of the banks could be improved by more effective management of deposits. Like previous studies, Avkiran (2015) developed a two-stage dynamic network DEA model. They included the number of referrals as one of the intermediate products between the periods and used robustness tests to examine the efficiency measurements better. They applied the model to 48 Chinese banks and concluded that there is no significant difference between the efficiency of foreign and domestic banks. Chao et al. (2015) considered the capability, efficiency, and profitability as three stages of the banking process. They assumed that the non-performing loans and loan loss reserves connect the periods. The application of the model to the Taiwan banking sector for the period 2005-2011 indicated that the profitability of the banks has been decreasing, due to the financial crisis.

Applications of DEA in the Canadian Banking Sector
The Banking sector has a pivotal role in Canada's economy and, accordingly, has been of interest to several researchers. Various studies investigated the application of DEA models in Canadian banks. Asmild et al. (2004) focused on five major Canadian banks for the period 1982-2000. They considered a production model where the deposits are considered as an output. They indicated that the two recessions that Canada faced during this period significantly impacted the efficiency of the banks. Paradi and Schaffnit (2004) investigated the performance of the branches of one large Canadian bank.
They also applied the production model and concluded that focusing on efficient branches can help the managers to grow their business. Yang (2009) also evaluated the branches of one Canadian bank using a production model. They indicated that branches should reform their processes to achieve efficiency. Paradi et al. (2012) incorporated a novel grouping approach with DEA to assess the efficiency of branches of a Canadian bank. The results indicated that the grouping approach could facilitate the identification of deficiencies. In another study, Paradi et al. (2015) developed two models to evaluate staff allocation and customer satisfaction. They applied their models to the branches of a large Canadian bank. By comparing the DEA scores with the bank's internal metrics, it was concluded that the DEA measurements are reliable. Paradi et al. (2011) developed three models (production model, intermediary model, and profitability model) to evaluate different aspects of a bank's branch operations. Afterward, they proposed a non-parametric index approach to aggregate the results of the models. They showed that their approach provides a more comprehensive insight into the performance of branches.
Our contribution to the literature is four-fold. First, we extended the current network DEA models by considering the revenue generation stage. This stage is related to the portfolio selection and pricing strategies of the banks. Despite the importance of this stage (Gambacorta 2008;DeYoung and Rice 2004), it has been either ignored (Fukuyama and Weber 2015;Akther et al. 2013;Fukuyama and Matousek 2011;Holod and Lewis 2011) or combined with other stages (Wang et al. 2014a;Ebrahimnejad et al. 2014;Lin and Chiu 2013;Yang and Liu 2012). In our study, we consider revenue generation as a separate stage and evaluate the efficiency of Canadian banks in this stage. Second, a glance at the pertinent literature reveals that despite the advantages of network DEA in evaluating the efficiency of the banks (Avkiran 2009), this approach has not been used in assessing the performance of Canadian banks. In our study, we implement network DEA to provide a better understanding of the performance of the Canadian banking industry. Third, most of the studies that have implemented DEA to evaluate the efficiency of Canadian banks focused on the operations of branches. The only study that has investigated the operations of the Canadian banks from the corporate level is the study of Asmild et al. (2004). In this study, we incorporate more recent data to elaborate on the recent changes in Canada's banking sector, and, especially, we evaluate the effect of the 2007 financial crisis on Canadian banks. Fourth, in order to account for the uncertainty (stochastic error) inherent to the real world and obtain robust and consistent efficiency estimators, we performed the bootstrapping of our efficiency ratios with bootstrap DEA Wilson 1998, 2000). No study has used bootstrapping in the performance assessment of Canadian banks.

Black Box CCR and BCC Models
Data Envelopment Analysis (DEA) is certainly one of the most prolific decision-making techniques of the last four decades. This technique has been used to measure the performance or relative efficiency of private and public organizations in almost all sectors, including in manufacturing (Wahab et al. 2008), logistics (Xu et al. 2009), telecommunication (Cooper et al. 2001), mining and oil and gas production (Dia et al. 2019a(Dia et al. , 2019b, and healthcare (Jacobs 2001). A recent survey of DEA applications can be found in Emrouznejad and Yang (2018).
The first DEA models, and by far the most used ones, so far, are the CCR (for Charnes, Cooper and Rhodes, the initials of the authors) (Charnes et al. 1978) and BCC (for Banker, Charnes and Cooper, the initials of the authors) (Banker et al. 1984) models. The CCR model measures the overall efficiency while the BBC model measures the managerial efficiency. With these two models, it is possible to obtain the scale efficiency (Banker et al. 1984) by dividing the overall efficiency by the managerial efficiency.
The first DEA model considered is the CCR oriented input, and is presented as follows: In the above model, n is the number of DMUs, t is the number of outputs, m is the number of inputs, x is is the value of the input s for the DMU i , y ir is the value of the output r for the DMU i , h i is the efficiency score of the DMU i (generally, index 0 is used to indicate the DMU being assessed in the models), µ r is the relative importance of the output r, ν s is the relative importance of the input s, and ε is a small positive real number. The second DEA model is a variant of the preceding one, known as the BCC model, when variable return to scale is considered: The DEA models allow for the identification and the quantification of the sources of inefficiencies by transforming the CCR and BCC models into equivalent dual models. For instance, the dual model of the CCR oriented input is as follows: n j=1 λ j y rj − s + i = y rj 0 , r = 1, .., t where the parameters λ j (j = 1, . . . , n) in Equations (10) and (11) identify the benchmark DMUs and define an envelope for the evaluated DMU 0 ; θ 0 in Equation (9) is the efficiency ratio of the evaluated DMU 0 ; z 0 in Equations (9) and (10) indicates the proportion of inputs, for an inefficient DMU, needed to produce outputs equivalent to its benchmark DMUs; and s i − and s r + in Equations (9)-(11) correspond to the slacks associated with the inputs i and the outputs r, respectively. Similarly, the dual of the BCC oriented input model is obtained by adding the following convexity constraint to the CCR dual oriented input model specified in Equations (9)-(11): The efficiency obtained from the CCR model can be further decomposed into efficiency related to the management effectiveness and the scale of operations. Since the BCC model computes the efficiency driven by the management effectiveness, we can evaluate the scale efficiency (SE) (Banker et al. 1984) as the ratio of the overall technical efficiency (computed by the CCR model) to the pure technical efficiency (computed by the BCC model), as in Equation (13).
The scale efficiency in Equation (13) evaluates if the DMU has the optimum scale size or, in other words, if it has the right amount of resources to operate.

DEA with Bootstrapping
The main limitation of the DEA methodology is that it is deterministic and does not consider the uncertainties characterizing the real world problems. Indeed, past studies focusing on the statistical properties of the conventional point estimators obtained through DEA have found that these estimators are not consistent efficiency estimators (Toma et al. 2017). Bootstrapping methodologies are well known and frequently used statistical resampling tools useful when one needs to perform statistical inferences for complex problems. Following Simar and Wilson (1998) approach, one can generate robust efficiency scores and build their confidence intervals (Simar and Wilson 2000). The bootstrap DEA consists in approximating the sampling distributions of the efficiency estimator, using a Monte Carlo resampling method to simulate the Data Generating Process (DGP). One can then calculate the robust estimators of the original unknown sampling distribution (Toma et al. 2017). The crucial step in the process lies in the proper specification of the DGP underlying the observed data.
For our previously presented DEA models, the DGP, P, generates a random sample, χ = (x ij , y rj )/i = 1, . . . , m; r = 1, . . . , t; j = 1, . . . n , to estimate their relative efficiencies θ j ( j = 1, . . . , n). An estimator,P, of the actual and unknown Data Generating Process (DGP) is then generated using the bootstrap procedure. A new dataset . . n} is constructed using the population obtained from the efficiency estimates. This dataset χ * defines the correspondinĝ x * andŷ * , whose distributions are known, given thatP is known. Using a Monte Carlo approximation in the analytical computation ofP, a collection of B pseudo-samples, χ * b (b = 1, . . . , B), are generated to produce the pseudo-estimates of the relative efficienciesθ b . Moreover, a kernel density estimation algorithm is used to ensure the generation of consistent bootstrap estimatesθ * b (b = 1, . . . , B) (Simar and Wilson 1998). Thus, we obtain a better estimation of the true efficiency scores, as compared to the regular DEA efficiency ones.
In our study, the bootstrap DEA efficiency scores are computed by running 2000 iterations of this procedure, which ensures enough convergence of the confidence intervals.

Our Three-Stage Network DEA Model
The controversial role of deposits in the banking process has long been a debated matter in the pertinent literature. This issue has resulted in extensive applications of two-stage network DEA in the measurement of the banking sector efficiency (Wang et al. 2014a). In these models, in the first stage, the resources are utilized to generate deposits, which, in turn, will be used to generate loans and securities or incomes (Fukuyama and Matousek 2017;Fukuyama and Weber 2015;Wang et al. 2014a;Akther et al. 2013;Holod and Lewis 2011).
One essential aspect of banking operations is how banks set interest rates and select their securities to create revenue. This process directly affects the profit and the performance of the banks (Gambacorta 2008;DeYoung and Rice 2004). Nevertheless, most of the studies in the literature have either ignored this process (Fukuyama and Weber 2015;Akther et al. 2013;Fukuyama and Matousek 2011;Holod and Lewis 2011) or combined it with the second stage (Wang et al. 2014a;Ebrahimnejad et al. 2014;Lin and Chiu 2013;Yang and Liu 2012).
In this study, in order to capture different aspects of the banking operations, we consider three stages: the production stage, intermediation stage, and revenue generation stage. In the production stage, the bank uses its resources to generate deposits, which is the lone output of this stage. We consider the total assets, number of employees, and non-operating costs as the inputs of the first stage. In the intermediation stage, the bank uses the deposits to generate loans and securities. One important aspect of the second stage is the risk that comes with impaired loans. It is one of the main responsibilities of the banks to oversee and monitor impaired loans to ensure the sustainability of the system (Office of Superintendent of Financial Institutions 2010). Hence, in our model, we consider impaired loans as an undesired output of the second stage, and we applied an inverse transformation to it as the output to consider (Seiford and Zhu 2002) in this stage. In the revenue generation stage, we examine the efficiency of the banks' investment strategies and interest fee setting policies. In this stage, the outputs are interest incomes and non-interest incomes. Figure 1 shows the inputs and outputs of each stage.
One essential aspect of banking operations is how banks set interest rates and select their securities to create revenue. This process directly affects the profit and the performance of the banks (Gambacorta 2008;DeYoung and Rice 2004). Nevertheless, most of the studies in the literature have either ignored this process (Fukuyama and Weber 2015;Akther et al. 2013;Fukuyama and Matousek 2011;Holod and Lewis 2011) or combined it with the second stage (Wang et al. 2014a;Ebrahimnejad et al. 2014;Lin and Chiu 2013;Yang and Liu 2012).
In this study, in order to capture different aspects of the banking operations, we consider three stages: the production stage, intermediation stage, and revenue generation stage. In the production stage, the bank uses its resources to generate deposits, which is the lone output of this stage. We consider the total assets, number of employees, and non-operating costs as the inputs of the first stage. In the intermediation stage, the bank uses the deposits to generate loans and securities. One important aspect of the second stage is the risk that comes with impaired loans. It is one of the main responsibilities of the banks to oversee and monitor impaired loans to ensure the sustainability of the system (Office of Superintendent of Financial Institutions 2010). Hence, in our model, we consider impaired loans as an undesired output of the second stage, and we applied an inverse transformation to it as the output to consider (Seiford and Zhu 2002) in this stage. In the revenue generation stage, we examine the efficiency of the banks' investment strategies and interest fee setting policies. In this stage, the outputs are interest incomes and non-interest incomes. Figure 1 shows the inputs and outputs of each stage.

Sample and Data
By the numbers, in 2017, there were 86 domestic and foreign banks operating in Canada (Canadian Bankers Association 2018). In this study, we focused on the Canadian banks, which have over $100,000 million of total assets. These banks consist of Royal Bank of Canada (RBC), Toronto-Dominion Bank (TD), Bank of Nova Scotia (Scotia), Bank of Montreal (BMO), Canadian Imperial Bank of Commerce (CIBC), and National Bank of Canada (NBC). These top six banks, also known as the "big six," are classified as systematically important financial institutions by Canadian regulatory bodies. In addition, they control over 90% of banking assets and 75% of the assets of the deposit institutions in Canada (Gueyié et al. 2019;Mohsni and Otchere 2018). Therefore, due to their importance for the Canadian economy and to get meaningful results by reducing disparity in our sample (McMillan and Datta 1998), we focus on these six banks. We gathered the data for the years 2000-2017 (108 DMUs in each stage) from the annual reports and other public financial reports. This set of data gives us a better understanding of the past two decades' major trends and phenomena in the Canadian banking industry. Table 1 provides a summary of the descriptive statistics of the inputs and outputs for each bank.
A glance at Table 1 reveals the different policies that the Canadian banks have implemented to generate profit. For instance, Scotia has a relatively small proportion of securities over loans. This demonstrates that the income of the Scotia is more reliant on loans rather than securities. This policy

Sample and Data
By the numbers, in 2017, there were 86 domestic and foreign banks operating in Canada (Canadian Bankers Association 2018). In this study, we focused on the Canadian banks, which have over $100,000 million of total assets. These banks consist of Royal Bank of Canada (RBC), Toronto-Dominion Bank (TD), Bank of Nova Scotia (Scotia), Bank of Montreal (BMO), Canadian Imperial Bank of Commerce (CIBC), and National Bank of Canada (NBC). These top six banks, also known as the "big six," are classified as systematically important financial institutions by Canadian regulatory bodies. In addition, they control over 90% of banking assets and 75% of the assets of the deposit institutions in Canada (Gueyié et al. 2019;Mohsni and Otchere 2018). Therefore, due to their importance for the Canadian economy and to get meaningful results by reducing disparity in our sample (McMillan and Datta 1998), we focus on these six banks. We gathered the data for the years 2000-2017 (108 DMUs in each stage) from the annual reports and other public financial reports. This set of data gives us a better understanding of the past two decades' major trends and phenomena in the Canadian banking industry. Table 1 provides a summary of the descriptive statistics of the inputs and outputs for each bank.
A glance at Table 1 reveals the different policies that the Canadian banks have implemented to generate profit. For instance, Scotia has a relatively small proportion of securities over loans. This demonstrates that the income of the Scotia is more reliant on loans rather than securities. This policy has resulted in a large amount of impaired loans for this bank. Despite the smaller size of Scotia relative to RBC and TD in terms of total assets, this bank has a larger amount of impaired loans.
The other important observation is the inconsistency between the total assets and the operating costs. For instance, although in terms of total assets BMO is a larger bank compared to CIBC, it has a lower number of employees and other operating costs.

Analysis of Results
Using the collected data, we have run the Bootstrap three-stage DEA models to measure the efficiencies. In total, we have considered nine inputs and outputs, the triple of which is 27. We define a DMU as a bank considered in a given year. This resulted in 108 DMUs. Therefore, our DEA sample satisfies the rule of thumb proposed by Sarkis (2007) to achieve acceptable results.
For each stage, we calculated the bootstrap overall technical efficiency (OTE), the bootstrap pure technical efficiency (PTE), and the scale efficiency (SE), as presented in Section 3. Please note that we measured the SE using the OTE and PTE from non-bootstrap models. In addition, we measured the overall efficiencies by applying a multiplicative approach. In this approach, for each DMU, the multiplicative aggregate efficiency is computed by multiplying its efficiency scores calculated individually in the three stages.

Annual Evaluation of Efficiencies
In this section, we analyze the behaviour of efficiencies for the years 2000-2017. Tables 2-4 provide the calculated OTEs, PTEs, and the SEs for the first, second, and third stages, respectively The first observation from Tables 2 and 3 is that the banking sector of Canada performed well in terms of the overall technical efficiency and the pure technical efficiency in the first and second stages. On average, the overall and pure technical efficiencies of the banks in the period 2000 to 2017 are 0.828 and 0.88 for the first stage, and 0.812 and 0.879 for the second stage. Besides, in the first two stages, the OTE is always above 0.74, and the PTE is above 0.8. In the third stage, the average OTE and PTE are relatively lower and are equal to 0.739 and 0.77, respectively (see Table 4).   The measured scale efficiencies for three stages suggest that the main source of inefficiency in the sector is the managerial issues rather than scale issues. Figures 2-4  The measured scale efficiencies for three stages suggest that the main source of inefficiency in the sector is the managerial issues rather than scale issues. Figures 2-4 show the variations of OTE, PTE, and SE for the period 2000 to 2017.   From Figure 2, it can be observed that, for the period 2000 to 2007, the OTE and the PTE are gradually decreasing. This shows that, during this period, the Canadian banks were gradually losing their ability to turn their resources into deposits. The data from the Office of the Superintendent of Financial Institutions Canada show that, during this period, the assets to capital ratio for Canadian banks was increasing (Office of Superintendent of Financial Institutions 2019), which results in lower efficiency in the first stage. The measured scale efficiencies for three stages suggest that the main source of inefficiency in the sector is the managerial issues rather than scale issues. Figures 2-4 show the variations of OTE, PTE, and SE for the period 2000 to 2017.   From Figure 2, it can be observed that, for the period 2000 to 2007, the OTE and the PTE are gradually decreasing. This shows that, during this period, the Canadian banks were gradually losing their ability to turn their resources into deposits. The data from the Office of the Superintendent of Financial Institutions Canada show that, during this period, the assets to capital ratio for Canadian banks was increasing (Office of Superintendent of Financial Institutions 2019), which results in lower efficiency in the first stage. The measured scale efficiencies for three stages suggest that the main source of inefficiency in the sector is the managerial issues rather than scale issues. Figures 2-4 show the variations of OTE, PTE, and SE for the period 2000 to 2017.   From Figure 2, it can be observed that, for the period 2000 to 2007, the OTE and the PTE are gradually decreasing. This shows that, during this period, the Canadian banks were gradually losing their ability to turn their resources into deposits. The data from the Office of the Superintendent of Financial Institutions Canada show that, during this period, the assets to capital ratio for Canadian banks was increasing (Office of Superintendent of Financial Institutions 2019), which results in lower efficiency in the first stage. From Figure 2, it can be observed that, for the period 2000 to 2007, the OTE and the PTE are gradually decreasing. This shows that, during this period, the Canadian banks were gradually losing their ability to turn their resources into deposits. The data from the Office of the Superintendent of Financial Institutions Canada show that, during this period, the assets to capital ratio for Canadian banks was increasing (Office of Superintendent of Financial Institutions 2019), which results in lower efficiency in the first stage. Figures 3 and 4 demonstrate that the average efficiency of the Canadian banks for the investment and revenue generation stages was increasing before the 2007 financial crisis. In the period 2002 to 2007, the demand for mortgages increased, which resulted in higher mortgage interest rates (Bank of Canada 2019). Hence, in this period, we observe increasing trends in the efficiencies for the second and the third stages.
In the period 2007 to 2011, we can observe a swift decline in the OTE and PTE for all three stages. This behaviour can be traced back to the 2007 financial crisis. Nevertheless, the decline in the measured efficiencies is not substantial. In the studied period, neither OTE nor PTE has ever gone below 0.74 for the first two stages and below 0.59 for the third stage. This pattern can be associated with the resilience of the Canadian banking system. In fact, according to a recent survey by the World Economic Forum, Canada has one of the most stable banking systems in the world (World Economic Forum 2018). Another explanatory element that describes the relatively good performance of the Canadian banks during the 2007 financial crisis is the high reliance of Canadian banks on retail and commercial deposits in comparison to American and European counterparts (Gauthier and Tomura 2011). Retail and Commercial deposits provide a more stable source of funds for the banks and, consequently, impose less risk during financial crises (Truno et al. 2017). Moreover, researchers have suggested other elements, such as stiff government supervision and regulations and a centralized structure of the Canadian banking system, as important factors that helped the Canadian banking sector to have a smooth transition during the 2007 financial crisis (Seccareccia 2012).
After 2011, we observe a gradual increase in the OTE and PTE of all stages. In fact, in the first stage, the OTEs and the PTEs for 2015, 2016, and 2017 are even higher than similar measures in 2000. Most of the banks responded to the financial crisis by reducing their operating and non-operating costs (Tsionas and Mamatzakis 2017). Canadian banks also followed a similar strategy that resulted in higher efficiencies in the first stage after 2011.
We observe similar behaviour for the PTE of the second stage. The PTE in the last three years is increasing and is higher than the PTE in 2000.
According to Figure 4, the revenue generation stage of the banks is also gently recovering. After 2010, we can observe a gradual increase in the OTE and PTE of the final stage. In fact, after 2011, the mortgage rate was relatively constant (Statistics Canada 2019); however, the decrease in the amount of impaired loans and the higher reliance of the Canadian banks on non-interest incomes, such as deposit and transfer fees (Seccareccia 2012), have resulted in higher efficiency in the last stage. In fact, the share of non-interest incomes in the total income of the studied banks has increased from 22% in 2007 to 39% in 2017.
In order to develop a better understanding of the variation of efficiency in different years, we conducted pairwise t-tests and compared the OTE, PTE, and SE of the studied years for all three stages. The results of this comparison are presented in Table 5. Please note that, in this table, we have only reported the p-values for the years with a significant difference at 1%, 5%, and 10%.
According to Table 5, in the first stage, although we observe a swift decrease in overall technical efficiency in the period 2009 to 2011, this decrease is not statistically significant. We have a similar situation for SE. In addition, the PTE during the financial crisis does not show a statistically significant difference to the years prior to the financial crisis. As mentioned before, this behaviour can be traced back to the resilience of the Canadian banking system. However, the improvement in PTE after the financial crisis is significant. It can be observed that the PTEs of 2008, 2010, and 2011 are significantly different from the years 2015, 2016, and 2017. After 2011, most of the banks around the world tried to reduce their costs to become more resilient to the fluctuations of the financial markets, and Canadian banks were no exception to this (Tsionas and Mamatzakis 2017). Table 5 reveals that Canadian banks were indeed successful in this process. In the second stage, the big six Canadian banks demonstrate a stable performance. We did not observe any statistically significant difference between the OTEs, PTEs, and SEs prior, during, or after the financial crisis. According to the results, the financial crisis did not substantially impair the loaning and investment processes of the big six Canadian banks (and therefore, no number is reported in Table 5 for Stage 2). This shows that the Canadian banks have not made a drastic change in the rate of turning deposits into loans and securities, and although the lower rate of impaired loans has caused an increase in the efficiencies, this increase is not statistically significant.
The third stage indicates the largest variations in efficiency from 2000 to 2017. According to Table 5, during the financial crisis, the efficiency of Canadian banks' revenue generation stage dropped significantly. The OTEs and PTEs during the financial crisis are significantly different from the years 2000 and 2001, in which the banks had the highest efficiencies. In addition, the performance of the banks before the financial crisis is significantly different from the years 2010 and 2011. These observations are due to the fact that, before the financial crisis, the banks were benefiting from a high mortgage demand and high mortgage rates, and the loss of this source of income resulted in significantly lower efficiencies in Stage 3 after the financial crisis.

Evaluation of the Banks' Efficiencies
In this section, we analyze and compare the efficiencies of the big six Canadian banks. Table 6 provides the OTE, PTE, and the SE of the banks for all three stages.
The first observation from Table 6 is that the six big Canadian banks, on average, have a lower performance in the second stage in terms of overall technical efficiency. The average OTE for the first and the third stages are 0.837 and 0.915, while this number for the second stage is 0.791. In terms of pure technical efficiency, the banks have the best performance in the third stage. The average PTE for the first, second, and third stages are 0.880, 0.879, and 0.948, respectively. Our second observation is that the standard deviations of the efficiencies for all the stages are relatively low. In the first and the second stages, the highest coefficient of variation for OTE, PTE, and SE is 0.118. In the third stage, we have a larger variation in the measured efficiencies; nevertheless, the coefficients of variation are still relatively low. The highest coefficient of variation in this stage is 0.248. This observation shows that due to the high centralization of the Canadian banking system and the higher level supervision of the government, Canadian banks are remarkably resilient to the fluctuations of the financial markets (World Economic Forum 2018).
Finally, we can see that the banks do not show consistent performance in all stages. For instance, NBC, despite having the highest OTE and PTE in the second stage, has the lowest OTE in the third stage. Figures 5-7 show the efficiencies of the banks in all three stages.
In the first stage, in terms of overall technical efficiency, CIBC has the best performance. Interestingly, although NBC has the lowest OTE, it has the highest pure technical efficiency. This observation shows that managerial issues play a greater role in NBC in comparison to other banks.
In the second stage, NBC is the most efficient bank in terms of overall technical efficiency and pure technical efficiency. In fact, by comparing the results with Table 1, we can see that NBC has a very low rate of impaired loans. This indicates that NBC has a proper process of risk measurement in their loans, and that is one of the main reasons they have high efficiency in the second stage. In addition, the results indicate that, in the second stage, the larger banks in terms of total assets have a lower performance. In terms of overall technical efficiency and scale efficiency, TD has the lowest score. Moreover, RBC is in the fifth position in terms of OTE, PTE, and SE.
Although RBC does not have a good performance in the second stage, it has the highest OTE and PTE in the third stage, followed by Scotia Bank. Despite the good performance of NBC in the second stage, it has the lowest OTE and SE. In terms of pure technical efficiency, BMO is in last place. Besides, note that although Scotia Bank does not have the highest OTE or PTE in any of the stages, it performs consistently well in all stages. Scotia Bank is in the second position in all the stages in terms of overall and pure technical efficiencies.
By comparing the results in the second and the third stages, we can observe the importance of considering revenue generation as a separate stage. Performing well in terms of the level of loans and investments does not necessarily translate to higher income. For instance, RBC and NBC have a substantially inconsistent performance in these two stages. It can be concluded that separating these two stages can provide a better insight about the efficiencies of the banks. J. Risk Financial Manag. 2020, 13, x FOR PEER REVIEW 19 of 26 By comparing the results in the second and the third stages, we can observe the importance of considering revenue generation as a separate stage. Performing well in terms of the level of loans and investments does not necessarily translate to higher income. For instance, RBC and NBC have a substantially inconsistent performance in these two stages. It can be concluded that separating these two stages can provide a better insight about the efficiencies of the banks.   We also conducted a pairwise t-test to compare the efficiencies of the banks in the first, second, and third stages. Table 7 presents the results of these tests. In this table, the first column shows the pair of banks that are compared, the second column shows the type of efficiency that is calculated, and the p-value of the t-test is reported in the third column. Please note that, in this table, we only report the p-values for the tests that are statistically significant at 1%, 5%, and 10%.  We also conducted a pairwise t-test to compare the efficiencies of the banks in the first, second, and third stages. Table 7 presents the results of these tests. In this table, the first column shows the pair of banks that are compared, the second column shows the type of efficiency that is calculated, and the p-value of the t-test is reported in the third column. Please note that, in this table, we only report the p-values for the tests that are statistically significant at 1%, 5%, and 10%. In the first stage, the difference between the overall technical efficiency of NBC and the other banks is statistically significant. This result suggests that NBC has a problem in attracting deposits. However, this problem is mainly due to managerial issues rather than the smaller size of NBC. This fact can be concluded from the higher PTE of NBC in comparison to other banks. CIBC has the highest OTE, and it is significantly higher than RBC's, TD's, and Scotia's OTE. One of the reasons for this observation is the proficiency of CIBC in online banking and digital services, which has resulted in lower costs for CIBC while maintaining a high level of deposits. According to Global Finance magazine, in 2019, CIBC was the best consumer digital bank in North America. In this stage, in terms of pure technical efficiency, NBC has the highest score. In fact, the difference between NBC's PTE and the other banks' PTE is statistically significant. One important factor that has resulted in high NBC's PTE is the relatively smaller size of NBC in terms of assets. The substantial difference between the performance of NBC in terms of OTE and PTE has resulted in a statistically significant lower performance of this bank in terms of scale efficiency.
In the second stage, NBC is the most efficient bank in terms of overall technical efficiency, and the difference between NBC and other banks is statistically significant. NBC also has the highest score in terms of pure technical efficiency and scale efficiency. In terms of PTE, the difference between NBC and Scotia and BMO is not statistically significant; however, in terms of SE, NBC has a significantly higher score compared to other banks. These observations indicate that NBC has an efficient risk management process. In fact, the percentage of impaired loans for NBC is 0.5%, which is the lowest among the studied banks. The percentage of impaired loans for the rest of the banks, on average, is 0.9%.
RBC has the best performance in terms of overall and technical efficiencies in the last stage. In fact, in terms of OTE, Scotia is the only bank that has difference with RBC that is not significant. In terms of PTE, RBC has a significantly higher efficiency in comparison to TD, CIBC, and BMO. In terms of scale efficiency, the top five Canadian banks do not show a significant difference in their efficiencies. Nevertheless, NBC has a significantly lower efficiency compared to these banks.

Network DEA vs. Black Box DEA
The network DEA provides a clear view of how different banks operate in different stages. Hence, banks can identify the operations that create inefficiencies in their system and implement more precise improvement strategies. In this section, we investigate the difference between the overall efficiencies measured by network DEA and Black box DEA. In the network DEA, we utilized a multiplicative approach to measure the overall efficiencies of the banks. In this approach, the overall efficiency is equal to the product of efficiencies in the three stages. In addition, in the black box model, we assumed that the banking process has one stage in which the total assets, number of employees, and the other operating costs are the inputs, and the interest and non-interest incomes are the outputs. The average values of the efficiency scores obtained from both DEA models over the study period (2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015)(2016)(2017) are presented in Table 8. A quick glance at Table 8 reveals that the calculated efficiencies in the network DEA are relatively lower than that of the black box model. This can be associated with the multiplicative approach that we have implemented to calculate the overall efficiencies in the network DEA.
However, it can be observed that the two models rank the banks differently in terms of their efficiencies. The results of the network DEA suggest that Scotia has the highest overall OTE, while black box model recommends RBC as the most efficient bank. We also see a discrepancy between the results of network and black box DEA in terms of PTE. For instance, while in the network DEA Scotia has the second-highest score, black box DEA suggests that Scotia has the fifth highest score. The results are even more divergent in terms of scale efficiency. The only alignment between the results is NBC, which is ranked last by both models.
The results of black box DEA is significantly affected by the revenue generation stage. In terms of OTE, the results of black box DEA and the revenue generation stage are completely aligned. This observation can be due to the importance of interest and non-interest incomes in black box DEA and the revenue generation stage. In addition, it shows how other aspects of the banking process are ignored in the black box DEA. For instance, the production stage can be significantly impacted by the digitization of the processes, which is an important facet of modern businesses. Nevertheless, black box DEA does not provide a clear picture of this aspect of the banking process.

Conclusions
As the pillar of the modern economy, the banking system has been of interest to many researchers. In this study, we focused on the efficiencies of the big six Canadian banks in the period of 2000 to 2017. We proposed a new three-stage network DEA model with bootstrapping to measure the efficiencies of the banks and identify the best practices in the industry that can be used as benchmarks to achieve economic sustainability. In this model, in addition to the production and investment stages, we considered the revenue generation stage as an important element of the banking process. Our results show that the big six Canadian banks have relatively high efficiency in the studied period for all the stages. The 2007 financial crisis has resulted in a decline in the efficiencies of the banks in all stages; however, our results suggest that this decline was not significant in the overall technical efficiency of the first stage and the overall and pure technical efficiencies of the second stage. This behavior can be traced back to the resilience of the Canadian banking system. The revenue generation stage of the Canadian banking sector received the biggest hit in the financial crisis. Nevertheless, after 2010, the big six Canadian banks demonstrated a steady recovery.
By comparing the efficiencies of the banks, we observed that the banks do not show a consistent performance in all the stages. In the production stage, CIBC has the highest OTE, while NBC has the highest PTE. In the investment stage, NBC is the most efficient bank in terms of both OTE and PTE. In the revenue generation stage, RBC has the highest OTE and PTE. Moreover, Scotia Bank shows the most consistent performance in all the stages by being ranked second in terms of OTE and PTE.
Finally, we compared the results of the network DEA with the black box DEA and concluded that not only the network DEA provides more insightful results, but the results between the network DEA and the black box DEA are not necessarily aligned. This observation is consistent with the findings of Wang et al. (2014b) and Fukuyama and Matousek (2011).
Our study has, however, a few limitations. The first one has to do with the choice of the input and output variables for the DEA models. Indeed, the results of the models are sensitive to these choices. An interested reader could refer to Botti et al. (2014) and Boubaker et al. (2018). Another limitation is the static nature of the DEA models. Extending the model into a dynamic multi-period setting can provide new insights into the efficiency of the Canadian banking system. In addition, in our study, we focused on the corporate level and did not evaluate the efficiencies of banks branches' efficiencies. Incorporating the model to measure the efficiencies of the branches can develop a better understanding of where the banks should focus on to improve their system.

Conflicts of Interest:
The authors declare no conflict of interest.