How to Assess Reverse Logistics of e-Waste Considering a Multicriteria Perspective? A Model Proposition

As e-waste exponentially increases on a global scale, some legislation and plans to implement reverse logistics have arisen in some countries. While reverse logistics is already well consolidated in developed countries, it is still in a state of infancy in developing countries. In this article, we propose a theoretical model to assess reverse logistics, based on a multicriteria decision aid perspective. This was done while considering a context in which the implementation of the reverse logistics of e-waste is at the beginning. To show the main characteristics of the literature and obtain the model’s inputs, we conducted a systematic literature review to help us understand the main criteria and methods used for this purpose. Then, we illustrated the use of the proposed model with a numerical example. This paper contributes to the current body of research, and aims to assist practitioners within the field of reverse logistics by systematizing the knowledge related to this issue, and offering a model that can guide the decision-making. This model differentiates from others already published because it uses the probabilistic composition of preferences (PCP) method, which does not require the assignment of weights. The model can also be used while considering the preferences of several decision-makers, therefore covering several points of view and providing a more holistic view of the decision problem.


Introduction
People from all over the world are increasingly being integrated into the information society and the digital economy, and benefiting from the opportunities arising from it. Examples of this include the delivery of new applications and services at very high speed in a wide range of areas from health and education to entertainment and commerce [1].
This integration is possible because of technological development and its consequent diffusion as a result of the efforts of large corporations to become global. Despite the undeniable advances that this brings, the same development is used as an instrument of consumption control. This can be observed in the increase in the consumption levels of durable goods such as electronic equipment [2].
As a result, the generation of residues from electro-electronic equipment (commonly referred to in the scientific literature as waste electrical electronic equipment (WEEE, or e-waste) has exponentially increased worldwide [3][4][5]. This specific type of waste corresponds to 5% of all urban waste produced

Multiple Criteria Decision Aid (MCDA)
The multicriteria decision aid approach aims to provide decision makers with some tools to enable them to move forward in solving decision problems, in which several-often contradictory-points of view must be taken into account [21].
A multicriteria decision problem consists of a situation where there are at least two alternatives of action to choose. The process of the choice of the most adequate alternatives or courses of actions occurs through the desire to meet multiple goals that often have conflicting relationships [21][22][23]. These objectives have associated variables that represent them and allow each alternative to be evaluated based on each objective, which can be called criteria, attributes, or dimensions [22]. To identify the system of preferences of a decision-maker, first it is necessary to consider the subjectivity of the decision-makers, the individual perceptions [21,24]; second, it is necessary to structure the problem according to the decision-maker or the group of decision makers' points of view [24,25]. It is important to point out that the multicriteria decision aid analysis process involves a few steps, which are presented below [24].

1.
Identification and structuring of the problem. Before any analysis begins, the various stakeholders including facilitators and technical analysts, need to develop a common understanding of the problem, the objectives, the decisions to be made, and the criteria by which those decisions are judged and evaluated;

2.
Construction and use of the model. A primary feature of the multicriteria approach is the development of formal models of decision-maker preferences, valuing trade-offs, goals, among others, to allow those alternatives and policies or actions under consideration to be compared and related to each systematically and transparently; and 3.
Development of action plans. The analysis does not solve the decision problem. All management science, and multicriteria decision analysis in particular, also concerns the implementation of results, translating analysis into specific action plans.
These steps should be considered to deal with the decision elements that involve the preference relations of the decision maker; the conflicting objectives; the criteria that represent them, when alternatives are evaluated; the rationality related to compensation or not between criteria, and the choice of multicriteria decision support approach to be used, which involves an extensive set of methods. These methods analyze the decision-making elements related to the consequences of the problem in question, providing an appraisal of the possible options considered in the decision-making process. Nevertheless, it is important to point out that there is no consensus amongst specialists in which the best method is to use, because this choice will depend on the features of the decision context approached [21,25]. Thus, the first movement is to know the characteristics of the decision problem and the preferences of decision-makers, to identify the elements of decision, and to make feasible the most appropriate method to choose.

Methods and Techniques
Considering the intention of analyzing how to assess the implementation of the reverse logistics of e-waste, considering a multiple criteria approach, we conducted two methodological procedures. First, to obtain the model's inputs, we conducted a systematic literature review (SLR) to understand the main characteristics of publications on this topic and to get the main criteria and proper methods to use in this context; and second, multicriteria modeling was used to propose the model, which was illustrated with a numerical example based on inputs from the SLR.
In the next sections, we present the two procedures used in this study. Section 3.1 presents the protocol and procedures followed to conduct the SLR and Section 3.2 describes the multicriteria modeling steps.

Systematic Literature Review
The literature reviews presented two perspectives: (i) revisions of topics that have accumulated knowledge and require analysis and synthesis; and (ii) revisions of emerging issues, whose contribution is the exposure of potential theoretical foundations, which is usually not as extensive as the first. This research fits the first perspective [26].
There are two types of literature review: (1) narrative or traditional literature reviews, and (2) a systematic review of the literature [27]. According to the authors, the narrative review condenses the literature, but does not make explicit to the reader the selection criteria of the sources used. On the other hand, the systematic literature review is characterized by well-defined steps and structured criteria in the procedure of searching through and analyzing the information related to the investigated matter.
This process guarantees greater rigor, robustness, and replicability to the research [28]. The rigor of this procedure originated from the medical area, in which it is extensively used [29,30]. The Cochrane Handbook was the first publication to formalize this methodology [29,30]. In the area of social sciences, the utilization of the procedure of SLR is considered a recent phenomenon [31].
For the reasons exposed, the procedure chosen for this research was a systematic literature review (SLR). It is also important to emphasize that the SLR considers only papers with empirical results, excluding papers of literature review from the total analyzed.
Due to the increase of the number of publications, concerns have been raised regarding the choice of the most relevant sources to be included in the research and the criteria to eliminate those that are not relevant [10]. There are several RS protocols of SLR. For this paper, the protocol Methodi Ordinatio was chosen [10]. This protocol was selected because it presents well defined steps and a more rigorous procedure to select the papers for a SLR. The authors proposed an adaptation of Proknow-C methodology [32], which is based on a multicriteria decision aid (MCDA) approach. The protocol considers three criteria in ranking articles: the year of publication, the number of citations obtained, and the impact factor of the journal in which it was published. The methodology aids the researchers to select, collect, rank, and systematically read scientific papers published in journals. The Methodi Ordinatio employs an equation to rank papers, the Index Ordinatio (InOrdinatio-IO) [10].
where -IF is the impact factor, which is divided by 1000 (one thousand), aiming to normalize its value concerning the other criteria; -α is a weighting factor ranging from 1 to 10, to be attributed by the researcher. The closer the number is to one, the lower the importance the researcher will attribute to the criterion year, while the closer to 10, the higher is the importance; -ResearchYear is the year in which the research was developed; -PublishYear is the year in which the paper was published, and -Ci is the number of times the paper has been cited [10].
After the calculation, the InOrdinatio of each paper is obtained, and from this point, it is possible to rank the papers according to their scientific relevance: the higher the InOrdinatio value, the more relevant the paper. Thus, when the ranking is provided, the researcher can define how many papers they will search for the full version, according to the priorities of the research, which will be analyzed.
The intention of the research was defined as: Analyzing the state-of-art of papers on the reverse logistics of electronic waste considering multiple criteria, published in last ten years, and the following aspects: university of origin; countries involved in research; partnerships between universities and other organizations; research methods; objectives; main criteria approached; utilization of methods from MCDA, PL, and other approaches; main results; the main concepts approached; and suggestions for future research.
A series of databases for research was highlighted: Web of Science, Scopus, Science Direct Online, Emerald, Springer Link, IEEExplore, Academic Search Premier-ASP (EBSCO), Scielo, Sage Pub, Taylor & Francis, Oxford University Press, and Cambridge University Press [33]. Three scientifical bases were selected for this SLR: Science Direct, Web of Science, and Emerald. The bases cover the main leading journals in the social sciences and are multidisciplinary in nature. Another reason for choosing the three bases was that most journals covered contain an impact factor in the Journal Citations Report (JCR ® ).
The keywords used in the search for papers were 'reverse logistics' AND 'electronic' OR 'e-waste' AND 'multiple criteria' OR 'multicriteria'. The keywords were searched in all fields of the paper (title, abstract, keywords and text). In both bases, the period of the publication of the papers defined was 2007-2017. Regarding the type of documents, it was limited to "full articles", published in journals, in the English language.
Considering the choice by the Methodi Ordinatio [10], only journals with impact factors were included in the systematic literature review, as proposed in this paper. Aside from that, the impact factor chosen was the Journal Citations Report (JCR ® ), which is a base from the Thomson Reuters Publisher. The JCR is a resource for evaluating and comparing scientific journals using citation data from academic and technical journals and their impact on the scientific community indexed by the Web of Science main collection. It reviews journals from 3300 publishers, about 200 disciplines, and 60 countries and, it is possible to check citation statistics from 2001 to the present. Considering the criteria of the inclusion of papers to be analyzed, it excluded papers published in the proceedings of events, chapters, patents and books, short survey, short communication, correspondence, letter, discussion, book review, product review, erratum, and editorial. Furthermore, papers published prior to January 2007 and subsequent to March 2017 were not considered. In addition, papers published in journals without JCR were also excluded.
Regarding the subject of the papers excluded, some of them deal with: the footwear industry; retail sector; packaging industry; pharmaceutical sector; food industry; integrated vehicle configuration system; green supplier selection; green supply chain management; eco-efficient supply chains; airline industry; environmental footprint; internet of things; supplier development; military aircraft industry; circular economy; construction industry; apparel industry; responsive supply chain; sustainable supply chain management; global agri-food supply chains; fashion retail supply chain; automotive industry; and healthcare quality innovation.
Considering the above-mentioned, subsequently, the duplicate checking of papers was carried out and these were also eliminated from the analysis.
It is important to emphasize that the JCR report of 2015 was considered related to the categories Sciences (SCIE) and Social Sciences (SSCI). Regarding the number of citations of each article, this was obtained from Google Scholar. Figure 1 illustrates the results found in terms of the quantities of papers found and the results after the filtering process and the application of Methodi Ordinatio.
Logistics 2020, 4, x FOR PEER REVIEW 6 of 31 Regarding the subject of the papers excluded, some of them deal with: the footwear industry; retail sector; packaging industry; pharmaceutical sector; food industry; integrated vehicle configuration system; green supplier selection; green supply chain management; eco-efficient supply chains; airline industry; environmental footprint; internet of things; supplier development; military aircraft industry; circular economy; construction industry; apparel industry; responsive supply chain; sustainable supply chain management; global agri-food supply chains; fashion retail supply chain; automotive industry; and healthcare quality innovation.
Considering the above-mentioned, subsequently, the duplicate checking of papers was carried out and these were also eliminated from the analysis.
It is important to emphasize that the JCR report of 2015 was considered related to the categories Sciences (SCIE) and Social Sciences (SSCI). Regarding the number of citations of each article, this was obtained from Google Scholar. Figure 1 illustrates the results found in terms of the quantities of papers found and the results after the filtering process and the application of Methodi Ordinatio. As can be noted in Figure 1, the systematic literature review covers two parts: Part 1 and Part 2. In Part 1, in Science Direct, with the terms reverse logistics AND electronic AND multiple criteria OR multicriteria, 307 results were found. In Web of Science, with the terms (TS = (reverse logistics AND electronics) AND TS = (multiple criteria OR multicriteria)), we found 23 results. In Emerald, with the terms reverse logistics AND electronics OR e-waste AND multiple criteria OR multicriteria, we found 278 papers. The number of articles in two bases was 608 papers.
After the first filtering process, which comprised the title and abstract reading in the bases Science Direct, Web of Science and Emerald, 256 papers, six, and 269 papers were eliminated, respectively, thus 77 papers remained.
At the second filtering process, which covered the introduction and conclusion reading, aside from duplicate checking, 24 papers were eliminated from Science Direct; three papers from Web of Science, and five papers from Emerald. The number of articles remaining was 45.
In Part 2, we applied the formula for calculating the index InOrdinatio (IO), which allows for a ranking of the remaining papers to be obtained, organized from the best to the worst. These 45 papers met all previously defined inclusion criteria in the protocol of the systematic literature review. After the calculation of IO, it was defined by the researchers to establish IO < 80 as the cut level. According to Pagani, Kovaleski, and Resende [10], this cut level should be defined by the researchers according As can be noted in Figure 1, the systematic literature review covers two parts: Part 1 and Part 2. In Part 1, in Science Direct, with the terms reverse logistics AND electronic AND multiple criteria OR multicriteria, 307 results were found. In Web of Science, with the terms (TS = (reverse logistics AND electronics) AND TS = (multiple criteria OR multicriteria)), we found 23 results. In Emerald, with the terms reverse logistics AND electronics OR e-waste AND multiple criteria OR multicriteria, we found 278 papers. The number of articles in two bases was 608 papers.
After the first filtering process, which comprised the title and abstract reading in the bases Science Direct, Web of Science and Emerald, 256 papers, six, and 269 papers were eliminated, respectively, thus 77 papers remained.
At the second filtering process, which covered the introduction and conclusion reading, aside from duplicate checking, 24 papers were eliminated from Science Direct; three papers from Web of Science, and five papers from Emerald. The number of articles remaining was 45.
In Part 2, we applied the formula for calculating the index InOrdinatio (IO), which allows for a ranking of the remaining papers to be obtained, organized from the best to the worst. These 45 papers met all previously defined inclusion criteria in the protocol of the systematic literature review. After the calculation of IO, it was defined by the researchers to establish IO < 80 as the cut level. According to Pagani, Kovaleski, and Resende [10], this cut level should be defined by the researchers according to their experience and sensitivity related to the subject researched, thus three papers were eliminated, resulting in 42 papers analyzed.

Multicriteria Modeling
Based on the systematic literature review conducted in this paper and considering that, although some papers addressing the decision problem were related to the implementation of reverse logistics of e-waste in a multicriteria context used the PCP such as [11,34], which does not require the assignment of the weights. It is recognized that this is one of the most contested steps by some researchers because the weights tend to be arbitrary and biased, especially when the decision context involves multiple decision-makers.
The probabilistic composition of preferences [11] is a multicriteria decision aid methodology able to take into account the presence of uncertainty in the preference evaluations. In this method, the measure of preference for an alternative is given by the probability of this alternative being chosen.
The PCP is developed in two steps: (1) Define the probability of being the best and/or the worst option; and (2) Carry out the composition of the probabilistic preferences. In the first moment, an ordering of the set of options is established through the determination of preferences according to the type of criteria. After determining this ranking, the maximum (Mij) and/or minimum (mij) probabilities of each option (alternative) i must be calculated with concern to the others, combining the preferences obtained for each criterion j. [35].
At the same time, it is important to point out that realistic situations usually involve multiple decision-makers, with conflicting views. Thus, this method can be used to avoid arbitrariness and bias, incorporated in the final decision. Then, the model is adequate to be used considering the preferences of several decision-makers, covering several points of view, thus providing a more holistic view of the decision problem.
In the earlier stages of the implementation of reverse logistics, in this case related to e-waste, it is necessary to continuously follow the advances, assessing the performance in order to implement changes and some improvements. For this purpose, MCDA methods can be very useful because they allow the systematization of decision elements in well-defined steps and measure the performance of some alternatives or options of action under a set of criteria. As a result, for example, it is possible to obtain information on what options of action should be prioritized to be improved.
The PCP method was used by [36], carrying out the prioritization of the actions considering the set of criteria, which was based on the systematic literature review. In this sense, the contribution of this proposition can be emphasized, the model approach the implementation of reverse logistics of e-waste from the perspective of multiple criteria, which can enable the constant revision of the model chosen to implement reverse logistics, and allows the timely correction of the course of action by decision-makers.
This model differentiates from others already published because it uses a non-compensatory approach to deal with the assessment of alternatives under a set of criteria, the probabilistic composition of preferences (PCP) method. This method has proven to be more appropriate than others from an outranking approach, considering that it does not require the assignment of weights such as Promethèe and Electre. The assignment of weights, in general, tends to be arbitrary and requires great effort in the elicitation process from the decision-makers. Another advantage is related to the evaluation of the criteria (exact values), which are converted into possible values by using a probability distribution, allowing an imprecision to be considered in the evaluation process. Therefore, this method contributes to avoiding mistakes in the elicitation process and tend to be cognitively easier for decision-makers.

Presentation of Results of Systematic Literature Review (SLR)
Considering that this paper used two methodological procedures: (i) a systematic literature review (SLR); and (ii) multicriteria modeling, we present Section 4 in two parts: Section 4.1 with the results of SLR and Section 4.2 with the proposition and illustration of the multicriteria model.

Analysis of the Papers Found in Systematic Literature Review (SLR)
The descriptive analysis of the bibliographic data frame shows the main characteristics of the publications. Therefore, it was possible to identify the evolution of annual scientific production on this topic; the most cited papers; most productive countries; most productive universities, the collaboration between researchers around the world related to this topic; the most relevant sources of publications (journals); the analysis of co-occurrence and co-citation network/the authors coupling; the main criteria or variables considered in the decision-making process of reverse logistics of e-waste, main methods or models used by the structure of these criteria/variables; and the main technical procedure to collect and organize the data. In addition, the qualitative analysis of the complete articles was carried out through content analysis, as proposed by Bardin [37]. In general, we expressed the desire to view the first ten authors or first ten sources. The results are displayed in Tables 1-8.  First, Figure 2 shows the annual scientific production and its evolution. Analyzing these data, we can point out that 2009 was the only one without publications, and the greatest number of publications occurred in 2014 with eight papers, followed by 2011 with seven, 2016 with six, and 2015 and 2017 with five papers each. The estimated annual percentage growth rate was 22.28.

Analysis of the Papers Found in Systematic Literature Review (SLR)
The descriptive analysis of the bibliographic data frame shows the main characteristics of the publications. Therefore, it was possible to identify the evolution of annual scientific production on this topic; the most cited papers; most productive countries; most productive universities, the collaboration between researchers around the world related to this topic; the most relevant sources of publications (journals); the analysis of co-occurrence and co-citation network/the authors coupling; the main criteria or variables considered in the decision-making process of reverse logistics of ewaste, main methods or models used by the structure of these criteria/variables; and the main technical procedure to collect and organize the data. In addition, the qualitative analysis of the complete articles was carried out through content analysis, as proposed by Bardin [37]. In general, we expressed the desire to view the first ten authors or first ten sources. The results are displayed in Tables 1-8.
First, Figure 2 shows the annual scientific production and its evolution. Analyzing these data, we can point out that 2009 was the only one without publications, and the greatest number of publications occurred in 2014 with eight papers, followed by 2011 with seven, 2016 with six, and 2015 and 2017 with five papers each. The estimated annual percentage growth rate was 22.28.  Table 1 shows the 10 most productive authors, in which the authors' frequency distribution was defined by Articles Fractionalized.   Table 1 shows the 10 most productive authors, in which the authors' frequency distribution was defined by Articles Fractionalized.
Tables 2-6 present the descriptive analysis of the papers more cited, the main journals of publication, main keywords, and the main authors. These results can be useful for researchers acting in this field, in order to identify the most relevant papers and journals.
It is worth mentioning the variety of areas of knowledge related to journals of publications analyzed in this SLR. This observation is an indication that concerns the issue of e-waste impacts and is impacted in many ways: social, environmental, technological, managerial, productive, economical, and others. Thus, their analysis and solution can originate from different theoretical approaches, but also, moreover, from the dialogue between them.
According to Baldé et al. [1], China, the United States of America, Japan, India, Germany, the United Kingdom, Brazil, Russia, France, and Indonesia are respectively the ten economies that generated the most waste in 2016 in absolute numbers per kg/ton. We can observe that four of the ten countries cited had publications in this period. This evidence may point to a relationship between the commitment of academia and the emergence of the problem. In addition, it is important to point out that most of these countries have specific legislation on e-waste management. Table 7 presents the total citations per country. Figure 3 shows the most productive universities, considering the link of corresponding authors with two or more papers: the Indian Institute of Technology Roorkee, India; Abant Izzet Baysal University, Turkey; Duzce University, Turkey; University of Brasilia, Brazil, Tilburg University, The Netherlands; University of Southern Maine, United States of America (USA); and York University, the United Kingdom (UK). Figure 3 shows the most productive universities, considering the number of papers per corresponding authors. Table 9 corroborates the results from Figure 3, considering that the most productive countries appear at the top of the ranking of the most cited countries, except for the case of Brazil and Turkey.  Figure 3 shows the most productive universities, considering the link of corresponding authors with two or more papers: the Indian Institute of Technology Roorkee, India; Abant Izzet Baysal University, Turkey; Duzce University, Turkey; University of Brasilia, Brazil, Tilburg University, The Netherlands; University of Southern Maine, United States of America (USA); and York University, the United Kingdom (UK). Figure 3 shows the most productive universities, considering the number of papers per corresponding authors. Table 9 corroborates the results from Figure 3, considering that the most productive countries appear at the top of the ranking of the most cited countries, except for the case of Brazil and Turkey. In addition, Figure 4 shows the collaboration between researchers, considering their country of affiliation. We can highlight partnerships between researchers from the USA and Thailand; USA and Canada; USA, Spain, and The Netherlands; USA and Malaysia; USA, Canada, and Austria; USA and Turkey; USA and Denmark; Brazil and Portugal; Brazil and China; Australia and Switzerland; and Australia and India, which means that the USA has a greater frequency of partnerships, followed by Australia, Brazil, Canada, and China. In addition, Figure 4 shows the collaboration between researchers, considering their country of affiliation. We can highlight partnerships between researchers from the USA and Thailand; USA and Canada; USA, Spain, and The Netherlands; USA and Malaysia; USA, Canada, and Austria; USA and Turkey; USA and Denmark; Brazil and Portugal; Brazil and China; Australia and Switzerland; and Australia and India, which means that the USA has a greater frequency of partnerships, followed by Australia, Brazil, Canada, and China.   Complementarily, regarding the publishers, it was found that 33 journals were from Elsevier; five from Emerald; three from Springer, one from Taylor & Francis, and one from Sage.
The co-occurrence network of the top-thirty terms on research is presented in Figure 5. The circle size of the terms represents the frequency of occurrence in the articles. The thickness of the line between any two terms displays the degree of cooperation. These terms include decision-making, supply-chain management, and design. It should be noted that the keyword co-occurrences analysis identified "China" as the only country. Complementarily, regarding the publishers, it was found that 33 journals were from Elsevier; five from Emerald; three from Springer, one from Taylor & Francis, and one from Sage.
The co-occurrence network of the top-thirty terms on research is presented in Figure 5. The circle size of the terms represents the frequency of occurrence in the articles. The thickness of the line between any two terms displays the degree of cooperation. These terms include decision-making, supply-chain management, and design. It should be noted that the keyword co-occurrences analysis identified "China" as the only country. According to Mo et al. [38], citation is not a perfect index to evaluate the author's impact in a specific field of research. However, co-citation patterns are found to differ significantly from bibliographic coupling patterns, but to agree generally with patterns of direct citation. Clusters of co-cited papers provide a new way to study the specialty structure of science. They may provide a new approach to indexing and creating SDI profiles. The co-citation frequency of two scientific papers can be determined by comparing lists of citing documents in the Science Citation Index and counting identical entries [39]. Article accessibility is a possible constraint for authors to cite articles. Some authors may have access to a large number of articles, while others may not. Therefore, the overlap between the accessible articles would be the only linkage for a coupling relationship between the authors in these two groups. Moreover, the authors' preferences for different kinds of articles differ [38].
From Table 3, we can see that most of the top manuscripts per citations were published after 2010. Similarly, the majority of articles presented in the co-citation network were published after 2010 ( Figure 5). On the other hand, the most frequently cited manuscripts (Table 5) were published before 2002. This may be attributed to the new emergency trending regarding world sustainability.
Finally, Figure 6 shows the authors' coupling, where two authors are coupled through a third party, the cited article in this case, and it can be demonstrated by the authors' publications and articles cited by them. Regarding author publication, it is true that authors need to have a certain number of publications in order to have the chance to be coupled with other authors through cited references [38].
The colors in Figure 6 represent the networks of connections between the authors, differentiating the networks present. As can be seen in Figure 6, some nodes (circles) were larger than others, as there was a greater quote. [38].
From Table 3, we can see that most of the top manuscripts per citations were published after 2010. Similarly, the majority of articles presented in the co-citation network were published after 2010 ( Figure 5). On the other hand, the most frequently cited manuscripts (Table 5) were published before 2002. This may be attributed to the new emergency trending regarding world sustainability.
Finally, Figure 6 shows the authors' coupling, where two authors are coupled through a third party, the cited article in this case, and it can be demonstrated by the authors' publications and articles cited by them. Regarding author publication, it is true that authors need to have a certain number of publications in order to have the chance to be coupled with other authors through cited references [38]. The colors in Figure 6 represent the networks of connections between the authors, differentiating the networks present. As can be seen in Figure 6, some nodes (circles) were larger than others, as there was a greater quote.    In addition, Figure 8 shows there are three groups (networks) of links between the most cited authors.  In addition, Figure 8 shows there are three groups (networks) of links between the most cited authors. In addition, Figure 8 shows there are three groups (networks) of links between the most cited authors.

AUTHORS
CRITERIA/VARIABLE CONSIDERED METHODS/MODELS TECHNICAL PROCEDURE [40] No one identified No one identified Survey [41] LCA criteria: Door-to-door collection by truck, Door-to-door collection by pickers, Delivery at electrical electronic equipment (EEE) shops collection Delivery at metro stations, Delivery at delivery points, Dismantled by recycling company, Refurbishment by social program, Informal dismantling Informal recycling, Adequately recycled regionally, Social; Enterprises/cooperatives, PWB recycling exported overseas, Landfill; Economic criteria: System feasibility System efficiency, Population awareness and adhesion to reverse logistics, Innovation and generation of new economic activities, Competitiveness of formal products in regard to informal ones; Social criteria: Social inclusion, Formal employment Generation of income, Opportunity for professional development, Health risks and working, Conditions, Access to healthcare, Access to education.
PCP Modeling [42] No one identified No one identified Survey [3] No one identified No one identified Interview [43] Legislation; customer demand; strategic cost/benefit; environmental concern; volume and quality; incentive, resource, integration and coordination Strategic: General enforcement of the law, qualification of the companies, partnerships, Incentives; Environmental: Promotion of environmental education, creation of natural resources savings policies, creation of policies for the correct disposal of e-waste by companies; Economic: Creation of policies for materials reutilization; savings through the use of recycled materials; revenues from the sales of e-waste; creation of advertising campaigns, installation of collection points for e-waste, creation of tax incentives for companies operating RL of e-waste; Social: Creation of policies for public awareness; creation of employment policies, creation of digital inclusion policies and, creation of qualification policies.

Strategic Options Development Analysis (SODA)
Structuring problem [46] Electrical fault, Mechanical fault, Electrical and mechanical faults No one identified Interviews [47] Reuse, recycling, return incentive, collection and shipping, inspection, sorting, average cost, average revenue No one identified Documental analysis Clear return policies, recognition of reverse logistics as a factor in creating competitive advantage, performance management system, information technology support, forecasting and planning.
No one identified Survey and interviews [49] Economic viability, environmental soundness, transport issues, No one identified Documental analysis [50] Economic: Operating cost (transportation and labor); Fixed cost: (capital investment, Operating time); Operational: Handling system capacity, Human resources (number of staff involved); Strategic: Accessibility (proximity to the customer/distance), Flexibility and responsiveness (product flexibility, type of products accepted) System response (time); Social: Stakeholders' participation/willingness to cooperate.
Fuzzy Analytic Hierarchy Process (AHP) Modeling [51] No one identified No one identified Documental analysis [52] Knowledge requirements: Accessibility of information, What and how to sort/segregate, Where to bring materials, How to get to drop-off site, When (days and hours) to go, Acceptance requirements, Drop-off procedures, Drop-off fees; Proximity to a collection site: No one identified Documental analysis [53] No one identified No one identified Survey [54] Channel conflict, investment, Closed-loop chains unlikely, control over designs/materials, Decouples design, contracting and oversight, Processes fast and flexible, Allows for recovery of valuable modules, Higher recovery rates, Long-term contracts, Higher variable costs, Closed-loop supply chains, levels of material recovery and learning, access to scarce materials No one identified Documental analysis [55] Total cost, number of disposed items, material sales revenue, and customers' satisfaction level ARTODTO model, linear physical programming model Modeling [56] The types, numbers and locations of storage sites, The types, numbers and locations of recycling facilities, The quantity of product categories to be allocated to storage sites, The quantity of product categories to be allocated to recycling facilities, The network flow of product categories through storage sites, recycling facilities and secondary market, The total cost of the reverse logistics system. Profit Game Theory Modeling [46] Quality; cost No one identified Survey and interviews [69] Internal barriers and external barriers No one identified Interviews [70] Investment; installing capacity; collection network; quantity of discarded products. MILP Modeling [71] Components of the TPB, socio-demographic and socio-economic variables, the degree of awareness towards the problem, and the personal assessment of the environmental situation of Brazil as predictors of e-waste recycling   Based in Table 8, we could identify the main criteria or variables considered in the decision problem related to reverse logistics and the variations (which can be encountered in some authors as part of its activities: recycling, reuse, refurbishing, remanufacturing, collection, disposal) involved. Regarding the main methods or models used to deal with these criteria/variables, it was possible to identify mainly methods from linear programming such as mixed integer linear programming (MILP), used in six of 42 papers analyzed in this systematic literature review, and from the multiple criteria decision aid (MCDA) approach such as the analytic hierarchy process (AHP), used in five papers. We could not fail to mention that 20 of 42 papers did not cite any structured or formal method of the analysis of the data. The remaining 11 papers used several other methods, as shown in Table 10. Other important data excerpted from Table 10 are related to technical procedures used to collect and organize the data. We can also observe that most of the papers, 20 of them, used the modeling as the main procedure; 10 used documental analysis; seven of them used interviews, and another six used survey. The modeling methods were used mainly to create models for structuring, managing, and optimizing reverse logistics chains/networks for e-waste. The papers using other procedures had the purpose of describing reverse activities of e-waste in some regions such as informal recycling in China, cases of industries in Turkey, the barriers to implement reverse logistics, the behavior of disposal of consumers from or the new demands that arise for stakeholders from Brazil due to legal demands. Some papers have analyzed the efforts made by different actors (government, manufacturers, consumers, distributors, third party logistics providers, among others) along the supply chain.

Mixed integer linear programming (MILP) Modeling
The main contribution of these papers is related to the dissemination of practices that can be reproduced in other contexts and that favor the expansion of the adoption of reverse logistics. Some studies presented data on the production and destination of e-waste with the demographic information. There were also studies dedicated to the evaluation of the efficiency of reverse logistics systems for e-waste. This kind of study, usually qualitative, contributes to the identification of bottlenecks and barriers in the reverse reinforcing the challenges to implement it, mainly in emerging economies, which can include factors related to technology, supply chain process governance, economics, knowledge technical, legislation/policy and market, among others.
Additionally, some papers have evaluated the economic, social, and environmental impacts of the implementation of reverse logistics for e-waste. These studies include comparisons related to the perception of different actors involved in the reverse logistics of e-waste, also related to management techniques and instruments. Another important aspect of the studies related to the reverse logistics of e-waste are the legal and political concerns. These papers describe the measures taken by governments of different countries to implement and control the generation and management of e-waste. It can contribute to the benchmarking of institutionalized practices, the evaluation of the efficiency of policies, and the leveling of national advances in issues related to e-waste. The use of an information system related to the operational process of reverse logistics for the recovery of e-waste can also be found in some papers, citing technologies such as radio frequency identification (RFiD) and enterprise resource planning (ERP). Furthermore, the analysis of consequences of the decisions taken in the development of products considering their life cycle was also encountered in this literature review.

The Model Proposed to Assess the Implementation of the Reverse Logistics of e-Waste in the Earlier Stages
The model proposed follows the MCDA approach, as derived from the operational research (OR) area. Considering that the modeling of problems, considering a MCDA approach, should be systematized not only in terms of technical inputs and analytical characteristics, but also in terms of the support and insight given to the implementation [24], we preview steps aimed at the elicitation process with decision-makers conducted by an analyst.
Some assumptions of the model, based on the SLR, should be pointed out: (a) This model includes several qualitative and quantitative criteria, related to economic, social, and environmental issues, which very often have conflicting relationships [3,4,25,36]; (b) This context of decision-making comprehends several stakeholders such as government, citizens, waste manager, industries of electrical electronic equipment (EEE), recycling industries of e-waste, third-party logistics providers, small, medium or large size retailers, waste pickers and, non-governmental organizations, and others [3,4,36]; (c) These stakeholders, very often, present conflicting points of view related to the implementation of reverse logistics of e-waste, considering economic, social, and environmental dimensions, considering their different interests related to the topic [4,36]; (d) Suppliers should have a balanced performance in all criteria proposed, which means that compensation between criteria is not allowed (a poor performance in a given criterion should not be compensated for by very good performance in another criterion) [4,25,80]; (e) In realistic situations, decision-makers may have difficulties to decide due to (i) there is subjective or insufficient data; (ii) the analyst may not know the preferences of the decision-makers; and (iii) it is hard, or even arbitrary to assign weights or exact values to the criteria chosen to evaluate the options of actions (alternatives) [36]; and (f) It is required that the model incorporates inaccurate, imprecise, uncertain, or ill-determined data and considers the imperfect nature concerning the values of the criterion variables in the decision-making process [4,25,36].
A framework of the model proposed can be seen in Figure 8. As presented in Figure 8, the model has the following steps: (1) Identifying the decision-makers (active participants to be involved in decision making). These people should be selected for their knowledge and experience about various functions in the company, and they do not have the same competence. Since conflicting judgments are unlikely to be resolved by consensus, stakeholders can be weighed according to their power to influence results. and prioritize the actions that need immediate actions to improve the process. The proposed methodology seeks to optimize according to all criteria and considers the probabilities of maximization; (6) Delivering results. The results can be presented in a ranking organized from the best to worst, or the worst to best, demonstrating what alternatives should be improved in order to reach the objective of the implementation of the reverse logistics of e-waste [p1]; and (7) Providing Feedback. Based on the results, it is possible for decision-makers to decide whether to maintain the course of action or whether to implement changes to improve the performance of indicators in a timely and accurate manner.
Additionally, it is also possible to conduct an analysis in two scenarios, using the beta and triangular distribution, to test the validity and robustness of the results.
Numerical Example to Illustrate the Use of the Model Proposed Step 1: The decision-makers can be the people in charge of the logistics, financial, production and operations sectors, among others.
Step 2: It is expected that the decision-makers analyze the criteria suggested based on the systematic literature review (Table 9), which can be categorized in four major groups: economic, technical and managerial, environmental, and social, and pick those more appropriate for the context. To illustrate the model, we will use the following criteria: Economic category: C1-Costs of transportation; C2-Cost of refurbishing, and operating the disassembly workstations; C3-Total revenue; C4-Total Investment; C5 Selling price of refurbished devices; C6-Costs of installation of collection points for e-waste; and C7-Incentives for customers return e-waste.
Environmental: C14-Promotion of environmental education; C15-Natural resources savings; C16-Energy use on the process level; C17-Carbon footprint; and C18-Water footprint.
Social: C19-Formal employment/Jobs; C20-Generation of income; C21-Health & Safety of employees; C22-Labor Conditions; C23-Digital inclusion; and C24-Engagement in community. In Step 3, there are some alternatives in the literature [37], which will be used for the purposes of the illustration of the application of the model: i.
A1-Door-to-door collection by truck; ii.
A2-Door-to-door collection by waste pickers; iii.
A3-Delivery at EEE shops collection; iv.
A5-Delivery at delivery points; vi.
Step 4: The elicitation of the preferences of decision-makers, for the illustration of the model, will be made by using questionnaires composed of ordinal scales, ranging from 1 to 5, in which 1 is the lowest level and 5 the highest level. Therefore, the matrix to evaluate will be composed of 10 alternatives (A1-A10) and 24 criteria (C1-C24), as in Table 10.
To obtain the probability of an option i being the best (M ij ) or the worst (m ij ), according to each criterion j, statistical independence must be assumed to simplify the calculation of the joint probabilities by the product of the marginal functions of probability, according to Equations (2) and (3) [81].
The parameters FX, fX, and DXi represent, respectively, the cumulative distribution function of the variable x−i, which represents the other options under the same criterion, except for the variable xi considered in calculation; the probability density function; and the support of the random variable xi, which indicates the lower and upper limits of the domain of the respective variable. In addition, it is important to state that in Equations (2) and (3), the parameters FX, fX, and DXi represent, respectively, the cumulative distribution function of the variable x−i, which represents the other options under the same criterion, except for the variable xi considered in calculation; the probability density function; and the support of the random variable xi, which indicates the lower and upper limits of the domain of the respective variable.
Then, depending on the approach considered, different joint probabilities can be used. In this sense, such approaches can be discriminated in terms of the choice between positions among two basic orientations, optimistic versus pessimistic and progressive versus conservative, as shown in Table 11 [35]. Table 11. Different approaches for combining probabilistic assessment.

Optimistic Pessimistic
It considers satisfactory to meet at least one criterion. It seeks optimization according to all criteria.

Progressive Conservative
Consider the probabilities of maximizing preferences. Consider the concern only in avoiding negative extremes; chances of not minimizing preferences.
Source: This table was adapted from [82].
The proposed methodology seeks to optimize according to all criteria and considers the probabilities of maximization (to obtain the highest risks), so it is necessary to use the pessimistic/progressive approach. Thus, to calculate the probabilities of preference in this study, it was only necessary to estimate the probabilities of maximum preference (Mij). Pessimistic/Progressive: (PPe) = j=1,...,m M ij ; For more details of this method, see [11].
Step 5: Application of the PCP model. Table 12 also shows a noticeable concordance between the ranking obtained employing the beta and the triangular distribution. The beta distribution is a weighted average in which more weight is given to the most likely estimate, while the triangular distribution is defined by its minimum, maximum and mean values and it does not have to be symmetric. The beta-PERT (Program evaluation and review technique) is a tool for modeling expert data, based on the likelihood of components. In the case of beta distribution, the package 'PCP' was used, through the implementation in software R. In the case of triangular distribution, preferences were normalized in the interval (0.1), and these parameters were assumed as the modes of triangular distributions with amplitude (0.1). It was possible to observe that the ranking was not modified in the two distributions (Beta-PERT and triangular), which demonstrates that the modeling is robust and stable.
Aside from the studies considered in the SLR, some other relevant papers discussed issues related to recycling and the reverse logistics of e-waste from several perspectives. The comparison of e-waste credits with the carbon credit market, in the context of Brazil, was made by [3]. The analysis of sectoral agreement for the reverse logistics of e-waste, also in the context of Brazil, was conducted by [4]. The authors used the SODA (strategic options and development analysis) approach to structure the decision-making context considering the opinion of several stakeholders involved. The decision support systems for the optimal location of e-waste treatment plants, in the context of Greece, was studied by [19].
The prioritization of barriers to be solved in the implementation of reverse logistics of e-waste, in the Brazilian context, considering the opinion of SMEs (small and medium companies), consumers, and government under an MCDA perspective was performed by [36]. The study of relevant factors in the implementation of end-of-life computer recycling in reverse supply chains was approached by [43]. The quality of recycling of e-waste from end-of-life computers was analyzed by [45]. The life-cycle perspective was approached by [83], which presented a quantitative framework to estimate life-cycle impacts and costs associated with e-waste management, with well-defined steps. Data from the State of Washington's EPR program were used to test the framework, involving the activities of collection, transportation, processing, and disposal.
Thus, the theoretical model proposed in our paper differentiates from the previously mentioned because it considers several criteria related to economic, social, environmental, technical, and managerial issues to assess the alternatives to be used in the early stages of implementation of reverse logistics of e-waste. This context of decision is very characteristic in developing countries such as those from BRICS (Brazil, Russia, India, China and South Africa), which are in earlier stages of the reverse logistics of e-waste, differently from developed countries, in which the reverse logistics of e-waste is already institutionalized and consolidated. Our decision model provides well-defined steps to select from the literature the proper criteria and alternatives and to analyze them under a multicriteria approach, which does not consider trade-offs between criteria and does not require the assignment of weights. The trade-off of criteria and the assignment of weights are very often considered as contradictory in the literature, so avoiding these issues, the model can be considered more user-friendly and provide more balanced solutions. Therefore, with the use of the model, it is possible to obtain inputs to make better decisions, considering a systemic analysis of the criteria and the alternatives of actions to implement the reverse logistics of e-waste in the earlier stages, which can be useful mainly for managers and policymakers acting in this field.

Conclusions
The purpose of this paper was to assess the implementation of the reverse logistics of e-waste in the earlier stages. The study was conducted in two steps. The first step was aimed to identify the main characteristics of publications related to reverse logistics considering multiple criteria, based on the literature from 2007 to 2017, in order to obtain inputs to insert in Step 2.
Step 2 is related to multicriteria modeling; in this paper, we illustrated the model proposed with a numerical example containing 10 alternatives and 24 criteria.
In the first step, we identified the main characteristics related to the evolution of annual scientific production on this topic; the most cited papers; most productive countries; most productive universities, the collaboration between researchers around the world related to this topic; the most relevant sources of publications (journals); the analysis of co-occurrence and co-citation network/the authors' coupling; the main criteria or variables considered in the decision context of reverse logistics of e-waste, main methods, or models used by the structure of these criteria/variables; and main technical procedure to collect and organize the data. Thus, besides the bibliometric analysis, we performed a content analysis considering the main contributions of the papers analyzed. This part of the paper can guide practitioners and researchers to identify the literature related to this topic of research.
Step 2 aimed to model the decision-making context, and for this purpose, we used the results from Step 1 to gather alternatives and criteria to apply the MCDA modeling, composed of seven steps: (i) identifying the decision-makers; (ii) validating the set of decision criteria; (iii) identifying the alternatives; (iv) eliciting the preferences of the decision-makers; (v) applying the PCP method; (vi) delivering results, and (vii) providing feedback.
Considering the results obtained in the first step, which was the SLR, we could verify that there is a large predominance of papers addressing decisions related to the optimization of reverse logistics networks as well as analyzing the barriers and challenges related to reverse logistics implementation. Many papers do not make clear which methods or models were used to analyze the criteria or variable, which contributes to the understanding that the authors qualitatively analyzed the factors related to the economic, social, and environmental impacts of reverse logistics. However, despite this, it is important to emphasize that a considerable part of the papers analyzed used structured methods/models from MCDA and linear programming approaches to analyze the dimensions involved in this context. Some papers have highlighted the characteristics of some regions/countries related to e-waste generation and management, others have described the new demands to the several actors involved in this context, and the behavior of consumers to dispose the e-waste. Considering the evolution of the number of studies published each year, it is clear that interest in the study of this topic is increasing, although 2009 did not present one paper published, considering the limitations of this paper related to the criteria of inclusion and exclusion of papers considered for this literature review.
Related to the second step, the MCDA modeling, it was possible to offer to decision-makers well-defined steps to guide the decision-making process in the context of the reverse logistics of e-waste. Considering the earlier stages in developing countries related to the implementation of the reverse logistics of e-waste, perhaps the alternatives as well as the criteria or performance indicators used to measure the implementation are not well-defined. Therefore, a framework aimed toward this purpose can be very useful, providing timing and accurate information in the form of feedback. Considering this feedback, it is possible to decide whether the current strategies can be maintained or changed, incorporating improvements in those alternatives with the worst performance in the criteria selected, covering the social, environmental, technical, and managerial dimensions. The consideration of these dimensions can provide a holistic view of the decision context. As a suggestion for further studies, we recommend the use of other protocols of systematic literature review related to this topic to update the SLR. The proposition of structured methods dealing with the dimensions of economic, social, and environmental dimensions of the sustainability of reverse logistics of e-waste can be the object of future studies. Additionally, the integration of methods from problem structuring methods (PSM) with multicriteria decision aid (MCDA) or with derivations of linear programming methods can also be studied. The use of the concepts of Logistics 4.0 and Supply Chain Management 4.0, Internet of Things (IoT), and respective technologies and systems related are topics still not well exploited in the literature, and constitute a great opportunity for future studies. Additionally, research comparing the developed and developing countries' best practices and challenges related to e-waste management and respective legal requirements can be the target of further studies.
The theoretical contributions of this paper are two-fold. First, we synthetized the existing knowledge on the reverse logistics of e-waste considering multiple criteria or variables of decision in the last years, which can be useful to researchers exploiting this topic. We also analyzed which topics were still underrepresented in the literature, identifying suggestions for future research, which can guide researchers and practitioners. Second, we propose a method with well-defined steps to guide decision-makers to provide more consistent and timing information. For business practice, this paper can be useful for managers to synthetize the main criteria and methods used in this context of decision, serving as a base for decision-making related to this topic.