Interviewing and Hiring Practices in Brazilian Academia: Proposals Towards Improvement
Jonathan Jan Benjamin Mijs
Round 1
Reviewer 1 Report
This interesting opinion piece presents the system of evaluation used in academic hiring in Brazil and argues it is "unmeritocratic." To this end, the author provides an unusually extensive presentation of the current system of evaluation. This, paradoxically, is also the article's major shortcoming. By focusing 11 of 13 its pages on description, little space remains to convey and develop the author's argument.
Obviously, the Brazilian case needs introducing in order for it to appeal to this journal's international audience, which in all likelihood is unfamiliar with the intricacies of Brazilian academia. Rather than providing an overwhelming amount of detail and graphics (15 figures!), it would be more helpful if the author could compare and contrast Brazilian practices to those in other countries that may be more familiar to readers, or refer to widely used international standards.
Alternatively, the author could (and I would very much urge them to) relate the Brazilian case to forms of evaluation that scholars of culture, organization and inequality have studied. That way it would be easier to understand and appreciate, for an international audience, the (non-meritocratic) practices in Brazilian academia.
As is, the article fails to engage with (or cite) critical scrutiny of meritocratic systems, forms of measurement, accountability and evaluation, in practice and in comparison with other countries where such systems are in effect. See for instance Michèle Lamont's work on the "Sociology of Valuation and Evaluation" (2012) and her "How Professors Think" (2010).
It also sidesteps important scholarly debates about the desirability of meritocracy in the first place. For recent contributions to this debate, see Jo Litter's "Against Meritocracy" (2017) or my own work on the "Unfulfillable Promise of Meritocracy" (2016).
As for the organization of the piece, it is not clear from the introduction to the article exactly why Brazilian academia doesn't function meritocratically. It would help the reader if the argument were introduced prior to the extensive discussion of the current state of hiring processes in Brazil. That way, readers know what to look for when reading the section (i.e. signs of non-meritocratic evaluation).
I understand that the author needs to devote some space to discussing the current system of evaluation, but I am not convinced this discussion needs to be so extensive. This article was submitted as an opinion piece. Therefore I would expect the majority of it to introduce, develop and convey a clear argument. Instead, pages 1 through 11 (of 13) are an elaborate description of current practices. It escapes me why the author needs to present 14 (!) figures (Figs 2-15) to illustrate points that could simply be made in a few lines of text. That leaves less than 2 pages for the author to argue what exactly is wrong with the current system of evaluation, and to discuss what a better would look like, how it would address the shortcomings of the current system, and how it would remedy the situation.
Author Response
Reviewer 1
This interesting opinion piece presents the system of evaluation used in academic hiring in Brazil and argues it is "unmeritocratic." To this end, the author provides an unusually extensive presentation of the current system of evaluation. This, paradoxically, is also the article's major shortcoming. By focusing 11 of 13 its pages on description, little space remains to convey and develop the author's argument.
Thank you for your thoughtful review. We have prepared a revised version of the manuscript. In this file, you can find an item-by-item response to your comments and suggestions, showing how we have implemented these in the revised version of the manuscript.
Section 3.2 has been reduced in length to half a page. The interested reader is referred to the preprint of the article to find a longer analysis, including screenshots of all missing and erroneous features of Lattes CV.
The new version of 3.2 reads as:
Several types of ineffective features found in Lattes CV are described next. For a more detailed analysis, showing screenshots of all elements mentioned in this analysis, please refer to the preprint of this work [1].
We first address essential information that cannot be included in Lattes CV, or that is not compulsory. The main missing element on a Lattes CV is the contact information, such as (i) email address, (ii) phone number, or (iii) personal website(s). The only information sometimes provided are the home or institutional postal addresses. Additionally, one cannot select more than one scientific field of expertise in Lattes CV, which does not serve researchers in multidisciplinary or interdisciplinary topics. An English version of Lattes CV is not always available (compare random profile [2] to [3]). Some journal papers are missing the Digital Object Identifier (DOI)’s hyperlink, issue number or even the title (see for example random profile [4] which is missing information of references [5,6]).
Next, we analyze superfluous features on Lattes CV. Since Lattes CV does not impose word count limits on some sections and fields, some profiles become overly long, which may be difficult for recruiters to extract the most important information (see for example randomly selected profile [7]). Additionally, the ‘Formal education/Degree’ section includes fields such as (information from random profile [8]) (i) Major Area (e.g., ‘Linguistics, Letters and Arts’), (ii) Area (e.g., ‘Letters’), (iii) Sub-Area (e.g., ‘Literaturas Estrangeiras Modernas’ – without English translation), and (iv) Specialty (e.g., ‘Literatura em Língua Inglesa’ – without English translation). These extra subdivisions clutter the profiles and the information can already be found in the ‘keywords’ field. Lastly, some sections on the CV could be shortened or excluded (e.g. [2]): (i) expanded abstracts and abstracts published in conference proceedings (for most purposes only full conference papers or presentations are evaluated), (ii) participation in examination boards, and (iii) academic advisory (information could be presented in a more concise form).
The Lattes platform [9] also contains a number of software issues that make creating and updating a CV difficult. Selecting an institution requires a number of steps. Multiple entries of the same institution exist, as users are allowed to create institutions. A link between institution and its country is also missing.
Finally, the Lattes platform does not translate well into English. Even though the platform is designed for researchers, many technical terms are not correctly translated into English. One striking error is that abstract is translated as summary (see for example profile [2]). For a list of erroneous translations for keywords in civil engineering, refer to the extended analysis in the preprint [1].
One feature in the Lattes platform that can be very useful for recruiters (although only available in the Portuguese version, as can be seen in [2]), is the indication per journal publication of the number of citations from the Web of Science (WoS), Scopus, and Scielo databases.
Obviously, the Brazilian case needs introducing in order for it to appeal to this journal's international audience, which in all likelihood is unfamiliar with the intricacies of Brazilian academia. Rather than providing an overwhelming amount of detail and graphics (15 figures!), it would be more helpful if the author could compare and contrast Brazilian practices to those in other countries that may be more familiar to readers, or refer to widely used international standards.
The revised version of the manuscript now contains just 3 figures. All other figures and screenshots are kept in the preprint, and the interested reader is referred to the preprint for a more detailed analysis of the current state of CV Lattes.
As for comparing and contrasting the Brazilian practices to those in other countries, we have included the following information:
In 4:
To improve hiring procedures in Brazil, one can analyze best practice manuals from international institutions, for example [10].
And
As compared to the international state of the practice, we can see that the ideal hiring procedures do not exist yet. We could turn to best practice manuals from universities in the United States [10], yet need to keep in mind that there is systematic inequality related to prestige in the hiring procedures in the United States [11]. Nevertheless, we consider it important to highlight the shortcomings of the current state of hiring processes in Brazil, so that Brazilian academia can move towards fairer hiring procedures. As mentioned previously, the main influence of the outcome of the public tenders is on the budget available for graduate students – the students are thus most affected by the hiring decisions, and they have no voice in these procedures.
Alternatively, the author could (and I would very much urge them to) relate the Brazilian case to forms of evaluation that scholars of culture, organization and inequality have studied. That way it would be easier to understand and appreciate, for an international audience, the (non-meritocratic) practices in Brazilian academia.
This is a great suggestion. We want back to the literature, and analysed a number of articles related to forms of evaluation from the perspective of scholars of culture, organization and inequality. The findings of this additional review of the literature lie at the basis of additional discussions:
Section 5:
As compared to the international state of the practice, we can see that the ideal hiring procedures do not exist yet. We could turn to best practice manuals from universities in the United States [10], yet need to keep in mind that there is systematic inequality related to prestige in the hiring procedures in the United States [11]. Nevertheless, we consider it important to highlight the shortcomings of the current state of hiring processes in Brazil, so that Brazilian academia can move towards fairer hiring procedures. As mentioned previously, the main influence of the outcome of the public tenders is on the budget available for graduate students – the students are thus most affected by the hiring decisions, and they have no voice in these procedures.
When developing improved methods for hiring faculty in Brazil, we should learn from the experience and the international state of the practice. At the same time, we should consider the unique cultural and social aspects of academia in Brazil, as earlier research comparing academia between two countries (United States and the Czech Republic) showed that there is no “one size fits all” at an international level [12]. One issue hiring panels should be aware of, is related to minorities. This problem lies deeper than just our personal biases, as the current status is one of systematic methodological disqualifications of contributions from members of underrepresented social groups [13]. As Saul remarked, this issue directly influences hiring panels: “[m]any admissions and hiring committees have a commitment to improving gender balance and perhaps even to choosing a woman over an equally qualified man but implicit biases may well prevent them from seeing which women are equally qualified.”As a solution to this deeply rooted problem, Leuschner [13] proposes mechanical methods. In terms of hiring, she proposes quota for hiring women and shows that affirmative action programs have been and are helpful.
As is, the article fails to engage with (or cite) critical scrutiny of meritocratic systems, forms of measurement, accountability and evaluation, in practice and in comparison with other countries where such systems are in effect. See for instance Michèle Lamont's work on the "Sociology of Valuation and Evaluation" (2012) and her "How Professors Think" (2010). It also sidesteps important scholarly debates about the desirability of meritocracy in the first place. For recent contributions to this debate, see Jo Litter's "Against Meritocracy" (2017) or my own work on the "Unfulfillable Promise of Meritocracy" (2016).
Thank you for pointing out this flaw in our article. We indeed used meritocracy as synonymous for finding the best candidate for the vacancy - without addressing the shortcomings of meritocractic systems. We have returned to the literature to add relevant discussions on the desirability of meritocracy in academic hiring procedures.
We have added the following paragraph in section 5, to address aspects of meritocracy and the neoliberal academic sphere:
The goal of improved hiring procedures should be finding the best person for the vacancy. While we have focused in this work on meritocratic procedures, we should here admit to the flaws of meritocracy itself. As Leuschner [44] pointed out, meritocracy has resulted in the self-production of elite groups and fails to adhere to pluralistic standards. At its worst, meritocracy is elitist and inequitable, obstructing consent and cohesion in society [45]. Paradoxically, opportunities for merit are themselves determined by non-meritocratic factors (talent is not distributed in a meritocratic way) [46], and the definition of “merit” itself implicitly favors some groups of society over others. Mijs [46] thus argues that meritocracy itself is an unfulfillable promise. Instead, we should strive to adhere to the principles of justice, need and equality in academia. While his analysis focused on higher education, and questions whether universities should train students for the labor market or build character and citizenship, further research seems necessary to identify how the principles of justice, need and equality can also be included during the academic hiring procedures to create a level playing field.
The comments in the previous paragraphs related to the obstacles minorities face, should be more than “good to know.” The guidelines for hiring procedures in Brazil should directly address these challenges, and the mechanical methods proposed by Leuschner [44] could be used in the context of Brazilian academia. From this perspective, the evaluation of a researcher’s CV as outlined in section 4 should also be evaluated from the gender perspective. Nielsen [47] shows that bibliometrics used as a driving factor during hiring procedures perpetuate existing gender inequalities in academia. We should also make sure that reduced productivity during and after maternity leave does not reflect negatively upon the evaluation of the CV of a woman with children (to remove the so-called “motherhood penalty”). His solution is then articulated as follows: “my suggestion therefore is to always supplement the use of quantitative proxies for merit with in-depth and systematic qualitative considerations about variations in expertise, experience, activities and career progression along gendered lines; even when comparing large numbers of researchers. Such an approach could help render visible some of the potential gender biases related to the use of quantitative performance metrics, hereby making academic recruitment and selection processes less gendered in their stratifying outcomes.” The neoliberal reality of today’s university landscape also poses significant limitations to researchers [48,49], which are difficult to evaluate and taken into consideration during hiring procedures.
For all good intentions on developing the best hiring procedures, we should keep in mind that human emotions play a role in these procedures [50]. These extra-cognitive emotional aspects of the hiring procedures are not well-known nor researched. We should be aware of these, how they influence how favourable we rate a candidate, and how we could possibly quantify these emotional aspects. Again, this knowledge could be used in the future to improve hiring procedures.
As for the organization of the piece, it is not clear from the introduction to the article exactly why Brazilian academia doesn't function meritocratically. It would help the reader if the argument were introduced prior to the extensive discussion of the current state of hiring processes in Brazil. That way, readers know what to look for when reading the section (i.e. signs of non-meritocratic evaluation).
We extended the final paragraph from the introduction to explicitly mention why the current system is not meritocratic:
This article aims to describe, analyze, and propose alternatives for two critical ‘obsessions’ taking place in Brazilian academia for a long time: (i) the way candidates are assessed in public tenders for faculty positions, and (ii) the mandatory Lattes CV. The current state of the practice for hiring faculty in Brazil in non-meritocratic for the following reasons: i) the written exams that form part of the evaluation procedure are standardized test for which privileged candidates can purchase forms of training [4]; ii) although the extensive form of hiring faculty, panel members from inside the faculty can manipulate the outcome [7], and iii) peculiarities of the Brazilian system, such as the use of Lattes CV and the fact that all exams are in Portuguese, make it difficult for foreigners to apply to positions in Brazil [8]. In the authors’ opinion, the structure of the public tenders and the use of the Lattes CV platform are seriously jeopardizing Brazil’s development and pluralism in science and higher education – two major pillars of any society.
I understand that the author needs to devote some space to discussing the current system of evaluation, but I am not convinced this discussion needs to be so extensive. This article was submitted as an opinion piece. Therefore I would expect the majority of it to introduce, develop and convey a clear argument. Instead, pages 1 through 11 (of 13) are an elaborate description of current practices. It escapes me why the author needs to present 14 (!) figures (Figs 2-15) to illustrate points that could simply be made in a few lines of text. That leaves less than 2 pages for the author to argue what exactly is wrong with the current system of evaluation, and to discuss what a better would look like, how it would address the shortcomings of the current system, and how it would remedy the situation.
The revised version of the manuscript is now more balanced in this sense: the introduction is +- 1 page, the introduction of the Brazilian situation is 2 pages, the analysis is 2 pages, the proposal is 1 page and then the discussion 2 pages. Thank you for pointing out this imbalance in the original manuscript.
References
1. Abambres, M.; Ribeiro, T.; Sousa, A.; Lantsoght, E.O.L. Ineffective obsessions in brazilian academia and proposals towards meritocracy. OSF Preprints 2018.
2. Barham, E. Elizabeth joan barham’s lattes cv. https://archive.is/37K6s (English) and https://archive.fo/A0bEt (Portuguese) - archived versions of the webpage (September 4th 2018),
3. Marchesi, I.M. Isadora moraes marchesi’s lattes cv. Archived at https://archive.fo/189pB (September 3rd 2018),
4. Missau, F.C. Fabiana cristina missau’s lattes cv. Archived at https://archive.fo/ILM4p (September 3rd 2018),
5. Pizzolatti, M.G.; Mendes, B.G.; Soldi, C.; Missau, F.C.; Bortoluzzi, J.H.; Carasek, E. Analysis of volatile compounds released from flowers and roots of polygala cyparissias and polygala paniculata by headspace/spme. Journal of Essential Oil Research 2009, 21, 255-258.
6. Franco, J.L.; Braga, H.C.; Stringari, J.; Missau, F.C.; Posser, T.; Mendes, B.G.; Leal, R.B.; Santos, A.R.S.; Dafre, A.L.; Pizzolatti, M.G., et al. Mercurial-induced hydrogen peroxide generation in mouse brain mitochondria: Protective effects of quercetin. Chemical Research in Toxicology 2007, 20, 1919-1926.
7. Kiill, C.P. Charlene priscila kiill’s lattes cv. Archived at https://archive.fo/6VfA5 (September 4th 2018),
8. Miguel, A.D. Alcebiades diniz miguel’s lattes cv. Archived at https://archive.fo/h5NEm (September 4th 2018),
9. Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq). Plataforma lattes – cadastrar novo currículo. Archived at https://archive.fo/9ZHGY (September 3rd 2018),
10. Columbia University. Guide to best practices in faculty search and hiring; Columbia University: New York, New York, 2016.
11. Clauset, A.; Arbesman, S.; Larremore, D.B. Systematic inequality and hierarchy in faculty hiring networks. Science Advances 2015, 1, e1400005.
12. Preiss, M.; Klein, H.A.; Levenburg, N.M.; Nohavova, A. A cross-country evaluation of cheating in academia—a comparison of data from the us and the czech republic. Journal of Academic Ethics 2013, 11, 157-167.
13. Leuschner, A. Social exclusion in academia through biases in methodological quality evaluation: On the situation of women in science and philosophy. Studies in History and Philosophy of Science Part A 2015, 54, 56-63.
14. Nielsen, M.W. Scientific performance assessments through a gender lens - a case study on evaluation and selection practices in academia Science & Technology Studies 2018, 31.
15. Richter, M.; Hostettler, U. Conducting commissioned research in neoliberal academia: The conditions evaluations impose on research practice. Current Sociology 2015, 63, 493-510.
16. Cannizzo, F. Tactical evaluations: Everyday neoliberalism in academia. Journal of Sociology 2018, 54, 77-91.
Reviewer 2 Report
The study examines the system of academic position placement in Brazil. The manuscript is well written. It is a clear improvement of earlier studies by Siqueira et al. (2012), Araujo (2016) and the working paper by the same author (Arauo, 2019. An Ethnography of Academic Job Position-Filling: The Case of the Brazilian "Concursos"). With the use of secondary data, the author shows that Brazilian academic market can become more open and competitive if certain measures are taken. The analysis suggests that if certain hurdles are taken, the huge potential of Brazilian academic market can be unleashed.
Author Response
Reviewer 2
The study examines the system of academic position placement in Brazil. The manuscript is well written. It is a clear improvement of earlier studies by Siqueira et al. (2012), Araujo (2016) and the working paper by the same author (Arauo, 2019. An Ethnography of Academic Job Position-Filling: The Case of the Brazilian "Concursos"). With the use of secondary data, the author shows that Brazilian academic market can become more open and competitive if certain measures are taken. The analysis suggests that if certain hurdles are taken, the huge potential of Brazilian academic market can be unleashed.
Thank you for your encouraging comments. We have prepared a revised version of the manuscript to address the concerns of reviewer 1. In addition, we have carefully read Araujo 2019’s working paper, and implemented some of his findings in this revised version, as we found them very relevant to our work.
Round 2
Reviewer 1 Report
I commend the author(s) for rewriting key parts of their opinion piece to more clearly and convincingly convey their argument. I have just two comments.
While the article is generally very well written, it could do with (minor) language editing and a thorough spell check -- e.g. the word "Albeit" in the abstract, the sentence "the current state of .... *in* meritocratic" and "are standardized *test*", the word "although" on p.1.
In Fig 1, I'd remove the incomplete data for 2015 which may be confusing for the lazy reader who quickly glimpses the graph and fails to see the details of the trend.
I look forward to seeing the article in print!
Author Response
I commend the author(s) for rewriting key parts of their opinion piece to more clearly and convincingly convey their argument. I have just two comments.
Thank you for your kind words.
While the article is generally very well written, it could do with (minor) language editing and a thorough spell check -- e.g. the word "Albeit" in the abstract, the sentence "the current state of .... *in* meritocratic" and "are standardized *test*", the word "although" on p.1.
We’ve done a final round of proofreading before resubmitting the manuscript now.
In Fig 1, I'd remove the incomplete data for 2015 which may be confusing for the lazy reader who quickly glimpses the graph and fails to see the details of the trend.
I agree, figure has been updated:
I look forward to seeing the article in print!
Thanks for your very helpful reviews!
