Is the external audit report useful for bankruptcy prediction ? Evidence using artificial intelligence

Despite the number of studies on bankruptcy prediction using financial ratios, very little is known about how external audit information can contribute to anticipating financial distress. A handful of papers show that a combination of ratios and audit data can provide significant predictive purposes, but a recent paper by Muñoz-Izquierdo et al. (2018) provided an 80% predictive accuracy solely by using the disclosures of audit reports. We complement this study. Applying an artificial intelligence method (the PART algorithm), we examine the predictive ability of more easily extracted information from the report and suggest a practical implication for each user. Simply by (1) finding the audit opinion, (2) identifying if a matter section exist, (3) and the number of comments disclosed, then any user may predict a bankruptcy situation with the same accuracy as if they had scrutinised the whole report. In addition, we also provide an extended literature review about previous studies on the interaction between bankruptcy prediction and the external audit information.


Introduction
During the last 70 years, the development of bankruptcy prediction models has been a challenging research topic worldwide (Sun et al. 2014;Cultrera and Brédart 2016;Altman 2018).Despite the number of studies in this field, recent literature indicates there is still a need to improve the accuracy of the prediction models (Balcaen and Ooghe 2006;Du Jardin 2015;Bauweraerts 2016) and there is a call to apply different sources of data and techniques, such as non-parametric techniques, to tackle this issue (Calderon and Cheh 2002;Zięba et al. 2016;Amani and Fadlalla 2017).This paper aimed to contribute to this line of research by examining the bankruptcy predictive ability of the external audit report using artificial intelligence methodologies.
The objective of this study was two-fold.First, it examined whether or not the audit report provided significant explanatory power when predicting bankruptcy by applying three artificial intelligence classifiers: the Projective Adaptive Resonance Theory (PART) algorithm, random forest, and support vector machine.Second, by testing the predictive ability of the audit report, a practical implication could be provided for each user, as it was easy to extract information from the report to make decisions related to a firm's viability.It was proposed that the three variables found without any difficulty in the audit report-the audit opinion, the type of paragraphs disclosed, and the number of comments included in those paragraphs-are significant variables to improve the detection of bankruptcy.
While considerable research has been devoted to bankruptcy prediction using financial ratios (Altman et al. 2017), very little is known about how external audit information can contribute to anticipating a firm's precarious financial condition.Considering that the auditing profession ensures the quality of financial statements with the issuance of an opinion in the audit report (Lennox 1999), it seems reasonable to expect that the information extracted from the report can represent a good indicator of a firm's insolvency.
Only a handful of papers have combined accounting ratios and some audit variables for predictive purposes (Altman and McGough 1974;Hopwood et al. 1989;Laitinen and Laitinen 2009;Altman et al. 2010;Piñeiro-Sánchez et al. 2013;Cenciarelli et al. 2018).Altman and McGough (1974) were pioneers in using audit information to predict bankruptcy.Hopwood et al. (1989) focused on some audit qualifications, finding that there was an association between bankruptcy and consistency, going concern, and other "subject to" qualifications.Altman et al. (2010) suggested that the audit opinion had high predictive power, and that firms with audit qualifications, such as severe qualifications or going concern, were more likely to fail, since the auditor was questioning its viability.Others, such as Piñeiro-Sánchez et al. (2013), examined the predictive ability of different auditor characteristics.According to their evidence, the auditor rotation, qualified reports, and noncompliance with deadlines (regarding approval and filing of financial statements) presented relevant differences between bankrupt and non-bankrupt firms.Similarly, Cenciarelli et al. (2018) posited that firms audited by industry-expert, large, and long-tenured auditors were less likely to fail.They also found that prediction models were more effective when the auditor attributed complementary financial characteristics.
A paper by Muñoz-Izquierdo et al. (2019) is the only one that has not combined audit data with accounting ratios to explain business failures.They anticipated the causes of business failure by solely applying external audit variables, namely, the narrative disclosures of the audit report.Narrative disclosures are the comments that auditors mention in the paragraphs regarding accounting and environmental issues that occur in the audited firms.They suggested that failure can be explained by specific internal causes, such as asset valuation and a firm's real and potential debts, as well as the external circumstances like the regulatory framework or changes in the market.However, obtaining this information from the audit report is not quick and easy.It takes time to interpret every comment, and users should have extended accounting or auditing knowledge to be able to understand the content of all disclosures.Therefore, the current paper contributes by simplifying the task of detecting business failure for the users of the audit report, so that stakeholders interested in the business's future viability will not need to scrutinize the complete report.
The current paper is a follow-up study to that of Muñoz-Izquierdo et al. (2019), and also uses information from the audit report in isolation (not combined with financial data) for bankruptcy estimation purposes.As previously stated, the aim of this paper was different because it attempted to anticipate bankruptcy, that is, to find an accurate percentage of prediction and not the particular causes of the bankruptcy situation.Furthermore, the aim was to do so by using the minimum number of variables extracted from the audit report, and those easiest to identify by any user.New variables were introduced that users could obtain without being experts on the valuation of assets and liabilities, policy changes, or the markets in which the firms operate.
The results of this study suggest that three easily extracted audit report variables could provide a reasonably accurate bankruptcy predictive power, similar to the one obtained in prior works that combined accounting with auditing variables.These variables are the audit opinion, the matter sections disclosed in the audit reports, and the number of comments included in the matter sections and qualification paragraphs.Thus, the findings indicate that there is a positive relationship between the propensity to file for bankruptcy and the issuance of qualified reports with a high number of comments disclosed by the auditors.There was even a positive relationship between the bankruptcy filing and unqualified reports with comments disclosed in the matter paragraphs.Since the audit report is a standardized and regulated document, these variables are easily detected by any user without the need to scrutinize the whole report or having any special accounting or auditing knowledge.This empirical evidence shows a practical implication for the users of the audit reporta simple identification of the opinion, the type of paragraphs disclosed, and the number of auditors' comments represents a substantive approach to detect and estimate bankruptcy.A contribution to the bankruptcy prediction literature was made by applying three different artificial intelligence methodologies, which have never applied before with auditing variables.Lastly, this study also provides an extensive review of the research into the interaction between bankruptcy and auditing research, thus contributing to both areas of knowledge.

Literature Review and Research Question
Auditors are required to express in the audit report if the likelihood of default is high during the one-year period following the issuance of the document (McKee 2003).Although the role of auditors is not expected to be a predictor of bankruptcy, stakeholders might be dissatisfied if a firm fails immediately after receiving an unqualified opinion.This issue has persisted for many years, and the auditing literature has considered audit quality from the viewpoint of the users of the financial statements.Indeed, during the last global financial crisis, companies sought financial support within a short period after receiving an unqualified opinion (Sikka 2009).Since then, researchers have paid more attention to the association between bankruptcy and the auditing profession, suggesting that the propensity to issue going concern opinions prior to bankruptcy increases after a crisis (Geiger et al. 2014).However, evidence about the auditors' ability to warn investors about upcoming failures is not unanimous, as other prior studies have suggested that investors perceive audit reports as informative (Dopuch et al. 1987;Piñeiro-Sánchez et al. 2013), emphasizing that audit opinions provide explanatory power for predicting bankruptcy (Kim et al. 2008;Altman et al. 2010).
A systematic literature review was conducted to present the state-of-the-art literature regarding the relationship between the external audit information and the prediction of bankruptcy, and to show the contribution of the current study.

Systematic Literature Review: Scope of the Review
All of the academic papers from the Institute for Scientific Information (ISI) Web of Knowledge database as of December 2018 were compiled according to two keywords: "audit" and "bankruptcy".The preliminary search identified many papers focused on the broad domains of bankruptcy.The process of reading the abstracts and introductions of the articles led to the further elimination of studies outside the scope of this research.After filtering the results, 86 articles on the integration between auditing and bankruptcy remained for this study.With the purpose of systematizing and organizing the literature, these papers were assembled and classified according to their main themes, by dividing them into four lines of research: the effects of auditing; audit quality and auditor independence; audit opinion prediction; and bankruptcy prediction using auditing.The contribution of the current study will fall into this fourth category, although all four lines are absolutely interconnected.A list of the reviewed articles appears in Table 1, explaining their samples, methodologies, and key findings.Qualified opinions issued by auditors can have an impact in different ways.Audited financial statements with qualifications, or even Emphasis of Matter paragraphs, should be reviewed more carefully than unqualified audit reports (Stanisic et al. 2013).This advice is especially salient when qualifications are related to going concern uncertainties.For example, loan contracts include information regarding qualifications (Chen et al. 2016).However, it is not only qualifications that impact debt contracting.Going concern opinions can be interpreted as a communication of risk to the equity market (Blay et al. 2011), as they have an adverse effect on market prices (Menon and Williams 1994) and can cause an increase in the subsequent cost of capital (Amin et al. 2014).In particular, Kausar et al. (2017) found that the reaction to a first-time going concern opinion is more adverse in a creditor-friendly regime, such as that in the UK, than in a debtor-friendly regime, such as that in the US.However, recent evidence has shown that market reactions to going concern opinions are statistically weaker than has been suggested in prior literature (Myers et al. 2018).

Line of Research: Auditor Independence and Audit Quality
As evidenced in the review of the research, many studies have evaluated audit quality.Audit quality is one of the most relevant issues facing the auditing profession, and it depends on the auditor's competence and independence (Vanstraelen 2000).Competence relies on the auditor's knowledge and technological capabilities, and prior studies have shown that auditors can discover errors in the accounting system (Kida 1980).
Since audit quality is crucial for the effectiveness of the auditing profession (Vanstraelen 2000), factors that can impact independence, such as the pricing of audit services (Vanstraelen 2002;Robinson 2008;Callaghan et al. 2009;Stanley 2011;Basioudis et al. 2012;Hope et al. 2017), auditor tenure (Geiger and Raghunandan 2002;Carey and Simnett 2006;Knechel and Vanstraelen 2007;Read and Yezegel 2016), audit report lags (Shu et al. 2015), auditors' decisions to resign (Krishnan and Krishnan 1997), auditor switching (Schwartz and Menon 1985;Carey et al. 2008), or the composition of the audit committee (Carcello andNeal 2000, 2003) have been extensively studied.The issuance of going concern opinions has also been accepted as a measure of auditor independence and quality throughout the literature (Carey and Simnett 2006;Robinson 2008;DeFond and Zhang 2014;Koh and Lee 2017;Berglund et al. 2018).As auditor independence is difficult to assess directly, other common proxies used in the literature have been linked to the characteristics of clients, such as their size (McKeown et al. 1991;Aguiar-Díaz and Díaz-Díaz 2015), financial condition (Pratt and Stice 1994;Louwers et al. 1999;Muñoz-Izquierdo et al. 2017), strategies (Chen et al. 2017), corporate payouts (Cao et al. 2017), and CEO incentives (Koh and Lee 2017).
Although the empirical evidence is not unanimous, many studies have supported auditor independence.For instance, Louwers et al. (1999) confirmed that their assessments focused on the client's financial condition and other indicators of financial distress, and not on factors related to litigation risk or loss of clients.Additionally, DeFond et al. (2002) did not find an association between non-audit fees and impairment of independence, and Knechel and Vanstraelen (2007) stated that independence was maintained over time.
Additionally, audit quality has received increased attention after regulatory changes, corporate collapses, or economic crises.Carcello et al. (1995) found that qualified opinions increased after the issuance of Statement of Auditing Standard (SAS) No. 34 and not after SAS No. 59, while Feldmann and Read (2010) argued that the likelihood of going concern opinions increased sharply immediately after the Enron collapse, and Carey et al. (2012) showed that the likelihood of these opinions returned to pre-Enron level shortly thereafter.Pedrosa Rodríguez and López-Corrales (2018) posited that the propensity to issue going concern opinions increased during the recent global financial crisis in Spain, and similar evidence was found using publicly listed Australian firms by Carson et al. (2016).After the crisis, Geiger et al. (2014) indicated that going concern opinions kept increasing, but Read and Yezegel (2018) pointed out that they decreased to the crisis levels after two post recovery periods.
All this evidence shows that qualifications are interpreted as external signals of potential financial instabilities (Buchman and Collins 1998), and they also communicate information about audit quality (Piñeiro-Sánchez et al. 2013).The current study added more evidence to this line of research about the independence of auditors as they disclosed signals of bankruptcy in their audit reports before, during, and after the global financial crisis.

Line of Research: Audit Opinion Prediction
Prior research has shown that auditors signal bankruptcy in approximately half of the cases in which companies subsequently file for bankruptcy (McKee 2003;Laitinen and Laitinen 2009).Ittonen et al. (2017) suggested that going concern opinions should be issued when there is an 8% chance that the firm is bankrupt.McKeown et al. (1991) found that auditors issued less qualified opinions to larger firms, firms with shorter time lags between the fiscal year end and the audit opinion dates, and when the probability of bankruptcy was ambiguous.Additionally, empirical evidence has also shown that accounting data can be used to identify qualified opinions.Different methodologies, such as logistic regression (Zdolsek and Jagric 2011), neural networks (Lenard et al. 1995), or inductive inferencing algorithms (McKee 1995) have been applied for this purpose.

Line of Research: Bankruptcy Prediction Using Auditing
Prior works have found associations between audit quality, financial distress, and qualified reports (Blay 2005;Arnedo-Ajona et al. 2012).However, there seems to be no consensus on the accuracy of auditing information for predicting bankruptcy or the survival of firms.On one hand, it can be argued that auditors are not able to predict either bankruptcy filings or resolutions (Casterella et al. 2000).On the other hand, some researchers have agreed with the idea that differences between failing and non-failing firms might be detected using financial ratios and audit data (Van Peursem and Chan 2014).Some audit information contributes to an increase in default prediction power, such as the recurrent issuance of qualified opinions, high auditor rotation or lack of meetings, and the lack of independence of the audit committee (Kim et al. 2008;Altman et al. 2010;Piñeiro-Sánchez et al. 2012, 2013;Appiah and Amon 2017;Armeanu et al. 2017).The present study contributes to this research line by examining the predictive ability of information easily extracted from the audit report, which has resulted in variables never tested before in prediction literature.

Research Question Development
Although the auditing profession ensures the credibility of a firm's financial statements in the audit report, it seems that the audit report has not been well studied as a measure of bankruptcy prediction.Research opportunities remain in this area, and the present study has attempted to fill this gap.
The main role of external auditors is to guarantee the reliability of the financial statements presented by any company.Thus, it seems that the information included in audit reports is likely to improve the accuracy of bankruptcy prediction modeling.Based on this reasoning, audit report information should be incorporated as explanatory variables in statistical models.With this in mind, the research question to be investigated is specified as follows: Research question: What information in the external audit report can accurately predict bankruptcy?
It is expected that information from the audit report will provide accurate bankruptcy predictive power, and therefore this work may complement previous studies.This is the first study to have applied the audit report in isolation as a bankruptcy predictor.Additionally, due to the vast number of audit report users (who do not necessarily need to be experts in auditing), the explanatory variables that could best anticipate the event of bankruptcy have been simplified.The variables investigated were the most accurate variables and were also the easiest to detect by any user of the report interested in a firm's insolvency.Finally, this study used novel techniques in the area of auditing for bankruptcy prediction: the non-parametric methodologies of the PART algorithm, random forest, and support vector machine.

Sample and the Dependent Variable
As in previous bankruptcy studies, the present work applied a matched sample of bankrupt and non-bankrupt firms (Schwartz and Menon 1985;Carcello and Neal 2003;Knechel and Vanstraelen 2007;Blay et al. 2011).An ad-hoc sample of 808 Spanish private non-financial audited firms was manually selected and prepared from the entire population of firms in the Spanish Bureau Van Dijk (hereafter BVD) database.The sample was evenly divided between bankrupt and non-bankrupt firms.
A company was considered to be bankrupt if it had filed for bankruptcy protection (Piñeiro-Sánchez et al. 2013).Thus, all bankrupt firms included in the database that had filed for bankruptcy proceedings as of 30 January 2015 were identified (1821 firms), and their financial and audit data extracted from this source for the fiscal year prior to the bankruptcy filing date.The filing dates were also manually collected from the "Registro Público Concursal" (the official Spanish source of bankruptcy data), as they did not appear in the BVD database.All of the bankruptcy filing dates along the sample belonged to the 2004-2014 period.Out of the 1821 firms, the final bankrupt sample consisted of 404 observations, with the remaining firms being removed due to missing data.
Subsequently, each bankrupt observation was manually matched with a non-bankrupt firm.For each non-bankrupt firm, financial and audit data was extracted from the BVD database 1 for the fiscal year prior to the bankruptcy filing date of its bankrupt pair.The matching procedure was performed 1 The audit data available in the BVD database is a section called the "Auditor's opinion".It consists of a maximum of 991 characters of the report representing emphasis paragraphs, qualification paragraphs, or both.From this field, we extracted the audit opinion (unqualified or qualified), the type of paragraphs disclosed, and the number of comments in those paragraphs.We identified the comments manually by reading and labeling the content of each report.according to year, firm size-using the measure of total assets-and industry, as in the prior literature (Schwartz and Menon 1985;Knechel and Vanstraelen 2007).Therefore, the process resulted in a total sample of 808 firms: 404 bankrupt, which were then matched with 404 non-bankrupt firms.
Following prior literature, a dummy variable was used (BANKRUPT) as the dependent variable (Piñeiro-Sánchez et al. 2013), because it represents the start of court bankruptcy proceedings, and therefore provides a legal, objective, and narrow definition of bankruptcy.The variable BANKRUPT takes the value of 1 if the firm has filed for bankruptcy proceedings; otherwise, it is 0.

Independent Variables: Audit Report Variables
In this paper, the independent (or explanatory) variables of bankruptcy were related to the audit report.A summarized definition of all variables is presented in Table 2.The first variable tested was the audit opinion issued in the period prior to bankruptcy, or in the corresponding year for the nonbankrupt firms.The role of the opinion in predicting bankruptcy was examined, using a dummy variable (AUDIT_OP) with the following two categories: unqualified (0) and qualified (1) opinion.An opinion is unqualified when the auditor determines that the financial statement gives a true and fair view in accordance with the reporting framework used for the preparation of the annual accounts.An opinion is qualified when the auditor points out that the financial statement is materially misstated, or when the auditor is unable to obtain evidence regarding any transaction or disclosure.The audit opinion dummy variable was chosen as it has been the most studied audit variable in prior bankruptcy literature (Altman and McGough 1974;Laitinen and Laitinen 2009).It was expected that the opinion would contribute to distinguishing between bankrupt and nonbankrupt firms, according to prior findings (Piñeiro-Sánchez et al. 2013).According to Altman et al. (2010), there is a positive relationship between the likelihood of filing for bankruptcy and the issuance of a qualified report.
The second and third independent variables of this study represent the type of paragraph that auditors include in the report (if any).The dummy variable EMPHASIS takes the value of 1 if an Emphasis of Matter paragraph is included in the report, and 0 otherwise.Auditors disclose matter paragraphs when it is necessary to mention significant uncertainties, appropriately included in the notes to the financial statements.Emphasis sections point out fundamental matters for the users' understanding.These paragraphs do not qualify as the opinion, but represent concerns for the auditor and relevant signals regarding the likelihood of the firm's survival (Blay 2005).
Similarly, the categorical variable SCOPE_VIOLATIONS has a value of 1 when either a qualification regarding a scope violation or a qualification due to a violation of generally accepted accounting principles (GAAP) is included; 2 when both qualifications are issued in the report; and 0 when none are disclosed.It was hypothesized that the inclusion of any of these paragraphs in the audit report reveals a sign of possible bankruptcy, as indicated by the auditor.This expectation is in line with the rationale of the AUDIT_OP variable, because prior evidence suggests that prediction accuracy increases once the audit report (unqualified or qualified) is included in the estimation models (Altman et al. 2010).
Furthermore, not only did was the type of paragraphs disclosed in the audit report tested, but also the number of comments mentioned by auditors in them.A categorical variable (NUMBER_COMM) was incorporated that had the value of 0 when no comments were disclosed, and otherwise 1 to 6 in agreement with the number of comments shown.Six comments was the maximum number found in a few firms in the database.Most comments were related to accounting elements, such as the valuation of assets, liabilities, accumulated losses, or negative working capital, but auditors also write about regulatory issues, concerns regarding markets in which firms operate, or companies being involved in legal processes 2 .A positive association was hypothesized between 2 To illustrate the process of identifying the comments in the audit report, here is one example of a firm included in our sample.This firm filed for bankruptcy in 2008 and issued an unqualified audit report with an emphasis on the matter paragraph in the same year.According to the BVD database, this paragraph said: "The company incurred a net loss of 3128 thousand of euros during the year ended 31 December 2008.Also, as of that date, its current liabilities exceeded its total assets so that the company may be unable an increase in the number of comments and the propensity to file for bankruptcy, which was based on the idea that more comments might point to higher chances of bankruptcy and more concerns for users when reading the audit report.A contribution was made to the literature by adding this variable to the estimation models as, to our knowledge, the predictive power of the accumulation of auditor comments has not been tested thus far.
Table 2. Classification and description of the audit report variables in this study.

AUDIT OPINION AUDIT_OP
Dummy variable with a value of 1 if the auditor's report is qualified, and 0 if it is unqualified.

TYPE OF PARAGRAPH EMPHASIS
Dummy variable with a value of 1 if the auditor's report has an Emphasis of Matter paragraph, 0 otherwise.

SCOPE_VIOLATIONS
Categorical variable with a value of 0 if no qualifications appear in the report, 1 if the audit report has a qualification due to a scope limitation or due a GAAP violation, and 2 if the report shows both.

NUMBER OF COMMENTS NUMBER_COMM
Categorical variable with a value of 0 if no comments are disclosed in the report, and 1 to 6 according to the number of comments shown.
Table 2 summarizes the independent variables used in this study and the audit report variables.

Artificial Intelligence Methodology: The PART Algorithm
Despite the popularity of parametric models (such as the commonly used multivariate discriminant analysis or the logit regression model) for bankruptcy prediction, another current research approach to tackle financial problems is based on non-parametric techniques such as artificial intelligence (Calderon and Cheh 2002;Zięba et al. 2016;Amani and Fadlalla 2017).While parametric techniques have shown satisfactory results, they have a drawback when applied to real bankruptcy data, as some hypotheses required are not satisfied (especially if outliers exist).However, artificial intelligence techniques, which are non-parametric, do not require the data to satisfy any concrete assumptions.This advantage allows them to predict bankruptcy more accurately.Indeed, artificial intelligence methods have already been used to explain insolvency risk (Kumar and Ravi 2007;Wu 2010;Koyuncugil and Ozgulbas 2012;Kirkos 2015) and prior research has also applied these methods for anticipating going concern issues (Lenard et al. 1995;Yeh et al. 2014).
In this paper, the PART algorithm was used as the main artificial intelligence technique.Additionally, the validity of the results was tested by applying two other machine learning methodologies, namely random forest and support vector machine, as robustness checks.
The PART algorithm is a rule induction classifier developed by Frank and Witten (1998) that generates rules by incorporating a modified form of the J48 decision tree and eliminating some of the paths found in an initial decision tree structure.This algorithm builds partial decision trees instead to convert its assets into funds that can be used to meet its financial obligations.Additionally, as stated in Note 16 in the financial statements, the company holds significant investments in group companies under serious financial distress.These events or conditions indicate that a material uncertainty exists that may cast significant doubt on the company's ability to continue as a going concern.Although the financial statements were prepared on a going concern basis, the company may be unable to realize its assets, specially the tax credit, and discharge its liabilities in the normal course of business."Therefore, the comments of the report basically total six in number: accumulated losses, negative working capital, short-term and long-term financial investments, going concern, and deferred tax assets.These are the comments that we took into consideration in our sample, so the independent variable NUMBER_COMM for this firm took the value of six.For this firm, the other three independent dummy variables of the study took the following values: AUDIT_OP value of 0 (unqualified), EMPHASIS takes the value of 1 (matter paragraph disclosed), and SCOPE_VIOLATIONS is 0 (no qualifications). of fully explored ones.Once the algorithm finds the partial tree, the tree-building stops and a rule is generated with the leaf that represents the greatest number of situations.Due to the fact that it is based on partial decision trees, the main advantage of the PART algorithm is its simplicity.However, its performance is similar to that of other machine learning algorithms.Consequently, this explicative technique was chosen because of the simplicity of its rules, with no loss of accuracy.The results are easier to interpret when compared to other classifiers as they are expressed in logical if/else statements (Díaz-Martínez et al. 2009).Additionally, the usefulness of this methodology has been corroborated in business studies (Camacho-Miñano et al. 2015), although it has not been applied before for bankruptcy prediction using audit report variables.
The rules created by the PART algorithm classify objects into decision classes depending on a series of variables or conditions.These rules are expressed in logical statements with the following form: IF < conditions are fulfilled > THEN < the object belongs to a given decision class >.
In this study, the objects were firms, the two decision classes were bankrupt and non-bankrupt, and the conditions were all audit report variables or independent variables.Thus, the PART algorithm was applied to classify firms (objects) into bankrupt and non-bankrupt (decision classes) depending on a set of audit report variables (conditions).

Summary Statistics
Summary statistics of the sample are provided in Table 3. Bankrupt and non-bankrupt firms had the same frequency per industry due to the matching technique used.The total sample included a variety of industries, with construction and real-estate firms making up the largest group (35%), mainly due to the impact of the housing bubble during the global financial crisis in Spain (Conefrey and Gerald 2010).Accordingly, along with the matching procedure, this study controlled for firm size in its statistical analyses, by measuring by the firms' total assets in thousands of euros.
Regarding the financial condition of the sample, bankrupt companies are generally more illiquid (lower working capital to total assets ratio), less profitable (lower return on assets ratio), and more leveraged (higher book value of equity to total liabilities ratio) than non-bankrupt firms, which is consistent with prior studies (Bellovary et al. 2007;Tascón-Fernández and Castaño-Gutiérrez 2012;Altman et al. 2017).Results of the estimation models using the PART algorithm are presented below.Model 1 is based on the audit opinion (AUDIT_OP) as the only explanatory variable of bankruptcy (see Figure 1).The classification power of the model was 68.20% (31.80% incorrectly classified cases).According to this model, this dummy variable classified 513 firms as non-bankrupt with 36% error (183 errors) and 295 as bankrupt (25% error).This result suggests that, in isolation, the audit opinion anticipated the financial condition of two-thirds of the sample accurately.Although this prediction does not appear to be very precise, it is relevant, considering that the model only includes one variable.This evidence complements previous studies where the audit opinion was a bankruptcy predictor (Hopwood et al. 1989;Laitinen and Laitinen 2009;Altman et al. 2010;Piñeiro-Sánchez et al. 2012, 2013) and those that interpreted qualifications as external signals of potential financial instabilities (Buchman and Collins 1998).However, this variable solely has never before been used 23 March 2019, probably due to the number of incorrectly classified cases obtained.In the following models, more explanatory variables were included, to increase the predictive accuracy.

PART Algorithm: Model 2
As the prediction was relatively low in Model 1, Model 2 added the two dummy variables of the type of paragraphs disclosed in the audit report (EMPHASIS and SCOPE_VIOLATIONS) to the initial model, when those paragraphs existed in the firms (see Figure 2).As mentioned previously, auditors place their comments in the audit report, either in the Emphasis of Matter sections that do not qualify the opinion (EMPHASIS) or in qualification paragraphs (SCOPE_VIOLATIONS).In Model 2, the predictive power increased to 76.49% (23.51% incorrectly classified cases).This result showed the relevance of the type of paragraph for explaining bankruptcy, as the prediction improved by 8% when compared to Model 1.Even more importantly, it provided evidence of the importance of Matter sections when predicting bankruptcy.The existence of qualifications did not appear in the model as it is embedded in the qualified opinion (when the opinion is qualified, a qualification paragraph is disclaimed), that is, when the dummy variable AUDIT_OP takes the value of 1.
As per Model 2, if the audit opinion was unqualified (AUDIT_OP = 0), the viability of the firm was determined by the existence of a Matter section.Without an Emphasis of Matter paragraph, the model classified the firms as non-bankrupt (AUDIT_OP = 0 and EMPHASIS = 0) in 324 cases (17% error).Interestingly, even when the opinion was unqualified, the algorithm classified a firm as bankrupt when a Matter section was disclosed (AUDIT_OP = 0 and EMPHASIS = 1), and this rule codified 484 firms (28% error).This empirical evidence validates prior studies that suggest that auditors issued unqualified reports to some bankrupt firms during the recent global financial crisis (Sikka 2009).However, the role of external auditors during that period could not be fully questioned, as their main task was to guarantee the reliability of the firms' financial statements and, according to the results of this study, were at least issuing Matter sections emphasizing their financial concerns about soon-to-be-bankrupt firms.In Model 3, the number of comments in the paragraphs was aggregated to the PART algorithm as a new independent and categorical variable (see Figure 3).This model strongly codified firms as non-bankrupt when there were no paragraphs, so in the absence of comments NUMBER_COMM = 0.This rule classified 324 firms as non-bankrupt (17% error).The same classification was provided by the algorithm when one comment was disclosed but the opinion was unqualified, so the comment was disclosed in a Matter section (55 firms classified as non-bankrupt, 20% error).However, with one comment in a qualification paragraph (NUMBER_COMM = 1 and AUDIT_OP = 1), the classification already moved to bankruptcy (121 bankrupt firms, 40% error).From one comment onwards (NUMBER_COMM = 2; 3; 4), there was a prevailing discrimination toward bankruptcy with a low percentage of error.In line with prior studies that incorporated different qualifications in their estimations (Hopwood et al. 1989;Piñeiro-Sánchez et al. 2012), the inclusion of the number of comments increased the predictive ability of the model (in this study, the classification went up 5%, from 76.49% to 80.82%), indicating that if auditors disclose several concerns in their reports, it is very plausible that the viability of the firm is being questioned.Thus, these results indicate that there is a positive relationship between the propensity of a firm to file for bankruptcy and a high number of comments disclosed in the audit report.The evidence from this model complemented the prior literature because this is the first time in which the accumulation of auditors' comments was tested for bankruptcy predictive purposes.
In conclusion, after analyzing the findings of the three models, the research question could be answered.The audit report information-mainly consisting of the easily extracted variables of the audit opinion, the disclosure of a Matter paragraph, and the number of comments shown-represents accurate measures for predicting bankruptcy.Thus, there is a positive relationship between these audit report disclosures and the propensity of a firm to file for bankruptcy.

Robustness: Random Forest and SVM
To ensure the reliability of the empirical evidence in this study, two robustness checks were performed, using other artificial intelligence methodologies.These checks examined whether the correctly classified instances remained constant, using random forest and support vector machine.These two methods were chosen because they are widely applied in the bankruptcy and going concern prediction literature (Min and Lee 2005;Min et al. 2006;Yeh et al. 2014).Random Forest is a machine learning classifier that uses a combination of independent decision trees to model data (Yeh et al. 2014).The trees are built using bootstrap samples, not all data are applied to build a tree, and features are randomly selected at each node.Therefore, Random Forest is one of the most accurate and efficient classifiers, even in large datasets with missing data, as it generates an unbiased estimate of the generalization error as the forest building progresses.
Support vector machine is a learning algorithm that applies a linear model to implement nonlinear classes through vectors into a high-dimensional space (Min and Lee 2005).It finds a maximum margin hyperplane that provides the maximum separation between decision classes.The support vectors are the examples used for defining the classes, and are the closest to the hyperplane.Support vector machine has been used in bankruptcy prediction because the results are easy to analyze, less theory-driven and more data-driven, and more distribution-free than conventional statistical techniques (Min and Lee 2005).
Motivated by these studies, random forest and support vector machine were applied to check the robustness of the results of the PART algorithm.The untabulated evidence of this study suggests that the confusion matrixes obtained by the random forest and the support vector machine revealed the same percentages of correctly-classified cases as the PART algorithm.For Models 1, 2, and 3, the percentages of classification accuracy were 68.2%, 76.5%, and 80.8%, respectively.Therefore, according to other machine learning algorithms, the ability to anticipate bankruptcy for the audit report variables of this study was robust.When auditors issue their reports, users may pay attention to the type of opinion, the existence of a Matter paragraph, and other comments included in qualification paragraphs, as these pieces of information represent signs of a firm's bankruptcy.

General Conclusions
The aim of this paper was to examine the explanatory power of the external audit report when predicting the bankruptcy situation of firms.New prediction models were introduced using artificial intelligence methodologies-the PART algorithm, random forest, and support vector machinewhich had never been applied before in bankruptcy prediction models using data from audit reports.The evidence indicates that the information extracted from the audit report is useful for analyzing the probability of filing for bankruptcy and anticipating precarious financial conditions with high accuracy.Specifically, using the PART algorithm, classification rules were found where the most significant variables that distinguished between bankrupt and non-bankrupt firms were the audit opinion, the Matter sections disclosed in the audit reports, and the number of comments included in the Matter sections and qualification paragraphs.This evidence shows the positive relationship between the tendency of a firm to file for bankruptcy and the issuance of a qualified report with a high number of comments disclosed by the auditors.Interestingly, our results even corroborated the positive relationship between the bankruptcy filing and an unqualified report issued with comments disclosed in a matter paragraph.Some implications can be drawn from these results.A contribution was made to the literature of bankruptcy prediction.This is the first study that used three non-parametric methodologies with only variables extracted from the audit report (the audit opinion, type of paragraphs, and number of comments in the report) to forecast a bankruptcy situation.The only study that was closely related to the present study was that by Muñoz-Izquierdo et al. (2019).However, the objectives of both papers were different.Muñoz-Izquierdo et al. (2019) used the content of the audit report to explain the specific causes of business failure, that is, they focused on justifying the reasons why firms failed, such as internal causes like accumulated losses or high debts, or external causes such as changes in the market in which they operate or policy changes that affect them.The current paper looked for an accurate percentage of prediction, and not the particular causes of the bankruptcy.It also attempted to find the minimum number of variables from the audit report that achieved the most accurate bankruptcy predictive power possible, and these variables also needed to be the easiest for any user to obtain without any specific knowledge.Therefore, the current work is a follow-up study of the aforementioned paper, with practical implications for the users of the report.A user can strongly benefit from this study, because it found that there was no need to be an expert in the accounting and auditing domains to be able to use the audit report to detect if a firm was going bankrupt.Simply by identifying the type of opinion, whether or not a Matter section was disclosed by the auditor, and the number of comments included in the section or in a qualification paragraph, any user could quickly and easily predict the chances of bankruptcy with the same accuracy as if they had scrutinized the complete audit report.Therefore, this result may save costs in terms of time and effort to financial analysts, creditors, firms' stakeholders, and other potential users of the audit reports.
Finally, the auditing profession might also benefit from this paper because the results confirmed that, during the global financial crisis, there was an important number of bankrupt firms that issued qualified reports or, at least, a warning in a Matter section about their imminent failure, whereas nonbankrupt companies issued unqualified reports.Thus, even though the role of auditors is to ensure the reliability of the financial information provided to stakeholders, the audit report can also be a "first glance" signal to evaluate a firm's probability of bankruptcy.
The limitations of this study are mainly related to the sample.First, the non-bankrupt group was selected based on a matching process, using the variables of firm size, year, and industry, in accordance with prior literature.Nevertheless, it seems reasonable to think that other variables could have been chosen.Second, this study focused on Spanish private non-financial firms, so some results might have been driven by the specific socio-demographic characteristics of the sample.For instance, due to the Spanish context, the frequency of firms in the different industries of the sample varied significantly.For comparison purposes, future research could focus on an extension of the study to other regulatory contexts and periods of time.Although this line of research could lead to very interesting results, the preparation of the dataset might be time consuming given that every audit report must be manually codified.

Figure 2 .
Figure 2. Model 2: The audit opinion and type of paragraphs PART model.

Figure 3 .
Figure 3. Model 3: The audit opinion, type of paragraphs, and number of comments in the PART model.

Table 1 .
Literature review of bankruptcy and auditing.
US/Public/1990/100 L&H clients and 4523 non-L&H clientsThe disclosure of bankruptcy has an adverse effect on market prices and the market does not react to an auditor's replacement/Multivariate test (OLS)

Line of research: Auditor independence and audit quality
Auditors were more likely to issue GCOs in the period after December 2001, with the number increasing even more in 2002-03 due to the Sarbanes-Oxley Act (2002)/Multivariate test (logit) Audit switching is positively associated with the issuance of GCOs; moreover, the issuance of a firsttime GCO leads to a loss of clients; however, there is no evidence of the self-fulfilling prophecy/Multivariate test (logit) Australia/Public/1995-1996and 2004-2005/142with GCOs Auditors maintained the reporting accuracy of GCOs before and after corporate collapses in 2001/Multivariate test (logit) Basioudis et al. (2012) US/Public/2000-2007 10,394 firmyear observations of distressed firms High non-audit fees affect auditor independence only when audit tenure is long or when auditor quality is poor/Multivariate test (logit)

of research: Bankruptcy prediction using auditing
In Table 1, AAER: Accounting and Auditing Enforcement Releases; GCO: going concern opinion; GFC: global financial crisis; L&H: Laventhol and Horwath; MDA: multiple discriminant analysis; OLS: ordinary least squares; SAS: Statement on Auditing Standards; SEC: Securities and Exchange Commission; SMEs: small and medium-sized enterprises; UK: United Kingdom; US: United States.2.1.1.Line of Research: Effects of Auditing

Table 3
presents the summary statistics of the sample.The total sample comprised 808 firms, 404 of which had filed for bankruptcy legal proceedings.The rest, or non-bankrupt group, were manually selected to match the year, size (total assets), and industry of one of the bankrupt observations.Industries of the sample were created based on NACE (Nomenclature of Economic Activities) codes.The age of the sample was expressed in years and the size in thousands of euros.WCTA stands for "Working capital divided by total assets", EBITTA for "Earnings before interest and taxes divided by total assets", and BVETL for "Book value of equity divided by total liabilities".Data used to calculate the financial ratios was winsorized at 1% and 99%.Finally, # of obs. is the number of observations.