An AI-Based Automated Continuous Compliance Awareness Framework (CoCAF) for Procurement Auditing

Abstract

Compliance management for procurement internal auditing has been a major challenge for public sectors due to its tedious period of manual audit history and large-scale paper-based repositories. Many practical issues and potential risks arise during the manual audit process, including a low level of efficiency, accuracy, accountability, high expense and its laborious and time consuming nature. To alleviate these problems, this paper proposes a continuous compliance awareness framework (CoCAF). It is defined as an AI-based automated approach to conduct procurement compliance auditing. CoCAF is used to automatically and timely audit an organisation’s purchases by intelligently understanding compliance policies and extracting the required information from purchasing evidence using text extraction technologies, automatic processing methods and a report rating system. Based on the auditing results, the CoCAF can provide a continuously updated report demonstrating the compliance level of the procurement with statistics and diagrams. The CoCAF is evaluated on a real-life procurement data set, and results show that it can process 500 purchasing pieces of evidence within five minutes and provide 95.6% auditing accuracy, demonstrating its high efficiency, quality and assurance level in procurement internal audit.

1. Introduction

The way business is currently conducted and how financial information is managed has significantly altered by the acceleration of information flows and advancement of emerging technologies under digital economics. Meanwhile, a rapidly growing number of organisations are conducting accounting information recording and financial reports generated in real-time using powerful online enterprise resource planning systems. Whether in the public or private sector, the traditional audit paradigm where the auditors provide ex-post financial opinions seems to remain in the pre-digital age [1]. Research has shown that in the audit profession, severe lags exist compared to the development and utilisation of technologies [2,3,4]. There is also evidence showing that auditors are relatively slow in taking up new technologies. A study conducted by KPMG found that 80 percent of respondents recognized that bigger samples and more advanced technologies should be used to gather and analyse data [5]. Audits need to adapt to the digital age.

In general, traditional methods of conducting audit activities are outdated in the current economic environment in the aspects of quality, efficiency and assurance level. This status quo applies to both internal and external auditing services. Specifically, in the case of this study, the research focus is compliance audit of internal auditing services in public sectors, where internal auditors will view compliance procedures related to procurement activities over the whole course of the compliance audit. As part of an internal audit, compliance will be discussed and examined under the category of internal auditing in this study. Previous research found that public sectors have a stronger emphasis and demand for improving audit quality, efficiency and assurance level over that of private sectors [6]. However, the manual nature of current internal audit procedures in the public sector has constrained audits by being a labour and time intensive business activity [7]. In addition to the low efficiency of manual auditing, traditional audit approaches also reveal a high frequency of fraudulent activities [8]. The 2018 report from the Association of Certified Fraud Examiners (ACFE) advised that a mere 15 percent of occupational frauds were initially uncovered by internal auditors, and 16 percent of government organisations are victimised by occupational fraud [9].

Among the business activities conducted in public sectors, procurement is always a big expense. Published figures suggest the Australian government spends over $110 billion each year on the purchase of goods, services and work [10]. It is indicated that high expenditure tends to attract criminals and fraudulent behaviour [11]. Therefore, procurement remains a high financial risk area of public sector management. It can also lead to legal and loss of reputation risks due to failing compliance rules. There are numerous examples of government procurement failure. Public sector internal auditors play an important role in identifying and mitigating procurement related risks while conducting procurement compliance auditing. Unfortunately, they still suffer from the deficiencies of manual checking process including high consumption of time and labour, inaccuracies of auditing results and delays in interpreting and representing auditing reports [12].

Therefore, combining the existing deficiencies of internal audit mentioned above and the current development of technologies (i.e., big data, blockchain, artificial intelligence), auditors can leverage new technologies to collect a wider range of real-time and relevant data through an automatic process to alleviate the repetitive workload [4].

Based on that, we proposed the continuous compliance awareness framework (CoCAF). In detail, the main methodologies used in our framework consist of text extraction from different forms of original purchasing evidence, automatic compliance check between system records and data extraction and a final compliance report rating system to facilitate an easy understanding of the compliance conditions.

This study is meaningful as the results provide a clearer and more scientific view for the procurement auditing in terms of compliance conditions, which can facilitate the provision of recommendations and suggestions to the procurement managers of the organisation in compliance and risk assurance management. The proposed CoCAF will significantly improve the quality, efficiency and assurance level of public sector internal audits and help to reach and retain a proper risk level of the procurement processes. We also evaluate the CoCAF on a real-life procurement data set. Results show that it can process 500 pieces of purchasing evidence within 5 min and provide 95.6% auditing accuracy, demonstrating its effectiveness and efficiency in procurement internal audits.

2. Related Work

As defined by the International Organization of Supreme Audit Institutions (INTOSAI) on internal public audit, the three areas to which internal auditing is targeted are economy, efficiency and effectiveness. Economy refers to keeping the cost of audit inputs low (auditors, materials, equipment, etc.), without compromising audit quality. Efficiency refers to the relationship between audit outputs (audit services and reports) and audit inputs. It is concerned with getting the most from available resources. Effectiveness refers to audit quality in terms of achieving audit objectives.

In our study, the literature review in this section will combine both economy and efficiency and discuss the two aspects under the term ‘audit efficiency’ as both are concerned with audit inputs. Internal audit effectiveness will be reviewed under ‘audit quality’. We also take audit evidence accumulation procedures into consideration. In current practice, auditing services are conducted in a backward-looking manner. The evidence verification process is provided through the sampling approach by analysing merely a snapshot of all the entity’s transaction data set, thus, the scope and frequency of samples are not able to guarantee a high-level audit assurance. This part will be discussed as ‘audit assurance’. Therefore, the literature review in this study will be developed in three areas: audit quality, audit efficiency and audit assurance.

As with general audit, the main goal of an internal public procurement audit is to assess how efficient and effective the procurement process is in terms of compliance with laws, established ethical standards and detecting possible fraud and misuse of funds based on sufficient and proper evidence. It is undoubtedly vital in providing reliable assurance of useful financial procurement information and can have a significant effect on detecting misuse and fraudulent procurement behaviours. However, studies have shown that the current internal auditing practices are not able to provide a high level of assurance for corporation business behaviours efficiently and effectively. A previous study has indicated that there are seven types of audit waste in the traditional audit profession: over auditing, waiting time, time delays, the audit procedure, work-in-process and review process, as well as errors and mistakes [13], which are also applied to public sector procurement activities. These audit wastes can be summarised into two categories: limited audit quality and low audit efficiency. The following section of the literature review will be conducted regarding the quality, efficiency and assurance levels of the current procurement audit process. In addition, technologies that can enhance current auditing services will also be reviewed. The first part will provide a clear view of current procurement audit deficiencies, which will thus benefit the establishment of our continuous auditing method. In the second part, a review of previous technology enhanced continuous auditing methods and practices will be conducted to provide a better understanding of the advantages of our CoCAF methodology.

Specific to this study, audit quality includes works related to whether current audit approaches are operating effectively in the aspects of detecting corporate fraud to maintain a proper risk level and controlling unintentional audit errors. An audit efficiency study involves audit resource occupation analysis in terms of time, money and labour. The audit assurance study contains the aspects of audit scope and reporting frequency. The overview of previous studies is shown in Figure 1.

Figure 1. Literature review framework of procurement internal audit.

2.1. Studies of Audit Quality

Internal fraud and corruption risks often do not draw enough attention from an organisation’s management, which may lead to severe compliance, financial and reputational consequences. Thus, an internal audit is expected to assess compliance risks and determine whether further action is needed or not.

Maintaining a high audit quality matters as it ensures the production of auditing reports that can be trusted in decision making processes. The scandals like Enron, WorldCom, Parmalat and Xerox exposed disadvantages in the traditional external financial reporting and audit system and illustrated the importance of audit quality [1]. The audit quality inspection report by the Financial Reporting Council indicated 25.5% of audits conducted needed improvement and 14.4% of the audits had to be stopped to make improvements. Problems related to audit quality also plague internal audit services. For example, Toshiba was caught up in Japan’s biggest accounting scandal since 2011 because of its internal audit failure [14]. Unfortunately, current audit approaches and sampling methods are not able to be fully relied upon to uncover most transaction errors or occupational fraud [13].

The Association of Certified Fraud Examiners (ACFE) found that only 15 percent of occupational frauds were uncovered by internal auditors, while 40 percent were detected through anonymous tips [9]. The reason is that financial report auditing is normally only conducted on a yearly basis and most audit procedures are based on small transaction sample data sets over fixed financial periods. This method is out-dated due to the significantly increased number of transactions happening in the current economy. Thus, the sampling method is not sufficient for evaluating internal control risks or for detecting and eliminating fraud. In the meantime, high levels of assurance will not be provided if auditors still rely on costly substantive testing and neglect analysis of transactions or utilisation of computer technologies [8].

In addition to the low detection rate of human fraud, current audit procedures do not perform effectively in uncovering unintentional financial errors caused by information overflow in the current business environment. It has been proposed that the quality of an auditor’s judgement will initially increase followed by a decline when the number of relevant pieces of available information increases [15]. This conclusion has also been proved by later researchers using an empirical study model [16]. In the context of big data, auditors under current audit practices face a similar situation of information overload and can produce cognitive errors, which will eventually result in the selection of irrelevant or insufficient accounting information to support a high quality audit opinion [17].

2.2. Studies of Audit Efficiency

The current audit approach that reflects the twentieth-century methodology would be described as a time-consuming process. This process normally begins with audit planning between auditors and customer companies after the establishment of an audit contract. During the audit planning stage, a company risk assessment and the objectives and scope of the audit activities will be settled. After this, auditors will be assigned to collect data, analyse evidence and provide professional opinions about the authenticity of a company’s financial statements and provide further management advice regarding internal controls and assurance levels. At the final stage, auditors will present a formal audit report expressing their opinions [18].

The traditional audit practice is costly and labour-intensive regarding the resource occupation level during the procedure. Traditional auditing is viewed as inefficient in the whole audit process, from the audit planning stage through to when the final audit report is provided. Redundant steps, inexperienced staff and inadequate data cause some of the wastages. There are also stops during the process that adds cost without value [19]. Some researchers believe that the manual testing and auditing of internal controls in current audit processes demands a large amount of time and labour and advice that the automation of controls should be considered, which can provide a more accurate and cost-efficient way of achieving complete, accuracy and integrity in financial reporting [20].

2.3. Studies of Audit Assurance

The definition of ‘assurance’ in the auditing profession varies to that of ‘audit assurance’ in this study. Here, audit assurance refers to providing more reliable audit reports and a reduction in audit risks by continuously reviewing the entire population of transaction data.

The current audit approaches are making client companies suffer from high costs and significant time delays in the aspects of information collection, processing and reporting. It has been argued that only timely information can be relied on to make effective business decisions [19]. In the current economic era, traditional audit methods are not able to generate real-time information in higher frequency because of the nature of its time lag during the process. The real-time transactions happening every day are providing real-time and continuous feedback to relevant stockholders. This trend has been recognised by auditing academics and professionals who are trying to provide more appropriate solutions to help auditing adapt to this speeding business environment [21]. Currently, financial information is normally reviewed on an annual or half-yearly basis in the traditional audit practice. As a result, material errors, omissions or frauds can easily be covered up for a relatively long time before detected by auditors [7]. This is how the audit lag happens. Thus, remaining a high audit assurance level in this information over-flow environment becomes difficult.

Heavily relying on the current audit mode, small transaction sampling on a limited period and data sets is not enough for evaluating internal controls or for detecting and eliminating fraudulent activities [8]. In addition, by relying primarily on substantive testing and ignoring the analysis of transaction details or just doing computer audits mechanically, financial audits are not only unable to provide high levels of assurance, but, also, maintain current auditing costs at a high level due to the complicated substantive testing process.

2.4. Studies of Technology Enhanced Continuous Auditing

By leveraging the benefit of emerging technologies, such as satellite imaging, data analytics, blockchain, artificial intelligence and the like, internal auditors can address current and newly occurring risks promptly and provide more trustworthy auditing reports that will assist company decision-makers in risk management, governance and internal control processes in a continuous manner, thereby delivering enhanced auditing value [4]. Audit researchers and practitioners have widely recognised that the next stage in audit development is the use of continuous auditing utilising the technologies of computer science. So far, researchers and practitioners have provided great solutions to the development of continuous auditing in the aspects of theory and applications. A large amount of contributions has been published in several top academic publications of multiple disciplines, including accounting, accounting information systems, management information systems and computer science [22]. Academic researchers have made great contributions to the development of continuous auditing theories and applications. Several enabling technologies have been identified for continuous auditing, which includes belief functions, databases, expert systems, information retrieval [23,24], neural networks and real-time accounting.

In general, the utilisation of technologies in the audit profession is discovered in both financial and non-financial fields based on the data types [25]. From the perspective of financial auditing, studies on the employment of technologies like big data in audit analytics [25,26,27], blockchain [4,28,29,30] and artificial intelligence [31,32] can be found over the last decade. In the meanwhile, publications on how technologies could benefit from non-financial data auditing also provide valuable insights into this study. For example, in environment and social compliance auditing, technologies like big data analytics, blockchain, sensors, satellite imaging and social media, all provide possibilities to improve the veracity of compliance auditing [12,33,34].

Despite the fruitful research output in the technology enhanced auditing area, very limited practical application case studies have been conducted. Thus, we will apply and evaluate our proposed framework on a real-life procurement data set. The focus of our research is financial procurement compliance auditing. We employ artificial intelligence in the auditing process to automate current procedures, enlarge evidence scope, reduce audit inputs, provide continuous reporting and eventually improve overall audit effectiveness and efficiency.

3. Public Sector Procurement Audit Case Background and Current Auditing Practices

This study focuses on a large-scale Australian government organisation and has meaningful practical application value. Generally, public sectors spend a large amount of money every year on procurement activities which normally happens daily as new resources are continuously demanded by all the departments for efficient operation [10].

In the specific public sector procurement audit case we are studying, when the purchased items are not in use, they are usually stored in multiple warehouses in different locations, which not only makes it difficult to keep track of where all stocks is but also difficulty in being able to confirm if the delivered items are received from the correct suppliers. To verify the authenticity and accuracy of the purchased activities and stock-in procedure, purchasing evidence auditing needs to be conducted quarterly as required by related regulatory authorities.

Manual auditing has been utilised for years in this organisation to check whether the purchases have occurred correctly without any wastage of funding. Normally, multiple operational staff are involved in auditing one purchase, being the auditing officer and the purchasing officer. There are also managers involved in the middle of the process, who are not discussed in this study, as their roles are not related to the investigation. When a quarterly audit is initiated, purchase records that occurred in that period are sampled from the procurement system database to be audited. The related supporting evidence could include purchase orders, tax invoices, supplier quotes and supply contracts. Upon receiving these documents, two major objectives of the auditing procedure are checked: (1) whether the purchase evidence requested by the procurement policy are sufficiently provided by the purchasing officers and (2) whether the unit price, quantity, total price, purchase date and foreign currency amount (if applicable) described in the evidence are matched to purchase records in the system database. According to the procurement auditing policy, one purchase is considered to be compliant only when both of the above conditions are satisfied, otherwise it is non-compliant.

This manual auditing process generally takes a long time every quarter to finish due to many practical problems. For instance, the auditing process needs to consider what currency was used when the item was purchased, meaning the exchange rate might also need to be considered. Along with the price being in foreign currency, the price might also be presented as the price per unit bought or the total combined price of every unit. The correct purchase date is sometimes hard to identify as there could be several dates on one tax invoice and the auditing officer needs to locate the corresponding one in the database, which also costs time. Many purchases could be contained in one tax invoice, which also makes it difficult to find the correct invoice. Therefore, the weaknesses of existing manual-based auditing are summarised below:

• Traditional audits are time lagging and cannot provide real-time information useful to management and stakeholder decisions.
• A traditional manual audit is labour and cost intensive.
• Compared to computers, there is a higher rate of human error during the auditing process that affects the accuracy level of audit work on enterprise risk assessment.
• The audit sampling method has limits and a full population audit cannot be achieved, which affects the quality of the audit report to a certain extent.

4. Framework of an Automated Continuous Compliance Awareness Powered by Text Mining

As discussed in the last section, auditors are currently provided with a certain number of evidence folders corresponding to the purchase records in the organisation’s system based on the sampling method. Manual checks are conducted and go through every file provided in the evidence folders of each purchase order. Auditors should look for evidence to verify the entire procurement process. Common purchasing evidence includes purchase contracts, purchase orders, invoices, received documents, inspection documents and supplier statements. This rather low-efficiency process may lead to increasing errors and mistakes of manual-based auditing operation such as missing data. Sample selection is not representative of the total case.

In this study, we propose a new compliance checking and rating framework, coined CoCAF. CoCAF is defined as an automated method to compare purchasing records compliance auditing against procurement policies and requirements. It aims to largely reduce the tedious workload, improve the compliance rating procedure and, hopefully, forward the development of continuous auditing in large-scale organisations. The framework is mainly composed of three stages, including text extraction, compliance checking and rating report, as shown in Figure 2.

Figure 2. The continuous compliance awareness framework (CoCAF).

The three stages are intuitively illustrated in a workflow as shown in Figure 3. On the enterprise level, an organisation provides policies regulating purchasing procedures and the required evidence to be audited based on which purchasing staff uploads the evidence after each purchase occurs. On the processing level, the text extraction stage will record the semantic auditing policies received from the enterprise database and then apply artificial intelligence technologies to automatically extract the required data from the uploaded evidence according to the specific requirements of the policies. Afterwards, the compliance checking stage takes the data extracted in stage one, and then checks lists generated by the enterprise database for a detailed cross-field matching and compliance rating. Finally, the matching and rating results are fed to the rating report stage, whereby the perceived information will be elaborately reorganised and visualised to produce a continuously updating report demonstrating the compliance performance of the ongoing purchasing activities. The details of the three stages are respectively described in the following chapter.

Figure 3. The workflow of CoCAF.

5. CoCAF Techniques

This section explains the main techniques used for each stage of CoCAF. To achieve the highest accuracy and performance, we utilized state-of-the-art technologies to implement those techniques.

5.1. Text Extraction

To generate up-to-date compliance reports continuously and automatically, the semantic information of the procurement evidence documents is required. That information includes the evidence type, date of purchase, quantity, description, unit price, total price and exchange rate if the purchase was transacted in a foreign currency. Therefore, this stage consists of three main tasks. First, detecting new evidence when it is uploaded by a purchasing officer. Second, extracting the text of the heterogeneous unstructured documents and storing them in a document-based database in a structured manner by applying state-of-the-art text extraction methods based on the file type of the evidence. Depending on the file type, this task can be fair or slightly more expensive. For example, if the file type is a digital pdf, the text can be easily transformed by parsing the pdf directly into raw text. However, if the file type is a scanned pdf, the pdf must be converted to an image and then Optical Character Recognition (OCR) can be applied to extract the text of the evidence. However, on average the text of one evidence can be extracted within 0.5 s. The third task is the extraction and storage of the semantic information from the text by using regular expressions, that information will be compared with the values from the check list, which will be explained in the next stage of the framework.

In the following, those three tasks are demonstrated through an example. A purchasing officer has transacted a new external purchase in a foreign currency. He then uploads the required purchasing evidence, which includes an invoice for the purchase and the currency conversion table of the date of purchase. Our framework detects the uploaded evidence automatically and starts to extract and store the text. Afterwards, it extracts the semantic information from the extracted text. In the case of the invoice, is the extracted text includes the date of purchase, quantity, description, unit price and the total price. In the case of the currency conversion table, this is the date of the conversation table and the exchange rate. This information will then be stored in a database and is ready to be used for the compliance check.

5.2. Compliance Check

After extracting the purchasing information provided in the evidence folders, compliance checking will then automatically be conducted with every purchase order. We assigned different scores to different compliance conditions. Specifically, the detailed algorithm for our compliance check process can be found in Algorithm 1. The evidence folders are audited individually. Firstly, we filter out all the empty folders by considering them as totally non-compliant. Secondly, if the folder is found containing valid evidence, we look through the provided evidence to find the price that was paid for the items. This will then be checked against a central database that contains the ground truth for the test evidence. If the price matches, the compliance level will be raised to one. If this does not match, the evidence will be classified as totally non-compliant. Next, if the correct price is found, the evidence will be searched to locate the number of items that are purchased and compare that with the quantities that are received. If this matches, the compliance level will be raised to two. In contrast, if this does not match, the compliance level will be left at one and the algorithm will be stopped. If a correct quantity is found, the evidence will be scanned for the item’s description. If the description matches the description provided in the ground truth, the compliance level will be raised to three. In contrast, if it does not match, it will be left at two. After that, the algorithm searches for the date when the purchase occurred. If this does not match, the compliance level will be raised to four, while if it does match, then it will be raised to five.

Algorithm 1: Investigation approach

CoCAF is designed to provide an effective and efficient method to conduct continuous auditing when combined with text mining and machine learning techniques. Without human intervention, CoCAF is able to continuously detect newly uploaded evidence, simultaneously extract the necessary data, automatically match it with the entries in all purchasing transactions rather than a sample list and, finally, provide a consistently updated compliance report.

Regarding the quality of the audit approach, the current sampling method can only cover a relatively small portion of the large volume of purchase transactions occurring each quarter. In our recent case study, only 500 samples were being checked compared to triple that amount of total purchase orders. It is noted that there are material transactions that are not included in the samples but are significant to the procurement activities’ risk assessment of the organisation. In addition, the auditing happens every quarter in some public sectors which is more frequent than the annual auditing in public companies, but, considering the need for monitoring public funding expenditure, even with all the efforts to conduct a procurement audit every quarter, such ex-post monitoring activities have not been effective in controlling risks and reducing the non-compliance rate. Public sectors are eager to receive continuous feedback to ensure a high compliance rate of purchasing activities. In addition to the problems we detected above, the investigation was also faced with problems like lack of consistent lodging guidelines, insufficient supporting documentation and inappropriate internal control systems. In contrast, by adopting the CoCAF, we can achieve a full transaction of continuous auditing by automatically reviewing every purchasing order to occur in real-time and a compliance report will be available in the meantime.

5.3. Rating Report

To be more specific with the compliance report in the last stage of the CoCAF workflow, we proposed a five-level procurement rating report system, which includes totally non-compliant, non-compliant, poorly compliant, partially compliant and totally compliant, as shown in Table 1. By rating the final reports, we can discover the specific issues that lead to different results and fully understand the issues and risks that may occur during the manual auditing procedure. In addition, the interest parties can clearly understand the compliance level of the purchase order and make more relevant management decisions.

Table 1. The semantics of compliance levels in CoCAF.

Semantic Types	Criteria
Totally non-compliant	Empty/no evidence
	Irrelevant evidence
Non-compliant	Invalid evidence
	100% mismatched information
Poor-compliant	Quantity mismatch, unit price match
	Unit price mismatch, quantity match
	Quantity and unit price match, description partially match
Partially compliant	Quantity, unit price and description match, exchange rates mismatch
	Quantity, unit price, description and exchange rates match, date mismatch
Totally compliant	All information matches

The criterion for each level is defined accordingly in the right-side column of the table. Using this table, different compliance levels are assigned to every purchase order. According to our recent investigation of the public sector procurement cases, we also discovered that the manual auditing progress is in of low efficiency in terms of both labour and time occupation.

Figure 4 gives an example of how the rating report is generated based on the compliance levels using CoCAF for purchasing record auditing. The report contains a pie chart and a spreadsheet. The pie chart provides overall information about the compliance conditions of the purchasing orders. In this specific example, it is obvious that the overall compliance rate is relatively low, which indicates further investigation by auditing officers or management authorities may be needed. In addition, to help with the investigation of non-compliant reasons, we provided a spreadsheet listing with all useful purchasing information.

Figure 4. An example of purchasing audit report.

6. Evaluation Results

In this section, we will evaluate CoCAF under a real-world data set provided by an Australian large-scale public sector organisation. This sector conducts procurement activities daily and currently suffers from difficulties caused by traditional auditing methods. Normally, the auditing procedure includes requesting procurement evidence from purchasing officers, receiving purchasing sample lists, comparing the data (date, unit price, quantity, currency, total amount, etc.) in both pieces of evidence and system records and compliance report generating.

In this case, the public sector provided 500 purchasing records and the corresponding evidence folders for evaluation. The records (checking list) are listed in an Excel form with basic items like date, unit price, quantity, total amount, exchange rate and so forth. While the evidence folders contain different evidence forms such as invoices, receipts, purchase orders, price lists and the like. There were a total of 1120 files in the evidence folders.

To evaluate the efficiency and effectiveness of CoCAF, a classification approach will be employed to compare the compliance results provided by the proposed CoCAF against the baseline. We will use a confusion matrix to define the classes of four situations being: True positive (TP), false negative (FN), true negative (TN) and false positive (FP). Accordingly, we measure three classification rates including Effectiveness, false positive rate (FPR) and false negative rate (FNR).

$$Effectiveness=\frac{TP+TN}{TP+FN+TN+FP}$$

(1)

$$FPR=\frac{FP}{FP+TN}$$

(2)

$$FNR=\frac{FN}{FN+TP}$$

(3)

Based on the results of this evaluation procedure, the effectiveness is shown in Figure 5. According to the figure shown in the matrix (refer to Table 2), CoCAF achieved an effectiveness rate of 95.6% by auditing the sample of 500 purchasing records. This represents that CoCAF has a chance of 95.6% compared with the true situation. At the same time, the FPR is 0.9%, which means the chance that CoCAF missing a non-compliant record is less than 1%.

Figure 5. Effectiveness confusion matrix.

Table 2. Effectiveness confusion matrix for CoCAF.

	Processing Time	Labour Consumption	Accuracy	Type I and II Error Rate
CoCAF	5 min	1 operator and 1 auditing officer	95.6%	0.007% and 7.3%
Manual audit	120 h	8 auditing officers	≈90%	≈5% and 5%

In addition to the effectiveness rate of the CoCAF, we also consider time and labour consumption factors. Regarding the time and labour spent conducting the compliance auditing, CoCAF only took a couple of minutes by one operator and one auditing officer to achieve 95.6% accuracy, while the corresponding manual check took two people more than two weeks to complete. Therefore, we can conclude that the CoCAF can be very reliable when undertaking compliance checking and also saves considerable time and labour.

Based on the evaluation results, it is clear that CoCAF significantly reduced the time and labour cost associated with compliance auditing while maintaining a higher accuracy rate compared with manual auditing. By conducting an audit on every purchase record instead of using the sampling approach, CoCAF has further reduced procurement related audit risks. Therefore, CoCAF demonstrates its ability in achieving high efficiency, quality and assurance level in procurement internal audit activities.

7. Conclusions

Auditing makes immense contributions to maintaining an organisation’s reporting risk levels and providing proper assurance to all interested parties but the traditional auditing approach has not kept pace with the real-time economy in this information rich era. The current audit approaches and techniques that were valuable in the past are now becoming increasingly out-dated.

To address this problem, we proposed the continuous compliance awareness framework (CoCAF), which can automatically and timely audit purchasing activities by intelligently understanding compliance policies and extracting the required information from purchasing evidence. Professionals and academics should continuously develop more efficient methods of conducting auditing activities that will help to allow their valuable resources to be utilised in the most cost-effective way.

Enhanced by artificial intelligence, CoCAF will benefit auditing services in terms of reducing errors, fraud and costs without compromising audit quality. It may also cause auditor redundancy to a certain extent. Therefore, auditors are required to adapt to the change and to assess the possible new risks in procurement activities or other management information used for decision making purposes occurred while using CoCAF.

As traditional auditing services, especially on-site visits, were significantly interrupted during the COVID-19 pandemic, the potential for remote auditing by adoption of advanced technologies needs to be considered by the auditing profession [35]. By employing artificial intelligence and conducting compliance auditing automatically, CoCAF provides the possibilities to realise remote auditing and provide continuous auditing services. At the same time, the study of procurement compliance in this article is limited to the authenticity of procurement data and the compliance situation with procurement policies without involving the interpretation of relevant laws and regulations. Thus, CoCAF does not address the complexities relating to legal semantics. These areas will be further explored and discussed in our future work.