Error Classification and Static Detection Methods in Tri-Programming Models: MPI, OpenMP, and CUDA

Abstract

The growing adoption of supercomputers across various scientific disciplines, particularly by researchers without a background in computer science, has intensified the demand for parallel applications. These applications are typically developed using a combination of programming models within languages such as C, C++, and Fortran. However, modern multi-core processors and accelerators necessitate fine-grained control to achieve effective parallelism, complicating the development process. To address this, developers commonly utilize high-level programming models such as Open Multi-Processing (OpenMP), Open Accelerators (OpenACCs), Message Passing Interface (MPI), and Compute Unified Device Architecture (CUDA). These models may be used independently or combined into dual- or tri-model applications to leverage their complementary strengths. However, integrating multiple models introduces subtle and difficult-to-detect runtime errors such as data races, deadlocks, and livelocks that often elude conventional compilers. This complexity is exacerbated in applications that simultaneously incorporate MPI, OpenMP, and CUDA, where the origin of runtime errors, whether from individual models, user logic, or their interactions, becomes ambiguous. Moreover, existing tools are inadequate for detecting such errors in tri-model applications, leaving a critical gap in development support. To address this gap, the present study introduces a static analysis tool designed specifically for tri-model applications combining MPI, OpenMP, and CUDA in C++-based environments. The tool analyzes source code to identify both actual and potential runtime errors prior to execution. Central to this approach is the introduction of error dependency graphs, a novel mechanism for systematically representing and analyzing error correlations in hybrid applications. By offering both error classification and comprehensive static detection, the proposed tool enhances error visibility and reduces manual testing effort. This contributes significantly to the development of more robust parallel applications for high-performance computing (HPC) and future exascale systems.

1. Introduction

Exascale computing relies on advanced hardware and accessible resources to dramatically improve computational performance. Although programming languages such as C, C++, and Fortran have evolved to support memory-level parallelism, they still exhibit limitations, particularly in efficiently distributing tasks across accelerators and multi-core processors. To overcome these limitations, developers adopt additional programming paradigms that support greater concurrency, optimize resource utilization, and accelerate execution. High-level programming models including the Message Passing Interface (MPI) [1,2], Open Accelerator (OpenACC) [3,4,5], Compute Unified Device Architecture (CUDA) [5,6,7,8], Open Multi-Processing (OpenMP) application program interface (API) [1,9] and Open Computing Language (OpenCL) [10] extend traditional languages by enabling the more efficient use of hardware resources such as shared memory, GPUs, and multi-core CPUs. Each model introduces distinct parallelization strategies and implementation practices. As a result, combining multiple models can significantly improve performance and portability. For example, hybrid applications that integrate OpenMP with OpenACC or MPI with CUDA can distribute workloads across heterogeneous systems, achieving superior throughput. However, integrating these models introduces substantial complexity. While some emphasize GPU utilization, others prioritize memory locality or thread-level parallelism. Consequently, developers must carefully manage these interactions to prevent subtle runtime issues, including race conditions, deadlocks, livelocks, and other nondeterministic behaviors that often elude compiler detection. Moreover, even when integration is syntactically correct, extensive testing is required to ensure functional correctness and runtime stability. Multiple scientific and engineering domains have demonstrated the potential of hybrid programming models, employing combinations of MPI, OpenMP, CUDA, and others to accelerate workloads in fields such as molecular dynamics, embedded computing, and machine learning. However, the concurrent execution of numerous threads and diverse memory hierarchies in such systems increases the likelihood of hard-to-detect runtime errors.

Several studies have demonstrated that the characteristics of different programming models can be integrated to generate hybrid models that are scaled between processing and connectivity depending on memory efficiency. These models are further classified as follows:

• Single-level, including independent models such as CUDA, MPI, or OpenMP.
• Two programming models, such as MPI + OpenMP or OpenMP + CUDA, are combined to generate dual-level (X + Y) systems that increase parallelism.
• Tri-level (MPI + X + Y) systems consist of three different programming models to improve parallelism.

As a result, parallel models generate more parallel errors and require a lot of effort for their testing. Furthermore, as parallel models are merged into a single application, the reasons for these errors fluctuate throughout their runtime with the integration of the models. To date, there are no compilers that can detect these forms of runtime errors.

To address this challenge, the present study proposes a static analysis tool specifically designed for C++-based applications that integrate MPI, OpenMP, and CUDA. A key contribution of this work is the development of a novel error dependency graph which systematically models the relationships between potential runtime errors in hybrid applications. The proposed tool also introduces a comprehensive classification scheme for these errors, supplemented by illustrative examples that demonstrate their causes and implications.

With the increasing significance of multi-GPU computing in high-performance computing (HPC), particularly in fields such as deep learning, climate modeling, and molecular dynamics, the combination of MPI, OpenMP, and CUDA is viewed as important for achieving performance and scalability. However, the complexity of these diverse systems increases the possibility of mild, hard-to-detect runtime errors. Scientific workloads heavily rely on hybrid multi-GPU environments, as demonstrated by real-world applications such as MDScale [11] and GPU-accelerated molecular dynamics, which employ state-of-the-art software performance and porting from NVIDIA CUDA to AMD HIP [12]. The above instances highlight the need to develop robust debugging and static analysis tools that can handle the complexities of tri-model programming. The suggested tool attempts to assist developers in improving the accuracy, portability, and general performance of HPC applications running at scale by addressing these issues.

Because the current compilers and existing tools are unable to detect these errors in tri-model applications, the proposed solution was rigorously evaluated using a suite of benchmarks. These tests reflect the latest MPI, OpenMP, and CUDA specifications, ensuring the relevance and applicability of the results. Accuracy and performance were compared against state-of-the-art static analysis tools.

To the best of our knowledge, this study represents the first comprehensive effort to detect and classify runtime errors arising from the integration of MPI, OpenMP, and CUDA in C++-based environments. It also introduces the first static analysis tool tailored specifically for this purpose. At present, no existing frameworks offer such capabilities.

The remainder of this paper is structured as follows: Section 2 outlines the programming models utilized. Section 3 introduces the challenges and testing techniques. Section 4 reviews related work identifies integration-specific runtime errors and presents a classification and taxonomy of these errors. Section 5 describes the implementation of the proposed tool, detailing the analysis and detection phases, including the algorithms employed. Section 6 presents the testing and evaluation, reporting the results and discussing the experimental findings. Finally, Section 7 concludes the study and outlines future research directions, summarizing the main contributions of this work.

2. Background

This section provides a brief overview of the MPI, OpenMP, and CUDA programming models, along with the presently available testing techniques.

The MPI was first introduced in 1994 (version 1.0) to support message passing paradigms across large-scale parallel systems. Its capabilities were expanded in its sub-sequent versions. For instance, parallel I/O and process management were introduced in MPI 2.0, while the number of procedures increased to approximately 430 in MPI 3.1. Meanwhile, in the latest release, MPI 4.0, support for hybrid programming models (MPI + X) was improved and fault tolerance mechanisms with which to enhance the robustness of large-scale applications were included [2].

The widely adopted OpenMP API, which was first introduced in 1997, facilitates parallelisms in shared-memory systems by distributing tasks across multiple threads, thereby allowing concurrent executions in a node. OpenMP 4.0, which was released in 2013, came with support for accelerator offloading, which enabled hybrid CPU-GPU parallelism. Meanwhile, in the latest version, OpenMP 5.2, which was released in 2021, compatibility across C, C++, and Fortran was enhanced and supported using multiple implementations [13,14,15,16,17].

The CUDA, which was developed by NVIDIA^® and released in 2006, is a specialized programming model that was designed to fully exploit GPU acceleration. It supports C, C++, and Fortran, offers direct access to GPU memory, and enables low-level optimizations that maximize parallel performance. CUDA 12.0, which was released in December of 2022, further optimized GPU resource utilization for parallel tasks [18].

Developing parallel applications for heterogeneous systems, particularly those that combine multiple programming models, leaves room for considerable complexities to arise. For instance, runtime errors may arise at the interfaces of the programming models, leading to race conditions, deadlocks, and data races, to name a few. As these errors often arise due to a combination of factors, such as input data and execution order, they are difficult to predict and even harder to diagnose once an application is running.

3. Challenges and Testing Techniques

Testing parallel applications is particularly challenging when the code is poorly structured or ambiguously written, as it becomes difficult to determine whether errors originate from a specific programming model, the user’s implementation logic, or the interactions between multiple models. Existing static testing tools analyze source code to detect potential runtime errors prior to execution. Compared with dynamic testing, which evaluates program behavior during execution, static testing is significantly faster and capable of identifying a wider range of issues early in the development cycle. Dynamic testing, while useful, is often impractical for large-scale parallel applications due to its computational overhead. Although comprehensive, model-checking techniques require exhaustive test case generation, rendering them unsuitable for large, complex systems.

To address these challenges, this study introduces a testing tool that integrates static analysis with error dependency graphs to detect runtime errors arising from the combined use of MPI, OpenMP, and CUDA in C++-based applications. In contrast to existing tools, the proposed approach enhances detection coverage and testing efficiency. Within the context of high-performance and exascale computing, this reduces both the testing burden and runtime overhead.

4. Literature Review

The use of parallel applications and HPC has been extensively studied for a range of objectives. Error detection techniques include static analysis, dynamic analysis, or a combination of both in hybrid approaches. Static testing evaluates the application codebase prior to execution [19,20]. On the other hand, dynamic testing evaluates the codebase of an application while it is running [21]. Although there are numerous advantages to dynamic testing, it requires several test cases and may fail to detect some errors during start-up. On the other hand, a hybrid testing tool integrates the best features of both static and dynamic testing [19,22].

Applications such as the Intel^® Trace Analyzer and Collector [23] and MPI-Checker [24,25] use relatively mild static methods to detect runtime issues in MPI. On the other hand, tools such as MEMCHECKER [26], Umpire [26,27], the Marmot umpire scalable tool (MUST) [28,29], MAD [30], MOPPER [31], and runtime error detection (RTED) [32,33] use dynamic methods to identify runtime errors in MPI. Apart from that, hybrid methods, which involve using different tools to evaluate different runtime issues, are used [33] to identify deadlocks in MPI. The ACC_TEST [34] is one such tool that was specifically developed to quickly and accurately pinpoint runtime issues such as data races, deadlocks, and mismatches, to name a few.

Applications such as PolyOMP [4], OmpVerify [35], and DRACO [36] are static methods of identifying runtime errors in the OpenMP API, whereas Helgrind [36,37], Valgrind [38,39], Intel^® Thread Checker [40,41], Sun Studio^® thread analyzer [40,42], ROMP [43], and RTED [32] are dynamic methods. On the other hand, AddressSanitizer (ASan) and ThreadSanitizer (TSan) [44,45] use OpenMP’s directives to identify runtime issues related to architecture. Meanwhile, ARCHER [46] uses a hybrid method to identify data races in extensive OpenMP applications. It is also used to validate dual-architecture-based systems such as Marmot, which is a combination of OpenMP and MPI programming models [47,48].

Applications such as GPUVerify [49,50] and PolyOMP [4] are static methods of identifying data races in CUDA, whereas RaceTM [51], GUARD [52], and KUDA [52,53] are dynamic methods. On the other hand, GMRace [54], GRace [55], Grace [56,57], and SESA [58] are hybrid methods of identifying runtime errors in CUDA. Our literature review revealed a broad divide concerning the testing options for tri-level programming frameworks. As such, the present study proposes a highly tailored validation tool for MPI, OpenMP, and CUDA programming models.

Although debuggers can successfully be used to validate code, they typically require clarification. Furthermore, some debugging tools are commercially available, while others such as PDT [59], AutomaDeD [60], Linaro DDT [61], which was formally known as AL-LINEA DDT [62], TotalView [63,64], MPVisualizer [65], Intel® Inspector [66], and Arm DDT [67] are not. Apart from that, as non-commercial debuggers identify the causes of a problem rather than evaluating or detecting issues, they cannot be categorized according to their evaluation methods. Many other tools have also been examined, such as a temporal-logic-based testing tool for runtime error detection [68].

Our literature review revealed a lack of testing tools capable of identifying runtime issues in a tri-level MPI + OpenMP + CUDA application. Therefore, the present study proposes a hybrid tool that pre-emptively pinpoints several real and potential issues in the C++ codebase.

Although extant studies have proposed many tools with which to evaluate parallel architectures, detecting dynamic and static issues in tri-level frameworks warrants closer attention. Moreover, for heterogeneous architecture, three-level frameworks still require further improvements. Therefore, it is critical to address the dearth of testing tools that are capable of identifying runtime issues for processes coded using a tri-level MPI + OpenMP + CUDA framework so that the processes can run effectively and efficiently.

Runtime Errors in MPI, OpenMP, and CUDA Programming Models

A compiler cannot identify runtime errors in parallel systems as the errors may arise only once the program is running. Hybrid models experience a higher number of errors. The explanations for these errors also differ from those of conventional errors [1,20,69,70,71,72,73,74]. In order for parallel systems to prove advantageous, they must be error-free. This section lists several errors that occur when MPI, OpenMP, and CUDA programming models are integrated into C++-based applications. Actual examples are used to illustrate these concerns. The following codes, written in C++, used the MPI, OpenMP, and CUDA programming models. It is possible for some mistakes to arise from errors in either of the three models, whereas others may occur even if none have been found. The dependency graph below illustrates how this dependency affects the errors.

Multiple studies have thoroughly examined deadlocks and their causes [75,76,77,78]. Deadlocks usually occur when two or more threads await an input from the other. This creates an infinite loop between the threads, which prevents either of the threads from moving forward. The MPI, OpenMP, and CUDA programming models are prone to deadlocks [72].

A data race can be created in the background, using a lot of threads. For instance, a data race occurs when a shared memory resource is accessed by multiple threads, with at least one of those threads being able to write. However, the result may differ if the threads are processed in a different order. Therefore, shared variables should be synchronized to prevent data races. Each programming model uses a different technique to detect data races [79,80,81,82,83,84,85,86]. In a multi-threaded application, a data race occurs when several threads simultaneously access the same shared variable.

The OpenMP model simplifies the development of a parallel program in a shared-memory setting. When working with shared resources, OpenMP requires synchronization tools, such as locks, to avoid data races and deadlocks. The basic locking and unlocking functions provided by OpenMP include omp_set_lock, omp_unset_lock, and omp_test_lock, while the omp_init_lock and omp_destroy_lock functions are used for initialization and destruction. To prevent concurrent changes to the data in a shared resource, these functions ensure that only a single thread can access a critical section at a time.

The OpenMP model also features nested locks, which enables threads to set locks more than once. Nested locks require the same number of unsets to release. They also record the number of times that they have been set by using a thread. These functions can be used to launch, destroy, set, unset, and test a nested lock. They also offer a higher level of control for complicated synchronization settings. When a lock is no longer required, it should be destroyed using the omp_destroy_lock or omp_destroy_nest_lock functions. As data races occur when multiple threads concurrently access the same shared resource without proper synchronization, their outcomes are unpredictable and depend on the timing of the threads. For instance, implementing a shared resource in a parallel system without establishing locks can lead to a data race. This can be solved by using the omp_set_lock and omp_unset_lock functions to protect the shared resource by ensuring that only one thread can modify it at a time.

A deadlock occurs when multiple threads acquire multiple locks but in different orders. This causes two or more of the threads to be blocked forever, perpetually waiting for the other thread to release them. In this case, both threads will be stuck indefinitely if one thread sets lock A and waits for lock B while another thread sets lock B and waits for lock A. Therefore, all the threads must acquire the locks in the same order to prevent deadlocks and avoid circular dependencies.

Parallel programs can use OpenMP’s basic and nested locks to synchronize their access to shared resources and prevent data races. However, it is imperative to ensure consistent locking across the threads and to destroy locks after they are no longer required to prevent deadlocks. It is believed that when developers understand and apply these concepts, parallel applications can be written safely and efficiently.

Listing A1 (Appendix A), below, provides an example of a deadlock that occurs when the same lock, lock_a, in this case, is set twice using the same thread without being unset between the lines. As the second omp_set_lock and lock_a do not release the lock in the same thread, the program hangs.

Listing A2 (Appendix A) below, provides another example of a deadlock that occurs when lock_a is set in each iteration of the loop in the parallel region of the OpenMP section. Here, each thread tried to set the lock, perform an operation, and then unset the lock. However, the lock was destroyed in the parallel region, which led to undefined behavior. Therefore, in this example, the improper handling of the lock caused the threads to wait indefinitely for the lock to be released, resulting in a deadlock.

Listing A3 (Appendix A), below, provides an example of a deadlock that occurs when lock_a is set in the first section, in Line 10, and then unset and destroyed in the second section, in Lines 15 and 16. As the sections ran concurrently, the first section never released the lock, causing the second section to wait indefinitely for the lock to be released before it could destroy it. Therefore, in this example, the lock should have been set and unset within the same section, and the lock should have been destroyed after all the sections had been completed.

The OpenMP model requires the use of lock and unlock commands in conjunction with conditional statements. As such, it is crucial to ensure that locks are handled correctly in all possible execution paths, as leaving a lock set indefinitely could result in a deadlock or threads that are blocked indefinitely. Listing A4 (Appendix A), below, provides an example of a deadlock that occurs in the first section of OpenMP in Lines 8–15. As observed, lock_a was set in Line 8 and conditionally unset based on the available flag in Lines 11–13. If the available flag was false, the lock was properly unset. However, if the available flag was true, the lock remained set while the remaining work was performed before it was unset at the end. Later on, if another section or thread attempts to set the lock without first ensuring that it has been properly released from all conditions, it causes a deadlock. Therefore, every path that sets the lock must also unset it before the end of the work or before another section or thread tries to set the same lock again.

In Listing A5 (Appendix A), the code enters the parallel region of the OpenMP section using the nowait clause with the #pragma omp parallel and #pragma omp functions. The nowait clause allows multiple sections to run concurrently without having to wait for each section to complete. However, a deadlock occurs due to the incorrect placement of the lock and the order of the locks acquired in the two OpenMP sections.

In the first section, which began at Line 2, lock_a was acquired using the omp_set_lock(&lock_a) function (Line 3). After some of the work was carried out, lock_b was acquired using the omp_set_lock(&lock_b) function (Line 5). After the work was complete, lock_b was released using the omp_unset_lock(&lock_b) function (Line 7), followed by lock_a using the lock_a omp_unset_lock(&lock_a) function (incorrectly missing in the code).

In the second section, which began at Line 11, lock_b was acquired using the omp_set_lock(&lock_b) function (Line 13). After some of the work was carried out, lock_a was acquired using the omp_set_lock(&lock_a) function (Line 15). After the work was complete, lock_a was released using the omp_unset_lock(&lock_a) function (Line 17), followed by lock_b using the omp_unset_lock(&lock_b) function (Line 18). This created a deadlock as each section had to wait for the other section to release the lock, creating a circular dependency.

In Listing A6 (Appendix A), Lines 36 and 37 caused a deadlock. If Process 2 (P2) arrives at the first receiver (MPI_Recv) in Line 36 but the second receiver is in Line 37 and is still waiting for a message from Process 2, it will end up waiting indefinitely. However, if Process 1 (P1) arrives first and sends its message, the program will run without issues.

Listing A7 (Appendix A) depicts the specific section of the MPI code where a deadlock occurs due to mismatched tags. The deadlock occurred as Process 0 was waiting to receive a message from Process 1 with Tag 2. However, Process 1 sent a message with Tag 1 and Process 0 waited to receive the message with Tag 2. As a result, both processes ended up waiting indefinitely for messages that the other process did not send.

Listing A8 (Appendix A) depicts the specific section of the MPI code where a deadlock occurred due to a mismatch of the types of data in the MPI_Send and MPI_Recv operations. More specifically, Process 0 sent an integer (MPI_INT) but expected to receive a double (MPI_DOUBLE). This mismatch caused undefined behavior and the program to crash, as the data types must match for successful communication. A deadlock can also occur due to a mismatch in the tags. For instance, as observed in Line 27 of Process 0, the MPI_Send operation sent a message with Tag 0 but the MPI_Recv operation in Line 28 expected to receive a message with Tag 2. Furthermore, as observed in Line 32 of Process 1, the MPI_Recv operation expected to receive a message with Tag 0 but the MPI_Send operation in Line 44 sent a message with Tag 1. As a result, both processes ended up waiting indefinitely for messages that the other process did not send.

In Listing A9 (Appendix A), Process 0 transmitted Array A to Process 1 with Tag 0 and Array B to Process 2 with Tag 0. Process 1 received Array A from Process 0 with Tag 0 and transmitted the updated Array A to Process 3 with Tag 0. At the same time, Process 2 received Array B from Process 0 with Tag 0 and then transmitted the updated Array B to Process 3 with Tag 0. Finally, Process 3 waited to receive Array A from Process 1 with Tag 0 and Array B from Process 2 with Tag 0. This sequence of actions causes a deadlock if any of the tags do not match or if the operations have not been properly synchronized, as Process 3 will wait indefinitely for messages that may not arrive.

In Listing A10 (Appendix A), below, in Line 2, Process 0 sent Array A to Process 1 with Tag 0. In Line 3, Process 0 sent Array B to Process 2 with Tag 0. In Line 4, Process 0 sent Array B to Process 3 with Tag 0 without a corresponding receiver, which can lead to hidden resource leak issues. In Line 8, Process 1 received Array A from Process 0 with Tag 0. Process 1 later sent the updated Array A to Process 3 in Line 16 with the same tag. Similarly, in Line 21, Process 2 received Array B from Process 0 with Tag 0 and then sent the updated Array B to Process 3 in Line 26. Finally, in Lines 31 and 32, Process 3 waited to receive Array A from Process 1 with Tag 0 and Array B from Process 2 with Tag 0. However, Process 3 ended up waiting indefinitely for messages that have not been sent. For example, if Processes 1 or 2 fail to send the updated arrays to Process 3, or if Process 3 starts receiving before the arrays are sent, a deadlock will occur.

In Listing A11 (Appendix A), an atomic compare-and-swap (CAS) operation was used to implement a semaphore with which to control access to a critical section of the CUDA kernel. As observed, a deadlock occurred in Line 13 due to the way that the atomic CAS operation was used. If the semaphore was not released properly, the threads spin indefinitely.

In Listing A12 (Appendix A), in Line 16, a data race occurred as the threads had not been synchronized properly after executing the kernel. The CUDA kernel does not ensure that the read–modify–write operations on device_vecA are performed atomically or synchronized across threads. In other words, it attempts to add the value of device_vecA[index] to itself for all indices except the last one. However, this can lead to a data race, where multiple threads may attempt to simultaneously read and write to device_vecA[index], leading to a data race as the read and write operations are not atomic.

In Listing A13 (Appendix A), the kernel had an infinite loop (while (true)) that continuously checked and updated elements of device_vecA between Lines 13 and 19. Therefore, the threads were perpetually running without ever exiting the loop. More specifically, the threads kept updating the same memory location in an infinite loop without any mechanism with which to break out of it, leading to a livelock.

Table 1. The errors that may occur in each programming model and their effects on the tri-programming model.

CUDA	MPI	OpenMP	Tri-Programming Model
Error-free	Error-free	Error-free	No error
Data race	Error-free	Error-free	Data race
Deadlock	Error-free	Error-free	Deadlock
Potential deadlock	Error-free	Error-free	Potential deadlock
Livelock	Error-free	Error-free	Livelock
Potential livelock	Error-free	Error-free	Potential livelock
Error-free	Deadlock	Error-free	Deadlock
Error-free	Potential deadlock	Error-free	Potential deadlock
Error-free	Data race	Error-free	Data race
Error-free	Potential data race	Error-free	Potential data race
Error-free	Error-free	Deadlock	Deadlock
Error-free	Error-free	Potential deadlock	Potential deadlock
Error-free	Error-free	Data race	Data race
Error-free	Error-free	Potential data race	Potential data race
Error-free	Error-free	Livelock	Livelock
Error-free	Error-free	Potential livelock	Potential livelock

lists the errors that may occur in each of the programming models and how these errors will impact the tri-programming model. For example, a CUDA model may experience a deadlock, data race, potential deadlock, or potential livelock, each of which may cause distinct runtime errors in the CUDA model; however, the MPI and OpenMP models may still function flawlessly. The table shows the serious repercussions of even one model failing, including data races that lead to unreliable outcomes as well as deadlocks that can cause the program to crash. Furthermore, apart from their complexity, deadlocks and livelocks can make programs sluggish and even cause them to fail.

Table 2. The errors that may occur in either of the two programming models and their effects on the tri-programming model.

CUDA	MPI	OpenMP	Tri-Programming Model
Data race	Deadlock	Error-free	Deadlock
Data race	Race condition	Error-free	Data race, with possibly incorrect results
Deadlock	Deadlock	Error-free	Deadlock
Deadlock	Race condition	Error-free	Deadlock
Livelock	Deadlock	Error-free	Stuck, depending on which error occurs first
Livelock	Data race	Error-free	Livelock
Data race	Error-free	Deadlock	Deadlock
Data race	Error-free	Data race	Data race, with possibly incorrect results
Data race	Error-free	Livelock	Livelock
Deadlock	Error-free	Deadlock	Deadlock
Deadlock	Error-free	Data race	Deadlock
Deadlock	Error-free	Livelock	Deadlock
Livelock	Error-free	Deadlock	Stuck, depending on which error occurs first
Livelock	Error-free	Data race	Livelock
Livelock	Error-free	Livelock	Livelock
Error-free	Deadlock	Deadlock	Stuck, depending on which deadlock occurs first
Error-free	Deadlock	Data race	Deadlock
Error-free	Deadlock	Livelock	Stuck, depending on which error occurs first
Error-free	Data race	Deadlock	Deadlock
Error-free	Data race	Data race	Data race, with possibly incorrect results
Error-free	Data race	Livelock	Livelock

lists the errors that may occur in any of the two constituting programming model and how these errors will impact the tri-programming model. For example, if the OpenMP model does not experience any errors but the CUDA model experiences a data race and the MPI model experiences a deadlock, the deadlock will cause a system-wide failure in the tri-programming model. Furthermore, data races and livelocks may yield wrong results as they keep the program busy with concurrent access problems.

Table 3. The errors that may occur in all three programming models and their effects on the tri-programming model.

CUDA	MPI	OpenMP	Tri-Programming Model
Data race	Deadlock	Deadlock	Stuck, depending on which deadlock occurs first
Data race	Deadlock	Data race	Deadlock
Data race	Deadlock	Livelock	Stuck, depending on which error occurs first
Data race	Data race	Deadlock	Deadlock
Data race	Data race	Data race	Data race, with possibly incorrect results
Data race	Data race	Livelock	Livelock
Deadlock	Deadlock	Deadlock	Stuck, depending on which deadlock occurs first
Deadlock	Data race	Data race	Deadlock
Deadlock	Deadlock	Livelock	Stuck, depending on which error occurs first
Deadlock	Data race	Deadlock	Stuck, depending on which deadlock occurs first
Deadlock	Deadlock	Data race	Stuck, depending on which deadlock occurs first
Deadlock	Data race	Livelock	Stuck, depending on which error occurs first
Livelock	Deadlock	Deadlock	Stuck, depending on which error occurs first
Livelock	Data race	Data race	Livelock
Livelock	Deadlock	Livelock	Stuck, depending on which livelock occurs first
Livelock	Data race	Deadlock	Stuck, depending on which error occurs first
Livelock	Deadlock	Data race	Stuck, depending on which error occurs first
Livelock	Data race	Livelock	Stuck, depending on which livelock occurs first

lists the errors that may occur in all three constituting programming models and how these errors will impact the tri-programming model.

This highlights the complex correlations between the potential errors. The present study discusses several scenarios including data races, deadlocks, and livelocks in CUDA, MPI, and OpenMP programming models. Deadlocks and wrong results due to data races are among the wide range of results. This table, particularly, identifies scenarios in which errors may result in livelocks, deadlocks, or possibly wrong results.

It also provides a thorough examination of how these errors affect the overall performance of multi-model applications as well as the specific models in which they arise. These data are vital for developers working in HPC environments, where distributed and parallel systems are critical, as they can use them to anticipate, diagnose, and resolve complex runtime errors.

The occurrence and interdependence of runtime errors, specifically data races, deadlocks, prospective deadlocks, and livelocks, in programs that combine the MPI, OpenMP, and CUDA programming models are depicted in Figure 1, Figure 2 and Figure 3. These images supplement the textual information by providing a visual summary of how such errors develop and change across multi-model systems.

The complex nature of hybrid parallel settings is highlighted in Figure 1, which also presents a dependency graph depicting how the introduction of one model affects the others. The paths illustrate the necessity of strict synchronization and resource coordination by displaying both optimal (error-free) execution and transitions into different error states.

In Figure 2, the error-handling pathways for two of the three models are traced, both independently and convergently. It illustrates how errors can affect adjacent models even when they originate in the same execution stream. Their systemic significance is further supported by the fact that identical errors are noted in several models.

This idea is applied concurrently to all three models in Figure 3. It shows that if a single error type is handled incorrectly, especially data races and deadlocks, the entire program may fail, producing either inaccurate results or a system hang.

All these statistics highlight an important point: runtime mistakes in hybrid parallel systems are not isolated. When runtime mistakes appear in one component (such as MPI), they frequently set off a series of problems in other components (such as CUDA or OpenMP). As a result, the diagrams serve as diagnostic tools for comprehending the propagation of failures and the crucial role that synchronized execution plays in HPC, which surpasses basic illustration.

5. Static Testing Approach Implementation

Parallel systems are very difficult to test due to the many challenges that arise during the implementation process. As such, multiple studies have examined the possibility of developing a tool that covers all potential test case situations and data [78]. A static automated testing tool was developed to detect errors in MPI, OpenMP, and CUDA programming models in C++-based applications. The design of the proposed tool, as well as the dependency errors, were taken into consideration, as outlined in Table 1, Table 2 and Table 3 and Figure 1, Figure 2 and Figure 3. The tool will enable programmers to identify problems in the source code before a program is executed, log the problems, and view them in a separate log file containing all the necessary information to locate the errors. Therefore, all these tasks must be completed before the program is compiled. The runtime errors will also be listed in a dependency graph, which will enable programmers to resolve an error before running the program.

5.1. Analysis Phase

The proposed tool was created using the C++ programming language. The source code to be tested was first entered into a C or C++ development environment which contained the source code for the MPI, OpenMP, and CUDA tri-level programming model. In order to detect runtime errors, the proposed tool analyses the code and gathers information as follows:

• The MPI, OpenMP, and CUDA data are stored in vectors.
• These data include the start line number and the end line number of each model.
• The tool gathers information about the explicit barriers in the OpenMP model.
• The kernel definition, kernel launch, data transfer, and memory cleanup regions contain information about the start and end lines.
• The tool sets the start and end values of the variables in every parallel region in the two programming models.
• Finally, the tool records the ‘while’, ‘for’, and ‘do while’ loop data, the dependent variables, and the start and end values of these loops.

An organized log file is then created, where all these data are stored and presented to the programmer for review. Figure 4 illustrates part of the log file, depicting all the data collected from the MPI section, and reveals how the processes communicate in a distributed computing environment. It includes comprehensive data, such as the total number of MPI send and receive operations; their locations within the code, with specific line numbers; and confirmation on whether the corresponding send/receive operations were successfully found. It also details the specific variables involved in the MPI operations, outlines the data types and the amount of data transferred, and describes the MPI tags and communicators used in each operation.

Figure 5 illustrates part of the log file depicting all the data collected from the CUDA and OpenMP sections, both of which are crucial components of parallel programming but with a focus on the different aspects of HPC. Using functions such as omp_init_lock, omp_set_lock, and omp_parallel, the OpenMP section targets parallel programming in a shared-memory computing environment. To help developers effectively manage synchronization, it highlights critical variables, such as locks (OMP Lock), and captures where they were initialized, set, and unset. As a result, the parallel sections of code run smoothly and efficiently.

The CUDA section focuses on the parallel computing capabilities of NVIDIA’s GPUs. The CUDA model’s memory management functions, such as cudaMalloc and cudaMemcpy, are examined and their call locations are identified to ensure efficient memory management in the GPU. It also documents the launch of the CUDA kernels, indicating where and how they were executed on the GPUs. In-depth details about each memory transfer, including the sources, destinations, and sizes, are also recorded. This meticulous documentation facilitates the optimal use of memory and increases the efficiency of data transfer between the host and the GPU. This demonstrates the synergistic correlation between these two powerful technologies in the field of modern computing.

5.2. Detection Phase

It is necessary to acknowledge and address conditions that are often overlooked, such as the interactions between various regions, critical regions, sections, shared resources, reduction clauses, and synchronization constructs—all of which are common when programming in parallel. The proposed tool is capable of identifying any potential problems with these runtimes. Even though these errors may not have been evident during compilation, the tool can identify and clearly display them to the programmer.

Algorithm 1 is capable of determining whether an MPI deadlock will arise when there is no matching MPI_Recv for an MPI_Send or if there is no matching MPI_Send for an MPI_Recv. Therefore, a deadlock will occur if there is any difference in the type of data, tag, communicator, or count.

Algorithm 1: The MPI deadlock print error when there is a send without a receive or a receive without a send
1:	OPEN THE INPUT CODE FILE
2:	SEARCH for mpi_send and mpi__recv which one happen first
3:	if (mpi_send is found)
4:	IDENTIFY receiver rank
5:	SEARCH for matching mpi_recv at the receiver
6:	if (mpi__recv is found)
7:	MATCH  tag, count, sender rank, communicator, datatype
8:	if (there is any MISMATCH)
9:	CONTINUE  Search
10:	else
11:	Matching  mpi__recv found
12:	end if
13:	if (no matching mpi__recv is found at the receiver rank)
14:	“Print Error: Deadlock No corresponding RECV found”
15:	end if
16:	end if
17:	if (mpi__recv is found)
18:	IDENTIFY  Sender rank
19:	SEARCH  for a matching mpi_send at the Sender
20:	if (mpi_send is found)
21:	MATCH  tag, count, receiver rank, communicator, datatype
22:	if (there is any MISMATCH)
23:	CONTINUE Search
24:	else
25:	Matching  mpi_send found
26:	end if
27:	if (no matching mpi_send is found at the sender rank)
28:	“Print Error: Deadlock No corresponding SEND found”
29:	end if
30:	end if

An MPI livelock occurs when processes are stuck in an infinite loop and exchanging messages with each other. As observed in Algorithm 2, if an MPI process is found to be communicating inside an infinite loop that is not broken anywhere, this error will pop up.

Algorithm 2: The MPI livelock print error when an infinite loop develops
1:	OPEN THE INPUT CODE FILE
2:	if (while(true) statement is found in any process)
3:	SEARCH  for a possible BREAK of the loop
4:	if (loop BREAK statement is not found)
5:	“Print Error: Livelock MPI process stuck in an infinite loop”
6:	end if
7:	end if

An MPI data race occurs when a process sends multiple immediate sends to another process without calling the MPI_Wait routine. As observed in Algorithm 3, this could occur due to a conflict in the order of receiving these messages. As such, it is possible for the messages to be processed out of order, leading to inaccurate results.

Algorithm 3: The MPI data race print error when an infinite loop develops
1:	OPEN THE INPUT CODE FILE
2:	if (mpi_Isend is found)
3:	IDENTIFY  receiver rank
4:	SEARCH  for a matching mpi_Irecv at the receiver
5:	if (mpi_Irecv is found)
6:	MATCH  tag, count, sender rank, communicator, datatype
7:	if (there is any MISMATCH)
8:	CONTINUE  Search
9:	else
10:	Matching  mpi_Irecv found
11:	end if
12:	if (matching mpi_Irecv is found)
13:	SEARCH  for mpi_wait statement after mpi_Isend
14:	if (there is no mpi_wait statement)
15:	“Print Error: Data Race Not waiting for messages to be received before sending again”
16:	end if
17:	else
18:	“Print Error: Deadlock No corresponding  mpi_Irecv  found”
19:	end if

An OpenMP deadlock will occur if a lock is set but not unset. In this scenario, the same logic is applied. As seen in Algorithm 4, the name of the lock variable is identified if a lock is set. If a corresponding unset lock is not found in the remaining lines of the code, it means that a deadlock will occur.

Algorithm 4: The OpenMP deadlock print error when a lock is misused
1:	OPEN THE INPUT CODE FILE
2:	if (OMPSetlock is found)
3:	IDENTIFY  lock  variable
4:	SEARCH  for a corresponding OMPSetlock
5:	if (OMPSetlock is not found)
6:	“Print Error: Deadlock  Lock  is set but not unset”
7:	end if
8:	end if

An OpenMP deadlock will occur if an infinite loop develops in a critical section as the threads access the critical section sequentially. Therefore, when a thread enters a critical section, the other threads wait for it to finish executing. As observed in Algorithm 5, in an infinite loop, one thread will be stuck in the critical section and the other threads will be stuck outside it, creating a deadlock.

Algorithm 5: The OpenMP deadlock print error when an infinite loop develops in a critical section
1:	OPEN THE INPUT CODE FILE
2:	if (OMPcritical construct is found)
3:	CHECK  that there is no infinite while loop in the critical section
4:	if (infinite while loop is found inside critical section)
5:	“Print Error: Deadlock because One thread enters the critical section and the others keep on waiting”
6:	end if
7:	end if

An OpenMP livelock occurs if an infinite loop develops in the OpenMP section. Algorithm 6 checks for a possible break in the loop. If a break statement is not found, a livelock will develop in the OpenMP section.

Algorithm 6: The OpenMP livelock print error when an infinite loop develops in a #pragma construct
1:	OPEN THE INPUT CODE FILE
2:	if (while(true) statement is found inside a #pragma construct)
3:	SEARCH for a possible  BREAK  of the loop
4:	else if (loop BREAK statement is not found)
5:	“Print Error: Livelock occur because threads stuck in an infinite loop”
6:	end If
7:	end if

A CUDA deadlock occurs if an atomicCAS operation is used improperly. In Algorithm 7, a semaphore for controlling access to a critical section statement was deliberately omitted from the kernel for illustration purposes. An improperly released semaphore can cause the threads to spin indefinitely as they wait to access the semaphore.

Algorithm 7: The CUDA print error when an atomicCAS operation is misused
1:	OPEN THE INPUT CODE FILE
2:	SEARCH for  cudakernel
3:	IDENTIFY  cudakernel  name
4:	if (atomicCAS statement if found inside cudakernel)
5:	“Print Error: Deadlock Because Only one thread allowed to access critical section”
6:	end if

As seen in Algorithm 8, a CUDA livelock occurs when it is called repeatedly in an infinite loop without a break statement.

Algorithm 8: The CUDA livelock print error when a kernel is called repeatedly in an infinite loop
1:	OPEN THE INPUT CODE FILE
2:	IDENTIFY  cudakernel  name
3:	FIND where  cudakernel  is being called
4:	if (cudakernel is being called in an infinite loop)
5:	“Print Error: Livelock: Kernel is being called repeatedly”
6:	end if

An OpenMP data race occurs when a thread tries to update or access values that are updated by another thread. As seen in Algorithm 9, when a thread tries to access the index of an array other than its assigned indices, a data race may occur. In the pseudocode below, the loop variable of an OpenMP loop is first identified. Then, if a thread accesses the index of an array other than its loop variable, a data race occurs.

Algorithm 9: The OpenMP data race print error
1:	OPEN THE INPUT CODE FILE
2:	SEARCH for #PRAGMA construct
3:	If (#PRAGMA construct is found)
4:	FIND if it is OMP PARALLEL or OMP PARALLEL FOR
5:	If (OMP PARALLEL is found || OMP PARALLEL FOR IS FOUND)
6:	CHECK if there is a for loop in this section
7:	If (for loop IS FOUND)
8:	IDENTIFY the loop variable
9:	If (an array is using an index other than the loop variable)
10:	“Print Error: DATA RACE: Accessing values that may be
	updated by other threads”
11:	end If
12:	end If
13:	end If
14:	end If

In Algorithm 10, a CUDA data race occurs if the threads are not synchronized after the kernel call. This is because, if there are multiple kernels in a program, some of the threads may start to execute the next kernel before all the threads have finished completing the first.

Algorithm 10: The CUDA data race print error
1:	OPEN THE INPUT CODE FILE
2:	SEARCH for CUDA Kernel
3:	IDENTIFY CUDA Kernel name
4:	“Print Error: DEADLOCK: Only one thread allowed to access critical section “
5:	end If

The proposed error dependency graph is a reliable tool with which to pinpoint issues after analyzing the source code and generating lists of runtime errors for each programming model. It clearly demonstrates that a deadlock or a livelock in any model can cause a deadlock in the entire program. Leveraging Algorithm 11, an error dependency scheme was effectively implemented. Furthermore, it is evident that data races in all three models can lead to data races or unexpected behavior in the program.

Algorithm 11: The dependency errors between the three models in the tri-programming model
1:	IF MPIDEADLOCK == TRUE
2:	“Print Error: Deadlock in the whole system because there is a deadlock in MPI”
3:	else if ompdeadlock == True
4:	“Print Error: Deadlock in the whole system because there is deadlock in OpenMP”
5:	else if  cudadeadlock ==True
6:	“Print Error: Deadlock in the whole system because there is deadlock inCUDA”
7:	if  mpiLivelock  == True
8:	“Print Error: Livelock in the whole system because there is livelock in MPI”
9:	else if  ompLivelock ==  True
10:	“Print Error: Deadlock in the whole system because there is livelock in OpenMP”
11:	else if  CudaLivelock ==  True
12:	“Print Error: Deadlock in the whole system because there is livelock inCUDA”
13:	else if  mpirace ==  True  omprace ==  True
14:	“Print Error: Race condition in the system because there is race condition in MPI and OpenMP”
15:	else if  mpirace ==  True AND  cudarace ==  True
16:	“Print Error: Race condition in the system because there is race condition in MPI and CUDA”
17:	else if  mpirace ==  True
18:	“Print Error: Race condition and wrong result in the whole systembecause there is race condition in MPI”
19:	else if  omprace ==  True AND  cudarace ==  True
20:	“Print Error: Race condition the system because there is race condition in OpenMP and CUDA”
21:	else if  omprace ==  True
22:	“Print Error: Race condition in the system because there is racecondition in OpenMP”
23:	else if  CudaRace ==  True
24:	“Print Error: Race condition in the system because there is race condition. in CUDA”

The threads have to be closely monitored to prevent deadlocks. Dynamic testing codes, which will be implemented in a subsequent study, will be seamlessly integrated into the header file, which will help to deliver comprehensive reports to programmers regarding the test’s status.

6. Testing and Evaluation

The runtime error detection performance of the proposed tool was compared with that of multiple existing benchmark programs. The tests were conducted on a laptop equipped with an 11th Generation Intel^® Core™ i7-10750H processor (2.66 GHz, 12 threads) that had 16 GB of RAM and an NVIDIA^® GeForce GTX 1650 GPU. The operating system (OS) was Ubuntu™ 20.04.4 LTS, which served as the development environment. Specialized compilers that suited each parallel computing framework were used in the compiling process. More specifically, the NVIDIA^® CUDA compiler (NVCC), which is provided in the CUDA Toolkit (recommended version 12.2 or newer), was used for the CUDA model. The MPI compiler wrapper (MPICC), which is provided in OpenMPI (version 4.1.5) or MPICH (version 3.4.3), was used for the MPI model. The GNU compiler collection (GCC) (version 11.3.0 or newer) was used for the OpenMP model as it supports OpenMP 5.0 and facilitates efficient multi-threading. Therefore, in the compilation process, NVCC was used for the CUDA code, MPICC was used for MPI-based parallelization, and the OpenMP flag in the GCC was used to enable the functionality of OpenMP.

A total of 2 criteria were utilized to benchmark the performance of the MPI section of the proposed tool. These included the NAS parallel benchmarks (NPB) [87], the Edinburgh Parallel Computing Centre (EPCC) OpenMP microbenchmarks [88], and mpiBench [89,90]. In addition, OpenMP DataRaceBench version 1.4.0 [91,92], NPB-C++ parallel programming (CPP) version 3.3.1 [93], NPB version 3.4.1 [94], and the HPC group at RWTH Aachen University’s DRACC microbenchmark suite [95] were used to benchmark the performance of the CUDA section. As shown in

Table 4. The benchmark programs used to compare the performance of the tri-programing model.

Model	Benchmark Program		Lines
MPI	NAS	DT	714
		IS	1179
	EPCC	PingPong	774
		PingPing	400
		Broadcast	169
		ParallelEnvironment	353
	mpiBench	mpiBench	1077
OpenMP	RWTH-HPC/DRACC	DRB001	19
		DRB017	22
		DRB021	24
		DRB023	15
		DRB031	21
		DRB037	17
		DRB104	29
	NPB-CPP/NPB-OMP	CG	1027
		EP	334
		FT	1184
		IS	796
		MG	1388
CUDA	RWTH-HPC/DRACC	DRAAC_CUDA_001	66
		DRAAC_CUDA_004	93
		DRAAC_CUDA_007	85
		DRAAC_CUDA_020	120
		DRAAC_CUDA_023	75
		DRAAC_CUDA_027	67

provides a brief overview of the benchmarks used in this study. Figure 6, Figure 7 and Figure 8 illustrate the effectiveness of the proposed tool. The x-axis indicates which codes were tested using the selected benchmark programs, the left y-axis indicates how many lines in each code were present, and the right y-axis indicates the execution time of each benchmark program, which was measured in clocks per second.

Figure 9, which is a screenshot of the log file of one of the tests, shows the errors that the compiler did not detect during the test.

Figure 10 shows the errors that the proposed tool detected in one of the codes during the test. The final summary highlights the primary problem with integrating these three programming models. The proposed tool provided a summary of the errors, which is displayed at the end of the report, that would occur in the program. In the dynamic testing stage, which will be developed in the next study, these potential errors will be verified.

The proposed tool is the first of its kind to analyze C++-based applications integrating MPI, OpenMP, and CUDA programming models. This study aimed to not only compare the proposed tool with existing static analysis tools but to also assess error detection challenges introduced by increasing model complexity. To this end, during these studies, we systematically evaluated runtime error patterns across single-model (MPI, OpenMP, or CUDA individually), dual-model (MPI + OpenMP), and tri-model (MPI + OpenMP + CUDA) applications.

The investigation found a significant increase in error complexity and propagation as more models were integrated. Single-model configurations were characterized by isolated errors in the model, such as data races or deadlocks. Tri-model applications had the highest frequency of interdependent runtime errors, whereas dual-model systems displayed interactions that led to severe synchronization problems. These results underscore the drawbacks of conventional tools in managing hybrid environments.

Table 5. The error coverage of the proposed tool for runtime error detection in tri-programming models.

Programming Model	Runtime Error	Our Tool (Static Approach) *	ACC_TEST *	ROMP *	LLOV *	ARCHER *	MPR ACER *	GPUVerify
OpenMP	Race Condition
	Loop Parallelization Race	✓	✕	✓	✓	✓	✓	✕
	NowaitClause	✓	✕	✓	✕	✓	✓	✕
	Shared Clause	✓	✕	✓	✓	✓	✓	✕
	Barrier Construct	✓	✕	✓	✕	✓	✓	✕
	Atomic Construct	✓	✕	✓	✓	✓	✓	✕
	Critical Construct	✓	✕	✓	✓	✓	✓	✕
	Master Construct	P	✕	✓	✓	✓	✓	✕
	Single Construct	P	✕	✕	✓	✕	✓	✕
	SIMD Directives	✕	✕	✕	✓	✕	✓	✕
	Teams Construct	✕	✕	✓	✓	✕	✓	✕
	Deadlock
	LockRoutines	✓	✕		✕	✕	✕	✕
	Section Construct	✓	✕		✕	✕	✕	✕
	Barrier Construct	✓	✕		✕	✕	✕	✕
	NowaitClause	P	✕		✕	✕	✕	✕
	Livelock	P	✕		✕	✕	✕	✕
	Device Directives	✕	✕		✕	✕	✕	✕
CUDA	Race Condition		✕		✕		✕
	Host/Device Synchronization Race	✓	✕	✕	✕	✕	✕	✓
	Asynchronous Directive Race	✓	✕	✕	✕	✕	✕	✓
	Deadlock
	Device Deadlock	✓	✕	✕	✕	✕	✕	✕
	Host Deadlock	P	✕	✕	✕	✕	✕	✕
	Livelock	P	✕	✕	✕	✕	✕	✕
MPI	Point-to-Point Blocking/ Nonblocking Communications
	Illegal MPI Calls	✓	✓	✕	✕	✕	✓
	Data Type Mismatching	✓	✓	✕	✕	✕	✓
	Data Size Mismatching	✓	✓	✕	✕	✕	✓
	Resource Leaks	✓	✓	✕	✕	✕	✕
	Inconsistence Send/Recv Pairs Wildcard Receive	✓	✓	✕	✕	✕	✓
	Race Condition	P	P	✕	✕	✕	✓
	Deadlock	P	P	✕	✕	✕	✓
	Collective Blocking/ Nonblocking Communications					✕
	Illegal MPI Calls	✓	✓	✕	✕	✕	✓
	Data Type Mismatching	✓	✓	✕	✕	✕	✓
	Data Size Mismatching	✓	✓	✕	✕	✕	✓
	Inconsistent Send/Recv Pairs—Wildcard Receive	P	P	✕	✕	✕	✓
	Race Condition	P	P	✕	✕	✕	✓
	Deadlock	P	P	✕	✕	✕	✓

* ✓: Fully detected; P: Partially detected; ✕: Not detected.

provides an overview of related runtime-testing tools. Notably, the proposed tool is the only one capable of detecting both deadlocks and livelocks across all three models in C++ applications. It also offers structured error logs and dependency graphs that contextualize runtime behaviors, further validating its applicability in multi-model parallel programming environments.

7. Conclusions and the Future Directions

In this study, the researchers designed a tool for identifying runtime errors caused by data races, deadlocks, and livelocks in C++-based systems that employ the MPI, OpenMP, and CUDA programming models. Before executing the code, errors were detected using a static testing technique. The proposed tool could also detect errors in any programming paradigm and generate an error dependency tree for MPI, OpenMP, and CUDA runtime issues. Furthermore, the researchers also proposed a taxonomy of the runtime mistakes that result from combining the MPI, OpenMP, and CUDA programming paradigms and described various strategies for monitoring these errors.

The proposed application demonstrated better performance than other tools in detecting issues during benchmarking tests using C and C++ code. The proposed tool was the only tool that could identify deadlock errors in the MPI, OpenMP, and CUDA programming models as well as identify the problems that could occur during integration. These features help to guarantee that the application is error-free and increase its dependability. It is worth noting that using the proposed tool does not slow down an application; instead, it displays a list of errors that the programmer needs to address. This paper presents the static component of our hybrid tool, which is available as open-source software in [96].

The researchers of this study aim to improve the effectiveness of the proposed tool by identifying several MPI, OpenMP, and CUDA runtime issues that are currently not supported by the tool. A hybrid version of the proposed tool that incorporates a dynamic technique to detect runtime problems as they are being executed will also be designed in a future study.