Adaptive-Hybrid Redundancy with Error Injection

Abstract

Adaptive-Hybrid Redundancy (AHR) shows promise as a method to allow flexibility when selecting between processing speed and energy efficiency while maintaining a level of error mitigation in space radiation environments. Whereas previous work demonstrated AHR’s feasibility in an error free environment, this work analyzes AHR performance in the presence of errors. Errors are deliberately injected into AHR at specific times in the processing chain to demonstrate best and worst case performance impacts. This analysis demonstrates that AHR provides flexibility in processing speed and energy efficiency in the presence of errors.

1. Introduction

Adaptive-Hybrid Redundancy (AHR) was developed to enable flexibility in radiation hardening redundancy methods for space vehicles. AHR incorporates Triple Modular Redundancy (TMR) and Temporal Software Redundancy (TSR) such that AHR can switch between TMR and TSR modes as needed [1]. This previous work demonstrated that AHR functions as designed, switches from TMR to TSR, and uses less energy to complete programs than TMR while completing those programs in less time than TSR in an error free environment [1]. The objective of this paper is to further illustrate the advantages and flexibility of AHR when compared to TMR or TSR alone both in error free and error prone simulated environments. This paper does not seek to prescribe how much time AHR should spend in TMR or TSR operating modes, but rather to provide a new redundancy framework to space vehicle designers, mission planners, and operators so they can decide how much time AHR should spend in each mode based upon radiation environment, processing speed requirements, energy consumption requirements, and mission requirements.

The remainder of this section discusses previous redundancy mitigation research and the architecture of AHR. Section 2 discusses the methods used to evaluate the performance of AHR. Section 3 discusses the results of the AHR performance evaluation. Section 4 discusses the impact of the results and concludes the paper.

1.1. Background

Previous Single-Event Upset (SEU) mitigation research has focused on hardware, software, hybrid, or adaptive redundancy techniques. This section will briefly review some of the research leading up to AHR.

1.1.1. Hardware Redundancy

Dual-Modular Redundancy (DMR) describes an SEU mitigation method that operates two processors in parallel by providing those processors simultaneous identical inputs and operating those processors on a common clock. A hardware comparator module compares the outputs of the two processors to ensure that they are identical. If the comparator detects a difference, an error has occurred, and both processors’ internal state is restored to a previous known state that was stored to radiation hardened/immune memory. In DMR, the processors are periodically interrupted to save their internal state to memory; this is called a Save/Restore Point [2,3,4,5,6]. To create the Save/Restore Point, the comparator issues commands to both processors to write each register to memory. Upon receiving register values from the two processors, they are compared to ensure they are equal before writing the result to memory. If an error is detected at this stage, the comparator enters error recovery mode and returns to the last Save/Restore Point. After all registers are written to memory, the Program Counter value is written to memory as well. During error recovery, the comparator resets both processors, then issues a series of load commands to both processors to load each register value in the Save/Restore Point into registers. The comparator finishes the error recovery process by issuing a branch command to return the processors to the Program Counter value stored in the Save/Restore Point. The comparator performs Save/Restore Point creation and error recovery by traversing a series of states in its internal finite state machine which is implemented in hardware.

The timing of Save/Restore Point creation is application specific and depends on factors such as radiation environment and operational needs. The Save/Restore Point time period is specified as a set number of instructions to complete between Save/Restore Points. Save/Restore Point creation takes time away from processors performing intended tasks and slows execution time so there is some pressure to maximize the amount of time between Save/Restore Point creation events. However, in the event an error occurs, the amount of time between Save/Restore Points dictates the amount of time that must be spent to recover from an error to the point at which the error initially occurred. This represents a pressure to reduce the amount of time between Save/Restore Point creation.

Triple-Modular Redundancy (TMR) describes an SEU mitigation method that operates three processors in a similar manner to DMR with some notable differences. First, the hardware comparator module is replaced with a hardware majority voter. Secondly, when the majority voter detects that two of three processors agree while one disagrees, the disagreeing processor is reset and the internal state of the two agreeing processors is copied to the disagreeing processor so that all three processors are in agreement. This greatly reduces the recovery time when compared to DMR. In the unlikely event that two or more processors encounter an error such that all three processors disagree, TMR restores all three processors to a previous Save/Restore Point in the same way as DMR: the voter contains a hardware implemented finite state machine to create the Save/Restore Point and recover from errors [7,8,9,10,11,12,13]. The voter is typically assumed to be immune to errors or is hardened in some way. This research assumes that the voter is immune to errors as well. While the voter is this research is not implemented in such a way to be error immune or radiation hardened, radiation hardening could be achieved through radiation shielding, hardware redundancy, or hardening by design.

N-Modular Redundancy (NMR) is a majority voting redundancy method that is similar to TMR, but uses N processors instead of three processors. It is considered more robust than TMR because a permanent single processor failure will result in N-1-Modular Redundancy. So long as N-1 is greater than or equal to three, the majority voting redundancy method still functions. However, there is an energy penalty to be paid because NMR uses at least N times as much energy to complete a program as a single processor with no redundancy [14,15,16].

1.1.2. Software Redundancy

The first software redundancy method this paper discusses, Error Detection by Duplicated Instructions (EDDI), is very similar to DMR, but is implemented in software instead of hardware. This software redundancy method runs on a single processor. Dual redundancy is implemented by duplicating all instructions that do not store data to memory. Each duplicated instruction stores its results to a different, physically separated register from the original instruction to achieve spatial separation of the original and duplicate results. This greatly reduces the likelihood that a single, or multiple, radiation event(s) will cause the exact same error in both the original and duplicated results. The DMR comparison function is implemented in software by adding a comparison instruction immediately prior to any store instruction. If the original and duplicate register are identical, the original is stored to memory. If the original and duplicate are not identical, an error has occurred and program execution jumps to code that performs error recovery. This error recovery code restores the state of the processor to a previously saved state called a Save/Restore Point in a similar manner to DMR, but use software instead of a hardware finite state machine. Similarly to DMR, EDDI periodically interrupts normal program execution to create Save/Restore points by executing code designed to create the Save/Restore points [17,18,19]. EDDI was proposed by Oh et al. [17,18] while Tokponnon et al. discuss a very similar software redundancy method [19].

Table 1. Simple software redundancy example.

Instruction Number	Original Set	Redundant Set
1	LUI R1 1	LUI R1 1
2	LUI R2 2	LUI R15 1
3	ADD R3 R1 R2	LUI R2 2
4	SW R3 R0 OFFSET	LUI R16 2
5		ADD R3 R1 R2
6		ADD R17 R15 R16
7		BNE R3 R17 ERR
8		SW R3 R0 OFFSET

shows an example of what EDDI code looks like in the “redundant set” when compared to a non-redundant instruction set called the “original set”. In this example, LUI is the MIPS load upper immediate instruction, ADD is the MIPS addition instruction, SW is the MIPS store word instruction, and BNE is the MIPS branch if not equal instruction. ERR is the value of the distance which the BNE should jump in code execution if R3 and R17 are not equal. OFFSET is the value that should be added to the memory location specified by R0 where R3 should be stored. Any value indicated by R# is a register number.

Oh et al. [20] and Reis et al. [21,22] improved EDDI by adding signature detection in order to determine when Program Counter errors have occurred as a result of a missed branch or illegal branch. Signature detection methods break a program into segments and computes segment signatures at compile time and inserts a segment signature computation and a segment comparison instruction into each segment. The signatures are unique and are dependent upon the preceding segment and the current segment. At runtime, the signature is recomputed and compared to the compile time signature. Any discrepancy between the two is interpreted as a Program Counter error as a result of a missed branch or illegal branch between segments [20,21,22].

1.1.3. Hybrid Redundancy

Hybrid redundancy can take many forms and can consist of any number of combinations of hardware, software and error correcting codes.

The first hybrid redundancy method this paper discusses is only applicable to Field Programmable Gate Arrays (FPGAs) which have a configuration memory that is vulnerable to Single Event Upsets (SEUs). Many FPGAs configuration memories are comprised of Static Random Access Memory (SRAM) cells which are highly susceptible to SEUs. These configuration memories specify constants, logic functions, and signal routing on the FPGA. Any of these can have a catastrophic effect on the intended function of the hardware designed into the FPGA. Those who wish to implement a processor on an FPGA typically combine a TMR-like method with a method of correcting configuration memory called internal scrubbing. The primary concern with FPGAs is the configuration memory rather than the registers used in the processor. Internal scrubbing detects and corrects errors in the configuration memory, but can only do so at the memory refresh rate. TMR is used as a stop-gap measure to ensure outputs are correct in spite of configuration errors until internal scrubbing can correct those errors. The TMR method only does majority voting and does not correct faulty processors [23,24,25,26,27,28].

Another method of hybrid redundancy duplicates instructions in software, similar to EDDI, but uses a hardware comparator similar to the DMR comparator [29]. A method that juxtaposes the software and hardware portions has also been implemented that uses hardware for redundancy and software for comparisons [30].

A few methods of hybrid redundancy combine hardware or software redundancy with error correcting codes to protect processor registers and/or memory [31,32,33].

1.1.4. Adaptive Redundancy

Only two adaptive redundancy approaches were found in the literature survey. The first is a very simple approach that uses a radiation sensor to detect the ambient radiation environment and determines when to implement TMR and when to operate using a single processor without any mitigation [34]. This is also the only example discovered in literature that applies adaptive redundancy to a processor.

The second uses three different software redundancy methods to protect memory. Each method differs in the degree of error protection, memory access speed, and energy consumption. When there are very few SEUs occurring, the method that provides the least error protection, operates the fastest, and uses the least amount of energy is utilized. As the SEU rate increases, the method that provides an intermediate level of error protection, intermediate memory access speed, and intermediate energy consumption is used. As the SEU rate becomes too great for the intermediate level of protection to handle, the method that provides the greatest level of error protection is used at the expense of the slowest memory access and greatest energy consumption [35].

1.2. Adaptive-Hybrid Redundancy

AHR, as implemented in this work and the previous work [1], consists of TMR and TSR. The TMR implementation is just as described in Section 1.1.1 and the TSR implementation is the EDDI method described in Section 1.1.2. For this research, the time between creation of Save/Restore Points was arbitrarily selected to be 10,000 instructions for TMR and AHR operating in TMR mode and 250 main program loops for TSR and AHR Operating in TSR mode (many real-world programs typically have a main loop which is repeated numerous times until program completion). These values were chosen to ensure that every program would create at least one Save/Restore Point during its execution. These are tunable parameters that a space vehicle designer, mission planner, or operator could change as needed based upon the program running on the processor, performance requirements, radiation environment, and mission needs.

A simple illustration of the TMR architecture is shown in Figure 1. The previous work demonstrated that a program running in an error free environment in TMR takes 65% longer to run than a program running on a single processor with no redundancy because the voter adds delay to all processor inputs and outputs. The TMR architecture also uses three times the instantaneous power of a single processor because there are three processors instead of one. A program running TMR takes approximately 430% more energy to complete due to the number of processors and the added time taken to run the program [1].

A simple illustration of the TSR architecture is shown in Figure 2. The EDDI TSR method uses a special compiler to take a normal program and make it into a TSR program. The previous work demonstrated that EDDI TSR programs take 113% longer to complete than non-redundant programs and the TSR architecture uses the same amount of instantaneous power as a single processor because the TSR architecture only uses a single processor. However, TSR uses 113% more energy to complete a program than a non-redundant program because it takes 113% longer to run that program [1].

The AHR architecture adds a module called the AHR Controller to the TMR architecture as shown in Figure 3. The AHR Controller is assumed to be immune from errors for this research just as the TMR voter is assumed to be immune from errors. When AHR operates in TMR mode, the TMR Voter and three processors operate normally and the signals between the TMR Voter and memory and from the TMR Voter to the three processors are passed through combinational logic in the AHR Controller with minimal delay. Figure 4 shows how signals flow when AHR is operating in TMR mode by illustrating connected signals and modules that are operational in black and those that are not in red. When operating in TSR mode, the AHR Controller turns off the TMR Voter and two of the three processors. The remaining single processor communicates directly with memory by passing signals through combinational logic in the AHR Controller with minimal delay. Figure 5 shows how signals flow when AHR is operating in TSR mode by illustrating connected signals and modules that are operational in black and those that are not in red.

AHR begins in TMR mode and switches to TSR mode after a predetermined number of TMR instructions are completed without encountering an error. AHR remains in TSR mode so long as two consecutive errors do not occur. Two errors are considered consecutive if a second error occurs after TSR recovers from a first error and the second error occurs before TSR can create a new save/restore point. If consecutive errors occur, AHR transitions to TMR when the second error is detected. If TSR creates a new save/restore point before encountering a second error, the second error is not consecutive and AHR continues in TSR mode. This approach gives TSR mode an opportunity to recover from errors so long as the error rate is sufficiently low.

The processor upon which AHR, TMR, and TSR were based on past work and are based in this work is the Basic MIPS processor which is a simplified MIPS32^TM processor that only supports 33 of the 168 MIPS32^TM instructions [36]. The details of the Basic MIPS, TMR MIPS, and AHR MIPS architectures are available in Air Force Institute of Technology technical reports [36,37,38].

AHR was shown to bridge the gap between these two methods by switching between TMR and TSR so that it runs faster than TSR and uses less energy than TMR at the expense of running slower than TMR and using more energy than TMR [1]. In past work, AHR started in TMR mode and switched to TSR mode after TMR successfully completed 15,000 instructions without an error in an error free simulation. This work will examine how AHR performs when the TMR to TSR switch point is varied as well as how it performs when errors are injected into the simulation.

An appropriate error rate to be used for analysis was determined to be approximately one Single Event Upset (SEU) per hour by conducting radiation experiments on an Intel Cyclone V Field Programmable Gate Array (FPGA) at Sandia National Laboratories and performing post experimental analysis. This is the expected average rate for a space vehicle using the Cyclone V FPGA over the life of the mission. However, the SEU rate for real missions fluctuates as a result of orbital position with reference to earth’s magnetic field lines (i.e., SEU rate increases over the South Atlantic Anomaly) and as a result of solar weather (i.e., changes in solar activity impact the SEU rate). This FPGA was chosen as a representative inexpensive commercial-off-the-shelf technology for past research and this research. Based on the experiment and the knowledge that nearly every program used in the previous work and the current work has a runtime of approximately 50ms or less, it is reasonable to expect no more than one error per program run.

2. Materials and Methods

The only materials required for this work were a computer with a network connection and MATLAB installed. Previous work implemented Basic, TMR, TSR, and AHR MIPS architectures in Very High Speed Integrated Circuit (VHSIC) Hardware Description Language (VHDL). Previous work also made use of Mentor Graphics Questa Sim to simulate the VHDL architectures in order to determine important timing parameters for use in MATLAB analyses to compute program runtimes on the various architectures. These timing parameters include the time to complete individual instructions, program start, program close, save/restore point creation, error recovery, and TMR to TSR transition. The results of those simulations are used in the simulations and analyses in the current work. Previous work also made use of the Intel PowerPlay Early Power Estimator for Cyclone IV and Cyclone V tool to estimate the instantaneous power used by Basic, TMR, TSR, and AHR MIPS [1].

Section 2.1 will discuss the mechanism used to inject errors into Basic MIPS. Section 2.2 will discuss when and where errors will be injected and provide the analysis tools necessary to calculate program runtime when an error is injected.

2.1. Error Injection Mechanism

In previous works, fault injection was performed using software manipulation or direct electrical injection for Hardware-in-the-Loop (HITL) simulations [16,17,23,24,27,28,35]. This research modifies the Basic MIPS Datapath to directly inject errors into a specific general purpose register at a specific point during program execution. This is software error injection because Basic MIPS is simulated in software for this research. The General Purpose Registers (GPRs) were selected as the injection points because general purpose registers account for 992 bits that are susceptible to SEUs in the Basic MIPS architecture whereas the remaining susceptible registers account for only 68 bits. These remaining registers are the program counter, instruction register, Finite State Machine (FSM) register, and additional Datapath registers. If an SEU were to occur, it has a 94% chance of occurring in a GPR as opposed to a 6% chance of occurring elsewhere. Additionally, EDDI is unable to detect and correct all errors occurring in these other registers, so it was determined that it would be better to inject errors into GPRs so that TSR and AHR operating in TSR could detect and correct the injected errors for a more accurate performance comparison between TMR, TSR, and AHR. A detailed schematic showing how the Basic MIPS Datapath was modified to incorporate GPR error injection is provided in Appendix A.

If an error occurred in the program counter of TSR or AHR operating in TSR mode, the error would cause program execution to jump backwards and repeat instructions or forwards and skip instructions. If a backwards jump were performed, the impact would be increased runtime and energy used to complete the program. Additionally, an incorrect result may be written to memory. If a forward jump were performed, instructions would be skipped. In some instances, this might be detected if one of two redundant instructions were skipped. In other instances, the illegal branch might go undetected and several results that should have been written to memory might not be written to memory. Additionally, program runtime and energy to complete the program would be reduced.

If an error occurred in the instruction register of TSR or AHR operating in TMR mode, the error would simply be corrected by Basic MIPS internal mechanism if the error resulted in an invalid instruction. If the error caused the instruction to change to another valid instruction, it might be detected if the error affected the result of a duplicated instruction. This same error might result in changing a store word instruction to something else and a result would fail to be written to memory resulting in an undetected error. A store word instruction error might also cause a result to be written to a wrong location in memory and this error would also go undetected. An instruction register error might also create a branch instruction where none existed or change the address of a branch instruction.

A FSM register error could cause TSR or AHR operating in TSR mode to incorrectly jump to another state which would most likely cause the processor to remain trapped in the incorrect state. Processing would cease and the program would never complete. At present TSR and AHR have no protection against such an error.

The other Datapath registers are used in determining whether to execute a program jump or not when evaluating a branch instruction. An error in one of these registers would cause the incorrect branch path to be taken. This could result in a longer program execution time with greater energy usage or a shorter program execution time with reduced energy usage.

2.2. Error Injection Timing

Errors are injected into GPRs that are going to be stored to memory so that the TMR Voter and the TSR comparison instruction will detect them and initiate error recovery operations. This is done immediately prior to the TMR instruction to store a GPR to memory and immediately prior to a TSR comparison instruction before storing a GPR to memory. Injecting an error for every store word instruction for all 1000 programs used in the current work is not feasible. Instead, errors are selectively injected to probe the minimum and maximum time and energy performance of the TMR, TSR, and AHR architectures. In all three architectures, the best-case errors will minimize the amount of time and energy expended on error detection and recovery and the worst-case errors will maximize the amount of time and energy expended on error detection and recovery.

Errors intended to maximize the amount of time and energy needed to recover from an error are injected immediately before Save/Restore Point creation. A TMR processor or AHR processor in TMR mode has to perform error recovery, then repeat 10,000 instructions to return to the point at which the error initially occurred. A TSR processor or AHR processor in TSR mode has to perform error recovery, then repeat 250 main program loops to return to the point at which the error initially occurred. Errors intended to minimize the amount of time and energy needed to recover from an error are injected immediately after Save/Restore Point creation. For all processors, this minimizes the number of instructions that must be repeated after error recovery to return to the point at which the error initially occurred. For a more detailed discussion concerning when errors are injected and the calculations performed to determine the program runtime for each type of error, please refer to Appendix B.

The 1000 programs used in this work were randomly generated and exercise the full range of Basic MIPS instructions and GPRs. Individual programs may not exercise the full range of Basic MIPS instructions or utilize all GPRs; however, some programs utilize most, if not all of the Basic MIPS instructions and some programs utilize all the GPRs.

2.3. Energy Used When Errors Are Injected

The energy computations are much simpler than the timing computations. The energy to complete a TMR or TSR MIPS program is the time to complete the TMR or TSR MIPS programs multiplied by the TMR or TSR MIPS instantaneous power respectively. The time to complete AHR programs is the time spent in TMR mode multiplied by the TMR MIPS instantaneous power plus the time spent in TSR mode multiplied by the TSR MIPS instantaneous power. For a more detailed discussion concerning the calculations performed to determine the energy used to complete a program for each type of error, please refer to Appendix C.

3. Results

This section examines the results of software simulations and computational analysis when errors are injected as described in Section 2.2 and Section 2.3.

Figure 6 shows the average time and energy to complete 1000 programs for each error type in each architecture (including no error injection) This figure illustrates how AHR MIPS bridges the gap between TMR MIPS and TSR MIPS performance. The AHR MIPS TMR Type A and Type B-Best errors appear to fall on a line between the TSR MIPS Best-case error and TMR MIPS Type A and Type B-Best errors. A similar pattern appears for AHR MIPS TMR Type B-Worst and TSR Worst-case errors which appear to nearly fall on a line between the TMR Type B-Worst and TSR Worst-case errors. However, this figure does not tell the entire story as the best-case, worst-case, early, and later errors define maximum and minimum bounds for AHR MIPS performance in the presence of errors.

Figure 7 shows the performance bounds as bounding boxes when the TMR to TSR transition point occurs at 15,000 instructions. Note that the points plotted in this figure are the same as those plotted in Figure 6 and represent the average program completion times; however, the TMR Type B-Best, TSR Best-Case, AHR TMR Type A Early, AHR TMR Type B-Best Early, and AHR TSR Best-Case errors have been omitted from this plot. The TMR Type B-Best case error result was nearly identical to the TMR MIPS with no error result. The TSR MIPS Best-Case error result was nearly identical to the TSR MIPS with no error result. The AHR TMR Type A Early, AHR TMR Type B-Best Early, and AHR TSR Best-Case error results were nearly identical to the AHR MIPS with no error result. The bounding boxes indicate that the average program completion time should fall somewhere within the bounding box when errors are present.

The corners of the bounding box encompassing the TMR MIPS Type B-Best and Type B-Worst errors is shown as a dotted blue line. This box indicates that average program completion time and energy usage for a TMR MIPS program encountering a Type B error will end up within this box and it nearly overlaps the second box indicated by a solid blue line. The second box is used to outline the average performance of TMR MIPS when errors may or may not be present. It includes the no error, TMR Type A, and TMR Type B errors.

The corners of the bounding box encompassing the TSR MIPS Best- and Worst-case errors is shown as a red dashed line with the red square and the red diamond at opposing corners to indicate the average Best- and Worst-case error program runtime and energy usage. A second box with a red solid line is used to outline the average performance of TSR MIPS when errors may or may not be present. It includes the no error, TSR Best-case, and TSR Worst-case errors. Once again, these boxes almost overlap one another.

The corners of the bounding box encompassing the AHR MIPS TMR Type A Early and Late errors is shown as a gray dashed line to indicate how a program will perform in terms of time and energy usage on average when a TMR Type A error will occur. Similarly, the purple dashed line indicates the average performance of a program experiencing a TMR Type B-Best case error. The orange dashed line indicates the average performance of a program experiencing a TMR Type B-Worst error, however, no such box is visible in this figure as the TMR Type B-Worst Early and Late errors are identical in this figure. A dark blue dashed line bounding box extends from the left most plus sign (+) to the right most “X” and from the AHR MIPS no error (green circle) to the top most “X” to show the average bounds of a AHR MIPS program that encounters any TMR Type B error. The dashed teal line indicates the bounds of a AHR MIPS TSR error. Finally, the solid green line indicates the average bounds for a AHR MIPS program experiencing any TMR error, TSR error, or no error.

Note that the portion of the bounding box extending to the left of the average AHR MIPS no error runtime and energy usage does not necessarily indicate that an error could occur such that the runtime would decrease without a change in energy usage. It should be expected that a decrease in runtime would correspond to a greater number of instructions being performed in TMR mode and a resulting energy increase; however, there is insufficient analysis at this time to determine a more precise boundary region and the creation of such a region is left for future work.

Now, because the bounding boxes indicate that the average time and energy used to complete a program in the presence of errors should fall within the boxes, they should not be treated as program specific bounding boxes. It would be trivial to create program specific bounding boxes, but these are not shown here for brevity. However, the next figures will begin to show the versatility of the AHR MIPS approach as the TMR to TSR transition point is varied.

Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14 and Figure 15 show what happens to the bounding boxes as the TMR to TSR transition point increases from 11,000, to 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, and 80,000 instructions. These figures are shown to illustrate the flexibility of AHR MIPS as the TMR to TSR transition point is varied. A space vehicle designer, mission planner, or operator can change the TMR to TSR transition point depending on the program being run on the processor, radiation environment, processing speed requirements, power requirements, and other mission needs. For example, it might be desirable to operate in TMR in a low radiation environment for the purpose of maximizing processing speed when energy usage is not a concern. This is achieved by setting the TMR to TSR transition point to such a large value that the TMR to TSR transition never occurs. Alternatively, it might be desirable to operate in a low power mode regardless of the external radiation environment and its potential impact on registers other than GPRs. This is achieved by setting the TMR to TSR transition point to zero so that AHR switches from TMR back to TSR as quickly as possible after AHR suffers multiple errors in TSR mode.

As the transition point increases, The overall bounding box begins to grow, then shrinks down to match the TMR MIPS bounding box. Note that in some of these figures, the AHR MIPS TMR Type B-Worst Early and Late errors do not always coincide. Note also how the size and shapes of the smaller bounding boxes change. As the TMR to TSR transition point increases, the AHR MIPS TMR Type B-Best bounding box increases in size, then decreases in size until it becomes nonexistent. The AHR MIPS TMR Type B-Worst bounding box increases in size from nonexistence, then decreases in size until becoming nonexistent again. The AHR MIPS TMR Type A box also increases then decreases in size until becoming nearly nonexistent. The AHR MIPS TSR box decreases in size until becoming nearly nonexistent.

While these figures represent the average performance for 1000 programs, they have greater utility when created for a specific program to show how the expected program runtime and energy usage change as the TMR to TSR transition point is changed. A satellite designer, mission planner, or operator could use these to determine the best transition point based on the needs of the system. For example, the TMR to TSR transition point could be selected in order to meet certain performance criteria such as staying under maximum runtime or energy constraints.

Figure 16 shows the same things as Figure 6, but allows the TMR to TSR transition point to vary from 11,000 to 80,000 instructions in increments of 1000. This figure also provides a slightly different view to the bounding boxes in the previous figures. It is most useful in visualizing how the average program runtime and energy usage for each error scenario changes as the TMR to TSR transition point changes. Curves for all AHR MIPS error scenarios, and the no error scenario, become evident. When there are only 11,000 instructions completed in TMR before the TMR to TSR transition point, AHR MIPS behaves much more closely to TSR MIPS. As the transition point moves towards 80,000 instructions, the AHR MIPS results begin moving up and to the left until they coincide with the TMR MIPS results. Note that the AHR MIPS TSR Best- and Worst-case scenarios collapse to the no error solution for TMR MIPS when very little, if any time is spent in TSR MIPS because the TMR to TSR transition point is no longer reached during the duration of most programs. Similarly, the AHR MIPS TMR Type B-Worst scenarios converge to the TMR Type B-Worst error scenario.

Another interesting comparison is to look at the average percent difference in runtime and energy usage for each error scenario and no error scenario when compared to Basic MIPS with no errors. The average percent difference for the no error scenarios were given in the previous work [1]. The average percent difference for the programs experiencing errors are given in Equations (1)–(26).

$$\begin{array}{c}P{D}_{TimeTMRErrAvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{TMRErrA}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(1)

$$\begin{array}{c}P{D}_{TimeTMRErrBBestvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{TMRErrBBest}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(2)

$$\begin{array}{c}P{D}_{TimeTMRErrBWorstvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{TMRErrBWorst}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(3)

$$\begin{array}{c}P{D}_{TimeTSRBestvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{TSRBest}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(4)

$$\begin{array}{c}P{D}_{TimeTSRWorstvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{TSRWorst}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(5)

$$\begin{array}{c}P{D}_{TimeCTMRAEarlyvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRAEarly}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(6)

$$\begin{array}{c}P{D}_{TimeCTMRALatevBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRALate}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(7)

$$\begin{array}{c}P{D}_{TimeCTMRBBestEarlyvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRBBestEarly}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(8)

$$\begin{array}{c}P{D}_{TimeCTMRBBestLatevBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRBBestLate}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(9)

$$\begin{array}{c}P{D}_{TimeCTMRBWorstEarlyvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRBWorstEarly}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(10)

$$\begin{array}{c}P{D}_{TimeCTMRBWorstLatevBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRBWorstLate}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(11)

$$\begin{array}{c}P{D}_{TimeCTSRBestvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTSRBest}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(12)

$$\begin{array}{c}P{D}_{TimeCTSRWorstvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{T_{CTMRWorst}\left(n\right)-T_{BasicMIPS}\left(n\right)}{T_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(13)

$$\begin{array}{c}P{D}_{EnergyTMRErrAvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{TMRErrA}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(14)

$$\begin{array}{c}P{D}_{EnergyTMRErrBBestvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{TMRErrBBest}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(15)

$$\begin{array}{c}P{D}_{EnergyTMRErrBWorstvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{TMRErrBWorst}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(16)

$$\begin{array}{c}P{D}_{EnergyTSRBestvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{TSRBest}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(17)

$$\begin{array}{c}P{D}_{EnergyTSRWorstvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{TSRWorst}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(18)

$$\begin{array}{c}P{D}_{EnergyCTMRAEarlyvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTMRAEarly}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(19)

$$\begin{array}{c}P{D}_{EnergyCTMRALatevBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTMRALate}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(20)

$$\begin{array}{c}P{D}_{EnergyCTMRBBestEarlyvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTMRBBestEarly}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(21)

$$\begin{array}{c}P{D}_{EnergyCTMRBBestLatevBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTMRBBestLate}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(22)

$$\begin{array}{c}P{D}_{EnergyCTMRBWorstEarlyvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTMRBWorstEarly}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(23)

$$\begin{array}{c}P{D}_{EnergyCTMRBWorstLatevBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTMRBWorstLate}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(24)

$$\begin{array}{c}P{D}_{EnergyCTSRBestvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTSRBest}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(25)

$$\begin{array}{c}P{D}_{EnergyCTSRWorstvBasic}=···\hfill \\ {\displaystyle \frac{{\sum }_{n=1}^{N_{programs}}\left[{\displaystyle \frac{E_{CTSRWorst}\left(n\right)-E_{BasicMIPS}\left(n\right)}{E_{BasicMIPS}\left(n\right)}}\times 100\%\right]}{N_{programs}}}\hfill \end{array}$$

(26)

The average percent difference equations were used to calculate the average percent difference for all programs experiencing errors and no errors when the TMR to TSR transition point from 11,000 to 80,000 in increments of 1000. The results for the runtime calculations are shown in Figure 17 and the results for energy usage calculations are shown in Figure 18. These figures really highlight how AHR MIPS runtime and energy performance changes when compared to Basic MIPS as the TMR to TSR transition point changes. As in some of the previous figures, the TMR Type A and TMR Type B-Best error results are omitted because they are nearly identical to the TMR no error results. The same is true for the TSR Best error results because they are identical to the TSR no error results. Additionally, the AHR TMR Type A Early, AHR TMR Type B-Best Early, and AHR TSR Best error results have been omitted because they are nearly identical to the AHR no error results. The first thing to note from these figures is how the AHR MIPS no error, AHR TMR Type A, AHR TMR Type B-Best, AHR TSR Best, and AHR TSR Worst average percent differences approach the TMR average percent difference as the number of instructions before the TMR to TSR transition increases. This is consistent with prior results because AHR MIPS performance is nearly identical to TMR MIPS performance as the number of instructions that AHR MIPS processes in TSR mode approaches zero and nearly all instructions are processed in TMR mode. Additionally, the AHR MIPS TMR Type B-Worst average percent differences approach the TMR Type B-Worst average percent difference, which is also expected for the same reasons just given.

There are a few other things to note from the percent difference figures. The first is that programs experiencing an AHR TMR Type A Late error complete faster than programs experiencing an AHR TMR Type B-Best Late error. Both of these complete faster than AHR MIPS programs experiencing no error, AHR TMR Type A Early, AHR TMR Type B-Best Early, and AHR TSR Best-case errors. The no error, AHR TMR Type A Early, AHR TMR Type B-Best Early, and AHR TSR Best-case error scenarios all take less time to complete than programs experiencing AHR TMR Type B-Worst Early, AHR TMR Type B Worst Late, and AHR TSR Worst-case errors. Programs experiencing AHR TMR Type B-Worst Late errors always complete faster than those experiencing AHR TMR Type B-Worst Early errors. AHR MIPS programs experiencing TSR Worst-case errors have the worst runtime when the TMR to TSR transition point is under about 30,000, but runs faster than programs experiencing TMR Type B-Worst Early errors when the transition point is greater than 31,000 instructions and faster than programs experiencing TMR Type B-Worst Late errors when the transition point is greater than about 37,000 instructions.

AHR MIPS programs experiencing no error, TMR Type A Early, TMR Type B-Best Early, and TSR Best-case all take about the same amount of energy to complete and use less energy than an AHR MIPS program experiencing any other type of error. AHR MIPS programs experiencing TMR Type B-Worst Late errors use the most energy followed by programs experiencing TMR Type B-Worst Early errors, then TMR Type A Late errors, then TSR Worst-case errors.

One final thing to note are the jump discontinuities in the AHR MIPS TMR Type B-Best Late and AHR MIPS TMR Type B-Worst Late timing and energy percent differences. These are a direct result of the TMR to TSR transition point moving passed one of the TMR save/restore point creation times which occur every 10,000 instructions. Note that these discontinuities occur as the TMR to TSR transition point passes 20,000, 30,000, and 40,000 instructions. This is because these late errors go from having a minimal impact when the TMR to TSR transition point occurs immediately before a save/restore point creation to a maximum impact when the TMR to TSR transition point occurs immediately after a save/restore point creation.

Figure 19 and Figure 20 are essentially derivative plots of Figure 17 and Figure 18 except that they use the average time and energy results rather than the percent differences. Each point on these graphs represent the difference in average time and average energy to complete 1000 different programs with the given error type (or no error at all) from one AHR transition point value to the previous AHR transition point value where these transition points started at 11,000 instructions, ended at 80,000 instructions, and had step sizes of 1000 instructions. Plots like these may help a mission planner determine the most optimal point, in terms of processing speed and energy usage, at which to transition AHR from TMR mode to TSR mode.

4. Discussion

AHR uses less energy than TMR and takes less time than TSR to complete programs when errors are injected. Additionally, changing the TMR to TSR transition point allows space vehicle designers, mission planners, and operators the flexibility to select operating points that meet mission processing speed and energy usage requirements not only under optimal error free conditions, but also in the worst-case error scenarios. This was demonstrated through simulation results shown in Section 3 where the time to complete programs varied between the TMR time to complete a program and the TSR time to complete a program as the TMR to TSR transition point was varied. This was also shown as the energy used to complete programs varied between the TMR energy used to complete a program and the TSR energy used to complete a program. The figures illustrated how a space vehicle designer, mission planner, or operator could choose a TMR to TSR transition point that meets the specific needs of their mission. As previously noted, if a mission needed to maximize processing speed at the expense of increased energy usage regardless of the external radiation environment, the TMR to TSR transition point could be set to such an arbitrarily large value that AHR always remains in TMR mode. In contrast, if a mission needs to minimize energy usage at the expense of slower processing speeds regardless of the radiation environment, the TMR to TSR transition point could be set to zero to ensure that AHR remains in TSR mode. For mission needs in between these two extremes, the TMR to TSR transition point could be set to a value that meets certain timing and energy performance criteria while accounting for the radiation environment and its impact on processing speed and energy usage. Additionally, the transition point can be program specific for a processor entrusted with running many different programs so that the transition point is optimized for each program. Furthermore, the transition point can be varied at any time over the course of the mission. It could even be changed during a single orbit to ensure an optimal value at all times when radiation levels and mission needs are taken into account.

Future work will implement TMR, TSR, and AHR on a Cyclone V FPGA to determine how they perform under error free and error injection conditions in terms of time and energy performance. This will be done in an effort to verify that this method works in application and not just in the realm of simulation and analysis.

Another area for future work is expanding AHR to include more redundancy methods such as dual modular redundancy, N-modular redundancy, and advanced TSR methods that can detect and correct program counter errors, which EDDI is unable to detect.

The views expressed in this paper are those of the authors, and do not reflect the official policy or position of the United States Air Force, Department of Defense, or the U.S. Government. This document has been approved for public release; distribution unlimited, Case #88ABW-2019-4400.