A Framework for Characterization of Optimal Decision Rules in Hypothesis-Testing Problems

In this review paper , we present a framework for the characterization of optimal decision rules in M-ary hypothesis-testing problems where the performance metric is defined as a function of pairwise error probabilities. This framework is based on the approaches developed in several recent studies in the literature, which are unified and presented in a tutorial fashion in this paper. A pairwise error probability represents the probability of selecting a specific hypothesis when a different hypothesis is true, and can be stacked into a pairwise probability vector for a given problem. In the considered framework, instead of optimizing the performance metric of interest over the infinite-dimensional set of all possible decision rules, the optimization is performed directly over the compact and convex set of all achievable pairwise probability vectors. We demonstrate that any pairwise probability vector within this feasible set can be realized via a randomization of at most two likelihood ratio quantizers (LRQs) with different sets of parameters. While one of these LRQs can always be selected as a deterministic LRQ, the other one is possibly a randomized LRQ, which can be written as a randomization of at most $M(M-1)$ deterministic LRQs, with M denoting the number of hypotheses. The main advantage of this framework is that it allows for the attainment of pairwise probability vectors that do not reside on the boundary of the feasible set and that are fundamentally inaccessible via LRQs, which are optimal for classical performance metrics such as the Bayes risk or the Neyman–Pearson criterion. Furthermore, we show that the characterization of decision rules with the presented framework is particularly advantageous for performance metrics based on prospect theory (PT), such as behavioral utility. Specifically, it is demonstrated that the optimal pairwise probability vector for a PT-based metric is not guaranteed to lie on the boundary of the feasible set of pairwise probability vectors. This results in suboptimal performance achieved by LRQs for such performance metrics. On the other hand, the randomized decision rules characterized in this paper can achieve pairwise probability vectors located in the interior of the feasible set, thereby yielding optimal performance. Numerical results corroborate these findings, demonstrating that the decision rules characterized within our framework yield optimal behavioral utility-based performance scores.