Polynomial-Time Constrained Message Passing for Exact MAP Inference on Discrete Models with Global Dependencies

Abstract

Considering the worst-case scenario, the junction-tree algorithm remains the most general solution for exact MAP inference with polynomial run-time guarantees. Unfortunately, its main tractability assumption requires the treewidth of a corresponding MRF to be bounded, strongly limiting the range of admissible applications. In fact, many practical problems in the area of structured prediction require modeling global dependencies by either directly introducing global factors or enforcing global constraints on the prediction variables. However, this always results in a fully-connected graph, making exact inferences by means of this algorithm intractable. Previous works focusing on the problem of loss-augmented inference have demonstrated how efficient inference can be performed on models with specific global factors representing non-decomposable loss functions within the training regime of SSVMs. Making the observation that the same fundamental idea can be applied to solve a broader class of computational problems, in this paper, we adjust the framework for an efficient exact inference to allow much finer interactions between the energy of the core model and the sufficient statistics of the global terms. As a result, we greatly increase the range of admissible applications and strongly improve upon the theoretical guarantees of computational efficiency. We illustrate the applicability of our method in several use cases, including one that is not covered by the previous problem formulation. Furthermore, we propose a new graph transformation technique via node cloning, which ensures a polynomial run-time for solving our target problem. In particular, the overall computational complexity of our constrained message-passing algorithm depends only on form-independent quantities such as the treewidth of a corresponding graph (without global connections) and image size of the sufficient statistics of the global terms.

1. Introduction

Many practical tasks can be effectively formulated as discrete optimization problems within the framework of graphical models such as Markov Random Fields (MRFs) [1,2,3] by representing the constraints and objective function in a factorized form. Finding the corresponding solution refers to the task of maximum a posteriori (MAP) inference, which is known to be NP-hard in general. Although there are plenty of existing approximation algorithms [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23], several problems (described below) require finding an optimal solution. Existing exact algorithms [24,25,26,27,28,29,30,31,32,33,34], on the other hand, either make specific assumptions about the energy function or do not provide polynomial run-time guarantees for the worst case. Assuming the worst-case scenario, the junction (or clique)-tree algorithm [1,35], therefore, remains the most efficient and general solution for exact MAP inference. Unfortunately, its main tractability assumption requires the treewidth [36,37] of a corresponding MRF to be bounded, strongly limiting the range of admissible applications by excluding models with global interactions. Many problems in the area of structured prediction, however, require modeling of global dependencies by either directly introducing global factors or enforcing global constraints on the prediction variables. Among the most popular use cases are (a) learning using non-decomposable (or high-order) loss functions and training via slack scaling formulation within the framework of a structural support vector machine (SSVM) [38,39,40,41,42,43,44,45,46,47], (b) evaluating generalization bounds in structured prediction [14,15,44,48,49,50,51,52,53,54,55,56,57,58,59], and (c) performing MAP inference on otherwise tractable models subject to global constraints [19,60,61]. The latter covers various search problems, including the special task of (diverse) k-best MAP inference [62,63]. Learning with non-decomposable loss functions, in particular, benefits from finding an optimal solution, as all of the theoretical guarantees of training with SSVMs assume exact inference during optimization [39,64,65,66,67,68].

Previous works [45,46,47,69] focusing on the problem of loss-augmented inference (use case (a)) have demonstrated how efficient computation can be performed on models with specific global factors by leveraging a dynamic programming approach based on constrained message passing. The proposed idea models non-decomposable functions as a kind of multivariate cardinality potential $\mathit{y}\mapsto \eta \left(\mathit{G}\right(\mathit{y}\left)\right)$, where $\eta :{\mathbb{R}}^P\to {\mathbb{R}}_{+}$ is some function and $\mathit{G}$ denotes the sufficient statistics of the global term. Although able to model popular performance measures, the objective of a corresponding inference problem is rather restricted to simple interactions between the energy of the core model F and the sufficient statistics $\mathit{G}$ according to $F\odot \eta \left(\mathit{G}\right)$, where $\odot :\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ is either a summation or a multiplication operation. Although the same framework can be applied to use case (c) by modeling global constraints via an indicator function, it cannot handle a range of other problems in use case (b) that introduce more subtle dependencies between F and $\mathit{G}$.

In this paper, we extend the framework for an efficient exact inference proposed in [69] by allowing much finer interactions between the energy of the core model and the sufficient statistics of the global terms. The extended framework covers all the previous cases and applies to new problems, including the evaluation of generalization bounds in structured learning, which cannot be handled by the previous approach. At the same time, the generalization comes with no additional costs, preserving all the run-time guarantees. In fact, the resulting performance is identical to that of the previous formulation, as the corresponding modifications do not change the computational core idea of the previously proposed message passing constrained via auxiliary variables but only affect the final evaluation step (line 8 in Algorithm 1) of the resulting inference algorithm after all the required statistics have been computed. We accordingly adjust the formal statements given in [69] to ensure the correctness of the algorithmic procedure for the extended case.

Furthermore, we propose an additional graph transformation technique via node cloning that greatly improves the theoretical guarantees on the asymptotic upper bound for the computational complexity. In particular, the above-mentioned work only guarantees polynomial run-time in the case where the core model can be represented by a tree-shaped factor graph, excluding problems with cyclic dependencies. The corresponding complexity estimation for clique trees (see Theorem 2 in [69]), however, requires the maximal node degree $\nu$ to be bounded by a graph-independent constant; otherwise, it results in a term that depends exponentially on the graph size. Here, we first provide an intuition that $\nu$ tends to take on small values (see Proposition 2) and then present an additional graph transformation that reduces this parameter to a constant $\nu =3$ (see Corollary 1). Furthermore, we analyze how the maximal number of states of auxiliary variables R, which greatly affects the resulting run-time, grows relative to the graph size (see Theorem 2).

The rest of the paper is organized as follows. In Section 2, we formally introduce the class of problems we tackle in this paper, and in Section 3, we present the constrained message-passing algorithm for finding a corresponding optimal solution based on a general representation with clique trees. Additionally, we propose a graph transformation technique via node cloning, which ensures an overall polynomial running time for the computational procedure. In Section 4, we demonstrate the expressivity of our problem formulation on several examples. For an important use case of loss-augmented inference with SSVMs, we show in Section 5 how to represent different dissimilarity measures as global cardinality potentials to align with our problem formulation. In order to validate the guarantees for the computational complexity of Algorithm 1, we present the experimental run-time measurements in Section 6. In Section 7, we provide a summary of our contributions, emphasizing the differences between our results and those in [69]. In Section 8, we broadly discuss the previous works, which is followed by the conclusions in Section 9.

2. Problem Setting

Given an MRF [1,70] over a set of discrete variables, the goal of the maximum a posteriori (MAP) problem is to find a joint variable assignment with the highest probability. This problem is equivalent to minimizing the energy of the model, which describes the corresponding (unnormalized) probability distribution over the variables. In the context of structured prediction, it is equivalent to maximizing a score or compatibility function. To avoid ambiguity, we now refer to the MAP problem as the maximization of an objective function $F:{\mathbb{R}}^M\to \mathbb{R}$ defined over a set of discrete variables $\mathit{y}=(y_1,\cdots ,y_M)$. More precisely, we associate each function F with an MRF, where each variable $y_m$ represents a node in the corresponding graph. Furthermore, we assume, without loss of generality, that the function F factorizes over maximal cliques ${\mathit{y}}_{C_t}$, $C_t\subseteq \{1,\cdots ,M\}$ of the corresponding MRF according to

$$F\left(\mathit{y}\right)={\sum }_{t=1}^T{f}_t\left({\mathit{y}}_{C_t}\right).$$

(1)

We now use the concept of the treewidth of a graph [36] to define the complexity of a corresponding function with respect to the MAP inference, as follows.

Definition 1

($\tau$-decomposability). We say that a function $F:\mathcal{D}\subseteq {\mathbb{R}}^M\to \mathbb{R}$ is τ-decomposable if the (unnormalized) probability $exp\left(F\right(\mathit{y}\left)\right)$ factorizes over an MRF with a bounded treewidth τ.

Informally, the treewidth describes the tree-likeness of a graph, that is, how well the graph structure resembles the form of a tree. In an MRF with no cycles going over the individual cliques, the treewidth is equal to the maximal size of a clique minus 1, that is, $\tau ={max}_t\left|C_t\right|-1$. Furthermore, the treewidth of a graph is considered bounded if it does not depend on the size of the graph in the following sense. If it is possible to increase the graph size by replicating the individual parts, the treewidth should not be affected by the number of variables in the resulting graph. One simple example is a Markov chain. Increasing the length of the chain does not affect the treewidth, which remains equal to the Markov order of that chain.

The treewidth is defined as the minimum width of a graph and can be computed algorithmically after transforming the corresponding MRF into a data structure called a junction tree or clique tree. Although the problem of constructing a clique tree with a minimum width is NP-hard in general, there are several efficient techniques [1] that can achieve good results with a width close to the treewidth.

In the following, let M be the total number of nodes in an MRF over the variables ${\left\{y_m\right\}}_{m=1}^M$, and let N be the maximum number of possible values each variable $y_m$ can take on. Assuming that the maximization step dominates the time for creating a clique tree, we obtain the following known result [71]:

Proposition 1.

The computational time complexity for maximizing a τ-decomposable function is upper bounded by $O(M·N^{\tau +1})$.

The notion of $\tau$-decomposability for real-valued functions naturally extends to mappings with multivariate outputs for which we now define joint decomposability.

Definition 2

(Joint $\tau$-decomposability). We say two mappings $\mathit{G}:\mathcal{D}\subseteq {\mathbb{R}}^M\to {\mathbb{R}}^P$ and ${\mathit{G}}^{\prime }:\mathcal{D}\subseteq {\mathbb{R}}^M\to {\mathbb{R}}^{P^{\prime }}$ are jointly $\tau$-decomposable if they factorize over a common MRF with a bounded treewidth τ.

Definition 2 ensures the existence of a common clique tree with nodes ${\left\{C_t\right\}}_{t=1}^T$ and the corresponding potentials ${\{{\mathit{g}}_t,{\mathit{g}}_t^{\prime }\}}_{t=1}^T$, where ${max}_t\left|C_t\right|-1=\tau$, and 

$$\begin{array}{c}\hfill \mathit{G}\left(\mathit{y}\right)={\sum }_{t=1}^T{\mathit{g}}_t\left({\mathit{y}}_{C_t}\right),{\mathit{G}}^{\prime }\left(\mathit{y}\right)={\sum }_{t=1}^T{\mathit{g}}_t^{\prime }\left({\mathit{y}}_{C_t}\right).\end{array}$$

Note that the individual factor functions are allowed to have fewer variables in their scope than in a corresponding clique, that is, $\mathrm{scope}\left({\mathit{g}}_t\right),\mathrm{scope}\left({\mathit{g}}_t^{\prime }\right)\subseteq C_t$.

Building on the above definitions we now formally introduce a class of problem instances of MAP inference for which we later provide an exact message-passing algorithm.

Problem 1.

For $F:\mathcal{Y}\to \mathbb{R}$, $\mathit{G}:\mathcal{Y}\to {\mathbb{R}}^P$, $H:\mathbb{R}\times {\mathbb{R}}^P\to \mathbb{R}$ with $\mathcal{Y}\subset {\mathbb{R}}^M$, $\left|\mathcal{Y}\right|⩽N^M$ and $P,M,N\in {\mathbb{N}}_{+}$, we consider the following discrete optimization problem:

$$\begin{array}{cccc}\hfill & \underset{\mathit{y}\in \mathcal{Y}}{\mathrm{maximize}}\hfill & \hfill & H\left(F\right(\mathit{y}),\mathit{G}(\mathit{y}\left)\right)\hfill \end{array}$$

(2)

where we assume that (1) F and $\mathit{G}$ are jointly τ-decomposable, and (2) H is non-decreasing in the first argument.

In the next section, we present multiple examples of practical problems that align with the above-mentioned abstract formulation. As our working example, here, we consider the problem of loss-augmented inference within the framework of SSVMs. This framework includes two different formulations known as margin and slack scaling, both of which require solving a combinatorial optimization problem during training. This optimization problem is either used to compute the subgradient of a corresponding objective function or find the configuration of prediction variables that violates the problem constraints the most. For example, in the case of slack scaling formulation, we can define $H\left(F\right(\mathit{y}),\mathit{G}(\mathit{y}\left)\right)=F\left(\mathit{y}\right)·\eta \left(\mathit{G}\right(\mathit{y}\left)\right)$ for some $\eta :{\mathbb{R}}^P\to {\mathbb{R}}_{+}$, where $F\left(\mathit{y}\right)={\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})$ corresponds to the compatibility score given by the inner product between a joint feature map $\Psi (\mathit{x},\mathit{y})$ and a vector of trainable weights $\mathit{w}$ (see [39] for more details), and $\eta \left(\mathit{G}\left(\mathit{y}\right)\right)=\Delta ({\mathit{y}}^{*},\mathit{y})$ describes the corresponding loss function for a prediction $\mathit{y}$ and a ground-truth output ${\mathit{y}}^{*}$. In fact, a considerable number of popular loss functions used in structured prediction can generally be represented in this form, that is, as a multivariate cardinality-based potential that depends on counts of different label statistics.

3. Exact Inference for Problem 1

In this section, we derive a polynomial-time message-passing algorithm that always finds an optimal solution for Problem 1. The corresponding results can be seen as a direct extension of the well-known junction-tree algorithm.

3.1. Algorithmic Core Idea for a Simple Chain Graph

We begin by providing an intuition of why efficient inference is possible for Problem 1 using our working example of loss-augmented inference for SSVMs. For margin scaling, in the case of a linear $\eta$, the objective $F\left(\mathit{y}\right)+\eta \left(\mathit{G}\right(\mathit{y}\left)\right)$ inherits the $\tau$-decomposability directly from F and $\mathit{G}$ and, therefore, can be efficiently maximized according to Proposition 1.

The main source of difficulty for slack scaling lies in the multiplication operation between $F\left(\mathit{y}\right)$ and $\eta \left(\mathit{G}\right(\mathit{y}\left)\right)$, which results in a fully-connected MRF regardless of the form of the function $\eta$. Moreover, many popular loss functions used in structured learning require $\eta$ to be non-linear, preventing efficient inference even for the margin scaling. Nevertheless, efficient inference is possible for a considerable number of practical cases, as shown below. Specifically, the global interactions between a jointly decomposable F and $\mathit{G}$ can be controlled by using auxiliary variables at a polynomial cost. We now illustrate this with a simple example.

Figure 1. Factor graph representation for the margin-scaling objective, with a decomposable loss $\mathit{G}$ (on the left), and a (non-decomposable) high-order loss $\eta \left(\mathit{G}\right)$ (on the right).

Figure 1. Factor graph representation for the margin-scaling objective, with a decomposable loss $\mathit{G}$ (on the left), and a (non-decomposable) high-order loss $\eta \left(\mathit{G}\right)$ (on the right).

Consider a (Markov) chain of nodes with a 1-decomposable F and 0-decomposable $\mathit{G}$ (e.g., Hamming distance), that is,

$$F\left(\mathit{y}\right)=\sum _{m=1}^{M-1}{f}_m(y_m,y_{m+1})\mathrm{and}\mathit{G}\left(\mathit{y}\right)=\sum _{m=1}^M{\mathit{g}}_m\left(y_m\right).$$

(3)

We aim at maximizing an objective $F\left(\mathit{y}\right)\odot \eta \left(\mathit{G}\right(\mathit{y}\left)\right)$, where $\odot :\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ is a placeholder for either a summation or a multiplication operation.

The case for margin scaling with a decomposable loss $\eta \left(\mathit{G}\right)=\mathit{G}$ is illustrated by the leftmost factor graph in Figure 1. Here, the corresponding factors $f_m$ and ${\mathit{g}}_m$ can be folded together, enabling an efficient inference according to Proposition 1. The non-linearity of $\eta$, however, can result (in the worst case!) in a global dependency between all the variable nodes, leading to a high-order potential $\eta \left(\mathit{G}\right)$, as illustrated by the rightmost factor graph in Figure 1. In slack scaling, even for a linear $\eta$, after multiplying the individual factors, we can see that the resulting model has an edge for every pair of variables, resulting in a fully-connected graph. Thus, for the last two cases, exact inference using the junction-tree algorithm is generally not feasible. The key idea of our approach is to relax the dense connections in these graphs by introducing auxiliary variables $\mathit{L}=({\mathit{l}}_1,\dots ,{\mathit{l}}_M)\in {\mathbb{R}}^{P\times M}$ subject to the constraints

$${\mathit{l}}_m=\sum _{k=1}^m{\mathit{g}}_k\left(y_k\right),m\in \{1,\cdots ,M\}.$$

(4)

More precisely, for $H\left(F\right(\mathit{y}),\mathit{G}(\mathit{y}\left)\right)=F\left(\mathit{y}\right)\odot \eta \left(\mathit{G}\right(\mathit{y}\left)\right)$, Problem 1 is equivalent to the following constrained optimization problem in the sense that both have the same optimal value and the same set of optimal solutions with respect to $\mathit{y}$:

$$\begin{array}{cccc}\hfill & \underset{\mathit{y},\mathit{L}}{\mathrm{maximize}}\hfill & \hfill & F\left(\mathit{y}\right)\odot \eta \left({\mathit{l}}_M\right)\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & {\mathit{l}}_{m+1}={\mathit{l}}_m+{\mathit{g}}_{m+1}\left(y_{m+1}\right)\forall m\in \{1,\cdots ,M-1\}\hfill \\ \hfill & \hfill & & {\mathit{l}}_1={\mathit{g}}_1\left(y_1\right)\hfill \end{array}$$

(5)

where the new objective involves no global dependencies and is 1-decomposable if we regard $\eta \left({\mathit{l}}_M\right)$ as a constant. We can make the local dependency structure of the new formulation more explicit by taking the constraints directly into the objective as follows:

$$Q(\mathit{y},\mathit{L})=F\left(\mathit{y}\right)\odot \eta \left({\mathit{l}}_M\right)-{\mathbb{1}}_{\infty }[{\mathit{l}}_1\ne {\mathit{g}}_1\left(y_1\right)]-\sum _{m=1}^{M-1}{\mathbb{1}}_{\infty }[{\mathit{l}}_{m+1}\ne {\mathit{l}}_m+{\mathit{g}}_{m+1}\left(y_{m+1}\right)].$$

(6)

Here, ${\mathbb{1}}_{\alpha }[·]$ denotes the indicator function such that ${\mathbb{1}}_{\alpha }[·]=\alpha$ if the argument in $[·]$ is true and ${\mathbb{1}}_{\alpha }[·]=0$ otherwise. The indicator functions rule out the configurations that do not satisfy Equation (4) when maximization is performed. A corresponding factor graph for margin scaling is illustrated by the leftmost graph in Figure 2. We can see that our new augmented objective (6) shows only local dependencies and is, in fact, 2-decomposable.

Applying the same scheme for slack scaling yields a much more sparsely connected graph (see the rightmost graph in Figure 2). This is achieved by forcing the majority of connections to go through a single node ${\mathit{l}}_4$, which we call a hub node. Actually, $Q(\mathit{y},\mathit{L})$ becomes 2-decomposable if we fix the value of ${\mathit{l}}_4$, which then can be multiplied into the corresponding factors of F. In this way, we can effectively reduce the overall treewidth at the expense of increased polynomial computation time (compared to a chain without the global factor), provided that the maximal number R of different states of each auxiliary variable ${\mathit{l}}_m$ is polynomially bounded in M, which represents the number of nodes in the original graph. In the context of training SSVMs, for example, the majority of the popular loss functions satisfy this condition (see Table 1 in Section 5 for an overview).

3.2. Constrained Message-Passing Algorithm on Clique Trees

The idea presented in the previous section is intuitive and enables the reuse of existing software. Specifically, we can use the conventional junction-tree algorithm for graphical models by extending the original graph with nodes corresponding to the auxiliary variables. Alternatively, instead of performing an explicit graph transformation, we can modify the message-passing protocol, which is asymptotically at least one order of magnitude faster. Therefore, we do not explicitly introduce auxiliary variables as graph nodes before constructing the clique tree but rather use them to condition the message-passing rules. In the following, we derive an algorithm for solving an instance of Problem 1 via constrained message passing on clique trees. The resulting computational scheme can be seen as a direct extension of the junction-tree algorithm to models with specific global factors. Note that the form of tractable global dependencies is constrained according to the definition of Problem 1.

First, similar to the conventional junction-tree algorithm, we need to construct a clique tree for a given set of factors that preserves the family structure and has the running intersection property. Note that we ignore the global term during this process. The corresponding energy is given by the function F according to the definition of Problem 1. There are two equivalent approaches for constructing the click tree [1,70]. The first is based on variable elimination and the second is based on graph triangulation, with an upper bound of $O(M·N^{\tau +1})$. Figure 3 illustrates the intermediate steps of the corresponding construction process for a given factor graph using the triangulation approach. The resulting clique tree example defines the starting point for our message-passing algorithm.

Assume that a clique tree with cliques $C_1,\cdots ,C_K$ is given, where $C_i$ denotes a set of indices of variables contained in the i-th clique. We denote a corresponding set of variables by ${\mathit{y}}_{C_i}$. Furthermore, we use the notations ${\left\{f_{C_i}\right\}}_{i=1}^K$ and ${\left\{{\mathit{g}}_{C_i}\right\}}_{i=1}^K$ to denote the clique potentials (or factors) related to the mappings F and $\mathit{G}$ in the definition of Problem 1, respectively. Additionally, we denote by $C_r$ a clique chosen to be the root of the clique tree. Finally, we use the notation $ne\left(C_i\right)$ for the indices of the neighbors of the clique $C_i$. We can now compute the optimal value of the objective in Problem 1 as follows. Starting at the leaves of the clique tree, we iteratively send messages toward the root according to the following message-passing protocol. A clique $C_i$ can send a message to its parent clique $C_j$ if it received all messages from the rest of its neighbors $C_k$ for $k\in ne\left(C_i\right)\setminus \left\{j\right\}$. In that case, we say that $C_i$ is ready.

For each configuration of the variables ${\mathit{y}}_{C_i\cap C_j}$ and parameters ${\mathit{l}}_i\in {\mathbb{R}}^P$ (encoding the state of an auxiliary variable associated with the current clique $C_i$), a corresponding message from a clique $C_i$ to a clique $C_j$ can be computed according to the following equation:

$${\mu }_{C_i\to C_j}^{{\mathit{l}}_i}\left({\mathit{y}}_{C_i\cap C_j}\right)=\underset{{\mathit{y}}_{C_i\setminus C_j},\left\{{\mathit{l}}_k\right\}}{max}{f}_{C_i}\left({\mathit{y}}_{C_i}\right)+\sum _{k\in ne\left(C_i\right)\setminus \left\{j\right\}}{\mu }_{C_k\to C_i}^{{\mathit{l}}_k}\left({\mathit{y}}_{C_k\cap C_i}\right)$$

(7)

where we maximize over all configurations of the variables ${\mathit{y}}_{C_i\setminus C_j}$ and over all parameters $\left\{{\mathit{l}}_k\right\}=\{{\mathit{l}}_k:k\in ne\left(C_i\right)\setminus \left\{j\right\}\}$ subject to the following constraint

$$\sum _{k\in ne\left(C_i\right)\setminus \left\{j\right\}}{\mathit{l}}_k={\mathit{l}}_i-{\mathit{g}}_{C_i}\left({\mathit{y}}_{C_i}\right).$$

(8)

This means that each clique $C_i$ is assigned exactly one (multivariate) auxiliary variable ${\mathit{l}}_i$ and the range of possible values ${\mathit{l}}_i$ can take on is implicitly defined by Equation (8). After resolving the recursion in the above equation, we can see that the variable ${\mathit{l}}_i$ corresponds to a sum of the potentials ${\mathit{g}}_{C_k}\left({\mathit{y}}_{C_k}\right)$ for each previously processed clique $C_k$ in a subtree of the graph of which $C_i$ forms the root. We refer to Equation (4) in the previous subsection for comparison.

Algorithm 1 Constrained Message Passing on a Clique Tree

Require:  clique tree ${\left\{C_i\right\}}_i$; Output: optimal assignment ${\mathit{y}}^{*}$ 1: while root clique $C_r$ did not receive all messages do 2:    if a clique $C_i$ is ready then 3:      for all ${\mathit{y}}_{C_i\cap C_j}$ and ${\mathit{l}}_i$ do 4:         send a message ${\mu }_{C_i\to C_j}^{{\mathit{l}}_i}\left({\mathit{y}}_{C_i\cap C_j}\right)$ to a parent clique $C_j$ according to Equation (7); save the maximizing arguments ${\lambda }_{C_i\to C_j}^{{\mathit{l}}_i}\left({\mathit{y}}_{C_i\cap C_j}\right):=[{\mathit{y}}_{C_i\setminus C_j}^{*};{\left\{{\mathit{l}}_k\right\}}^{*}]$ 5:      end for 6:    end if 7: end while 8: ${\mathit{l}}^{*}\leftarrow {\mathrm{argmax}}_{\mathit{l}}H(\mu \left(\mathit{l}\right),\mathit{l})$,  , where $\mu \left(\mathit{l}\right)$ is defined by Equation (9) 9: Let ${\mathit{y}}_{C_r}^{*}$ be a maximizing argument for $\mu \left({\mathit{l}}^{*}\right)$ in Equation (9); starting with values ${\mathit{l}}^{*}$ and ${\mathit{y}}_{C_r}^{*}$, recursively reconstruct an optimal configuration ${\mathit{y}}^{*}$ from $\lambda$ according to Equation (7).

The algorithm terminates if the designated root clique $C_r$ received all messages from its neighbors. We then compute the values

$$\mu \left(\mathit{l}\right)=\underset{{\mathit{y}}_{C_r},\left\{{\mathit{l}}_k\right\}}{max}{f}_{C_r}\left({\mathit{y}}_{C_r}\right)+\sum _{k\in ne\left(C_r\right)}{\mu }_{C_k\to C_r}^{{\mathit{l}}_k}\left({\mathit{y}}_{C_k\cap C_r}\right)$$

(9)

maximizing over all configurations of ${\mathit{y}}_{C_r}$, and $\left\{{\mathit{l}}_k\right\}=\{{\mathit{l}}_k:k\in ne\left(C_r\right)\}$ subject to the constraint ${\sum }_k{\mathit{l}}_k=\mathit{l}-{\mathit{g}}_{C_r}\left({\mathit{y}}_{C_r}\right)$, which we use to obtain the optimal value $p^{*}$ of Problem 1 according to

$$p^{*}=\underset{\mathit{l}}{max}H(\mu \left(\mathit{l}\right),\mathit{l}).$$

(10)

The corresponding optimal solution of Problem 1 can be obtained by backtracking the additional variables $\lambda$, saving optimal decisions in intermediate steps. The complete algorithm is summarized in Algorithm 1. As an alternative, we provide an additional flowchart diagram illustrating the algorithmic steps in Appendix B (see Figure A1 for further details). We underline this important result by the following theorem, for which we provide the proof in Appendix A. It should be noted that this theorem refers to the more general target objective defined in Problem 1 and should replace the corresponding statement in Theorem 2 in [69].

Theorem 1.

Algorithm 1 always finds an optimal solution to Problem 1. The computational complexity is of the order $O(M·N^{\tau +1}·R^{\nu -1})$, where R denotes an upper bound on the number of states of each auxiliary variable, and ν is defined as the maximal number of neighbors of a node in a corresponding clique tree.

Besides the treewidth $\tau$, the value of the parameter $\nu$ also appears to be crucial for the resulting running time of Algorithm 1 since the corresponding complexity is also exponential in $\nu$. The following proposition suggests that among all possible cluster graphs for a given MRF, there always exists a clique tree for which $\nu$ tends to take on small values (provided $\tau$ is small) and effectively does not depend on the size of the corresponding MRF. We provide the proof in Appendix C.

Proposition 2.

For any MRF with treewidth τ, there is a clique tree for which the maximal number of neighbors of each node is upper bounded according to $\nu ⩽2^{\tau +2}-4$.

To support the above proposition, we consider the following extreme example, which is illustrated in Figure 4. We are given an MRF with a star-like shape (on the left) with $M=7$ variables and treewidth $\tau =1$. One valid clique tree for this MRF is shown in the middle. In particular, the clique containing the variables $y_1,y_2$ has $\nu =M-1$ neighbors. Therefore, running Algorithm 1 on that clique tree results in a computational time exponential in the graph size M. However, it is easy to modify that clique tree to have a small number of neighbors for each node (shown on the right), upper bounded by $\nu =\tau +1=2$.

Although Proposition 2 assures the existence of a clique tree with a small $\nu$, the actual upper bound on $\nu$ is still very pessimistic (exponential in the treewidth). In fact, by allowing a simple graph modification, we can always reduce the $\nu$-parameter to a small constant ($\nu =3$). Specifically, we can clone each cluster node with more than three neighbors multiple times so that each clone only carries one of the original neighbors. We then connect the clones by a chain that preserves the running intersection property. To ensure that the new cluster graph describes the same set of potentials we set the potentials for each copy of a cluster node $C_i$ to zero: $f_{C_i}\left({\mathit{y}}_{C_i}\right)=0$ and ${\mathit{g}}_{C_i}\left({\mathit{y}}_{C_i}\right)=0$. The complete modification procedure is illustrated in Figure 5. We summarize this result in the following corollary.

Corollary 1.

Provided a given clique tree is modified according to the presented procedure for reducing the number of neighbors for each cluster node, the overall computational complexity of running Algorithm 1 (including time for graph modification) is of the order $O(M·N^{\tau +1}·R^2)$.

Note that in the case where the corresponding clique tree is a chain, the resulting complexity reduces to $O(M·N^{\tau +1}·R)$. At this point, we would like to provide an alternative perspective of the computational complexity in this case (with $\tau =1$, $R\sim M^2$), which shows the connection to the conventional junction-tree algorithm. Specifically, the constrained message-passing algorithm (Algorithm 1) can be seen as conventional message passing on a clique tree (for the mapping F in Problem 1) without auxiliary variables, but with an increased size of the state space for each variable $y_i$, from N to $N·M$. Then, Proposition 1 guarantees an exact inference in time of the order $O(M·{(N·M)}^{\tau +1})$. The summation constraints with respect to the auxiliary variables can be ensured by extending the corresponding potential functions $f_{C_i}$ to take on $-\infty$, forbidding inconsistent state transitions between individual variables. The same observation holds for message passing on factor graphs. To summarize, by introducing auxiliary variables, we can remove the global dependencies imposed by the mapping H in Problem 1, thereby reducing the overall treewidth. However, this comes at the cost of the label space becoming a function of the graph size (R is usually dependent on M).

We conclude our discussion by analyzing the relation between the maximal number of states (of the auxiliary variables) R and the number of variables M in the original MRF. In the worst case, R can grow exponentially with the graph size M. This happens, for example, when the values that the individual factors ${\mathit{g}}_{C_k}$ can take on are scattered across a very large range that grows much faster relative to the graph size. For practical cases, however, we can assume that the individual factor functions ${\mathit{g}}_{C_k}$ take values in an integer interval, which is either fixed or grows polynomially with the graph size. In that case, R is always a polynomial in M, rendering the overall complexity of Algorithm 1 a polynomial in the graph size, as we demonstrate with several examples in Section 6. We summarize this in the following theorem. The corresponding proof is given in Appendix D.

Theorem 2.

Consider an instance of Problem 1 given by a clique tree with M variables. Let $T\in \mathbb{N}$ be a number that grows polynomially with M. Provided each factor ${\mathit{g}}_{C_k}$ in a decomposition of $\mathit{G}$ assumes values from a discrete set of integers $[-T,T]\cap \mathbb{Z}$, the number R grows polynomially with M according to $R\sim T·M$.

4. Application Use Cases

In this section, we demonstrate the expressivity of Problem 1 by showcasing the diversity of existing (and potentially new) applications that align with our target objective. In particular, practitioners can gain insight into specific examples to verify whether a given task is an instance of Problem 1. The research conducted within the scope of this paper has been motivated by the following use cases:

• Learning with High-Order Loss Functions
  SSVM enables building complex and accurate models for structured prediction by directly integrating the desired performance measure into the training objective. However, its applicability relies on the availability of efficient inference algorithms. In the state-of-the-art training algorithms, such as cutting planes [64,65], bundle methods [67,68], subgradient methods [18], and Frank–Wolfe optimization [66], inference is repeatedly performed either to compute a subgradient or find the most violating configuration. In the literature, the corresponding computational task is generally referred to as the loss-augmented inference, which is the main computational bottleneck during training.
• Enabling Training of Slack Scaling Formulation for SSVMs
  The maximum-margin framework of SSVMs includes two loss-sensitive formulations known as margin scaling and slack scaling. Since the original paper on SSVMs [39], there has been much speculation about the differences in training using either of these two formulations. In particular, training via slack scaling has been conjectured to be more accurate and beneficial than margin scaling. Nevertheless, it has rarely been used in practice due to the lack of known efficient inference algorithms.
• Evaluating Generalization Bounds for Structured Prediction
  The purpose of generalization bounds is to provide useful theoretical insights into the behavior and stability of a learning algorithm by upper bounding the expected loss or the risk of a prediction function. Evaluating such a bound could provide certain guarantees on how a system trained on some finite data will perform in the future on unseen examples. Unlike in standard regression or classification tasks with univariate real-valued outputs, in structured prediction, evaluating generalization bounds requires solving a combinatorial optimization problem, thereby limiting its use in practice [48].
• Globally Constrained MAP Inference
  In many cases, evaluating a prediction function with structured outputs technically corresponds to performing MAP inference on a discrete graphical model, including Markov random fields (MRFs) [1], probabilistic context-free grammars (PCFGs) [72,73,74], hidden Markov models (HMMs) [75], conditional random fields (CRFs) [3], probabilistic relational models (PRMs) [76,77], and Markov logic networks (MLNs) [78]. In practice, we might want to modify the prediction function by imposing additional (global) constraints on its output. For example, we could perform a corresponding MAP inference subject to the constraints on the label counts specifying the size of the output or the distribution of the resulting labels, which is a common approach in applications such as sequence tagging and image segmentation. Alternatively, we might want to generate the best output with a score from a specific range that can provide deeper insights into the energy function of a corresponding model. Finally, we might want to restrict the set of possible outputs directly by excluding specific label configurations. The latter is closely related to the computational task known as (diverse) k-best MAP inference [62,63].

In the following, we provide technical details about how the generic tasks listed above can be addressed using our framework.

4.1. Loss-Augmented Inference with High-Order Loss Functions

As already mentioned, Problem 1 covers as a special case the task of loss-augmented inference (for margin and slack scaling) within the framework of SSVM [39,46]. In order to match the generic representation given in (2), we can define $F\left(\mathit{y}\right)={\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})+\mathrm{const}$, and $\eta \left(\mathit{G}\left(\mathit{y}\right)\right)=\Delta ({\mathit{y}}^{*},\mathit{y})$ for suitable $\mathit{G}:\mathcal{Y}\to {\mathbb{R}}^P$ and $\eta :{\mathbb{R}}^P\to {\mathbb{R}}_{+}$. Here, $\Psi :\mathcal{X}\times \mathcal{Y}\to {\mathbb{R}}^d,d\in \mathbb{N}$ denotes a joint feature map on an input–output pair $(\mathit{x},\mathit{y})$, $\mathit{w}\in {\mathbb{R}}^d$ is a trainable weight vector, and $\Delta :\mathcal{Y}\times \mathcal{Y}\to \mathbb{R}$ is a dissimilarity measure between a prediction $\mathit{y}$ and a true output ${\mathit{y}}^{*}$. Given this notation, our target objective can be written as follows:

$$H\left(F\right(\mathit{y}),\mathit{G}(\mathit{y}\left)\right)=F\left(\mathit{y}\right)\odot \eta \left(\mathit{G}\right(\mathit{y}\left)\right),\odot \in \{+,·\}.$$

(11)

We note that a considerable number of non-decomposable (or high-order) loss functions in structured prediction can be represented as multivariate cardinality-based potentials $\mathit{y}\mapsto \eta \left(\mathit{G}\right(\mathit{y}\left)\right)$, where the mapping $\mathit{G}$ encodes the label statistics, e.g., the number of true or false positives with respect to the ground truth. Furthermore, the maximal number of states R for the corresponding auxiliary variables related to $\mathit{G}$ is polynomially bounded in the number of variables M (see Table 1 in Section 5 for an overview of existing loss functions and the resulting values for R). For the specific case of a chain graph with $F_{\beta }$-loss, for example, the resulting complexity $O(M^3·N^2)$ of Algorithm 1 is cubic in the graph size.

4.2. Evaluating Generalization Bounds in Structured Prediction

In the following, we demonstrate how our algorithmic idea can be used to evaluate the PAC-Bayesian generalization bounds for max-margin structured prediction. As a working example, we consider the following generalization theorem, as stated in [48]:

Theorem 3.

Assume that $0⩽\Delta ({\mathit{y}}^{*},\mathit{y})⩽1$. With a probability of at least $1-\delta$ over the draw of the training set $\mathcal{S}=\{({\mathit{x}}_1,{\mathit{y}}_1),\cdots ,({\mathit{x}}_1,{\mathit{y}}_n)\}$ of size $n\in \mathbb{N}$, the following holds simultaneously for all weight vectors $\mathit{w}$:

$$\begin{array}{c}\hfill {\mathit{E}}_{(\mathit{x},\mathit{y})\sim \rho }[\Delta (\mathit{y},h_{\mathit{w}}\left(\mathit{x}\right))]⩽\frac{{\parallel \mathit{w}\parallel }^2}{n}+\sqrt{\frac{{\parallel \mathit{w}\parallel }^{2}ln\left(\frac{2dn}{{\parallel \mathit{w}\parallel }^2}\right)+ln\left(\frac{n}{\delta }\right)}{2(n-1)}}\\ \hfill +\frac{1}{n}\sum _{i=1}^n\underset{\widehat{\mathit{y}}}{max}{\mathbb{1}}_1\left[{\mathit{w}}^{\top }(\Psi ({\mathit{x}}_i,{\mathit{y}}_i)-\Psi ({\mathit{x}}_i,\widehat{\mathit{y}}))⩽{\Delta }_{\mathrm{HD}}({\mathit{y}}_i,\widehat{\mathit{y}})\right]·\Delta ({\mathit{y}}_i,\widehat{\mathit{y}})\end{array}$$

(12)

where $h_{\mathit{w}}\left(\mathit{x}\right)=\underset{\widehat{\mathit{y}}}{\mathrm{argmax}}{\mathit{w}}^{\top }\Psi (\mathit{x},\widehat{\mathit{y}})$ denotes the corresponding prediction function.

Evaluating the second term on the right-hand side of the inequality in (12) involves a maximization over $\mathit{y}\in \mathcal{Y}$ for each data point $(\mathit{x},{\mathit{y}}^{*})$ according to

$$\underset{\mathit{y}\in \mathcal{Y}}{max}{\mathbb{1}}_1\left[{\mathit{w}}^{\top }\left(\Psi (\mathit{x},{\mathit{y}}^{*})-\Psi (\mathit{x},\mathit{y})\right)⩽{\Delta }_{\mathrm{HD}}({\mathit{y}}^{*},\mathit{y})\right]·\Delta ({\mathit{y}}^{*},\mathit{y}).$$

We now show that this maximization term is an instance of Problem 1. More precisely, we consider an example with $\tau >1$ and an $F_1$-loss (see Table 1). Next, we define $F\left(\mathit{y}\right)={\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})$, $\eta \left(\mathit{G}\left(\mathit{y}\right)\right)={\Delta }_{F_1}({\mathit{y}}^{*},\mathit{y})$, and set

$$H(F\left(\mathit{y}\right),\mathit{G}\left(\mathit{y}\right))={\mathbb{1}}_1\left[{\mathit{w}}^{\top }\Psi (\mathit{x},{\mathit{y}}^{*})-F\left(\mathit{y}\right)⩽\left|{\mathit{y}}^{*}\right|-G_1\left(\mathit{y}\right)+G_2\left(\mathit{y}\right)\right]·\eta \left(\mathit{G}\left(\mathit{y}\right)\right)$$

where we use ${\Delta }_{\mathrm{HD}}({\mathit{y}}^{*},\mathit{y})=FP+FN$, $FN= |{\mathit{y}}^{*}|-TP$, which removes the need for additional auxiliary variables for the Hamming distance, reducing the resulting computational cost. Here, $TP$, $FP$, and $FN$ denote the numbers of true positives, false positives, and false negatives, respectively. $|{\mathit{y}}^{*}|$ denotes the size of the output ${\mathit{y}}^{*}$. Both $|{\mathit{y}}^{*}|$ and ${\mathit{w}}^{\top }\Psi (\mathit{x},{\mathit{y}}^{*})$ are constant with respect to the maximization over $\mathit{y}$. Note also that H is non-decreasing in $F\left(\mathit{y}\right)$. Furthermore, the number of states of the auxiliary variables is upper bounded by $R=M^2$ (see Table 1). Therefore, here, the computational complexity of Algorithm 1 (according to Corollary 1) is given by $O(M^5·N^{\tau +1})$.

As a final remark, we note that training an SSVM corresponds to solving a convex problem but is not consistent. It fails to converge to the optimal predictor even in the limit of infinite training data (see [60] for more details). However, minimizing the (non-convex) generalization bound is consistent. Algorithm 1 provides an effective evaluation tool that could potentially be used for the development of new training algorithms based on the direct minimization of such bounds. We leave the corresponding investigation for future work.

4.3. Globally-Constrained MAP Inference

Another common use case is performing MAP inference on graphical models (such as MRFs) subject to additional constraints on the variables or the range of the corresponding objective including various tasks such as image segmentation in computer vision, sequence tagging in computational biology or natural language processing, and signal denoising in information theory. We note that an important tractability assumption in the definition of Problem 1 is the $\tau$-decomposability of F and $\mathit{G}$ with a reasonably small treewidth $\tau$. In areas such as computer vision, we usually encounter models (e.g., Ising grid model) where the treewidth is a function of the graph size given by $\tau =\sqrt{M}$. In this case, we can use Algorithm 1 by leveraging the technique of dual decomposition [5,16,79]. More precisely, we decompose the original graph in (multiple) trees, where the global factor is attached to exactly one of the trees in our decomposition. We also note that from a technical perspective, the problem of MAP inference subject to some global constraints on the statistics $\mathit{G}\left(\mathit{y}\right)$ is equivalent to the MAP problem augmented with a global cardinality-based potential $\eta \left(\mathit{G}\right(\mathit{y}\left)\right)$. Specifically, we can define $\eta$ as an indicator function ${\mathbb{1}}_{-\infty }[·]$, which returns $-\infty$ if the corresponding constraint on $\mathit{G}\left(\mathit{y}\right)$ is violated. Furthermore, the form of $\eta$ does not affect the message passing of the presented algorithm. We can always check the validity of a corresponding constraint after all the necessary statistics have been computed.

4.3.1. Constraints on Label Counts

As a simple example, consider the binary-sequence tagging experiment, that is, every output $\mathit{y}\in \mathcal{Y}$ is a sequence, and each site in the sequence can be either 0 or 1. Given some prior information on the number b of positive labels, we can improve the quality of the results by imposing a corresponding constraint on the outputs:

$$\begin{array}{cccc}\hfill & \underset{\mathit{y}\in \mathcal{Y}}{\mathrm{maximize}}\hfill & \hfill & {\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & \sum _{i=1}^M{y}_i=b\hfill \end{array}$$

(13)

We can write this as an instance of Problem 1 by setting

$$H(F\left(\mathit{y}\right),G\left(\mathit{y}\right))=F\left(\mathit{y}\right)+{\mathbb{1}}_{-\infty }[G\left(\mathit{y}\right)\ne b],$$

(14)

where $F\left(\mathit{y}\right)={\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})$, and $G\left(\mathit{y}\right)={\sum }_{i=1}^M{y}_i$. Since all the variables in $\mathit{y}$ are binary, the number of states R of the corresponding auxiliary variables $l_m={\sum }_{i=1}^m{y}_i$ is upper bounded by M. In addition, because the output graph is a sequence, we have $\tau =1$, $\nu =2$. Therefore, here, the computational complexity of Algorithm 1 is of the order $O(M^2·N^2)$.

4.3.2. Constraints on Objective Value

We continue with the binary-sequence tagging example (with pairwise interactions). To force constraints on the score to be in a specific range, as in

$$\begin{array}{cccc}\hfill & \underset{\mathit{y}\in \mathcal{Y}}{\mathrm{maximize}}\hfill & \hfill & {\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\hfill & \hfill & a⩽{\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})⩽b\hfill \end{array}$$

(15)

we first rewrite the prediction function in terms of its sufficient statistics according to

$${\mathit{w}}^{\top }\Psi (\mathit{x},\mathit{y})={\displaystyle \sum _{o,s}}{w}_{o,s}\underset{=:G_{o,s}\left(\mathit{y}\right)}{\underbrace{\sum _{t=1}^M{\mathbb{1}}_1[x_t=o\wedge y_t=s]}}+\sum _{s_1,s_2}{w}_{s_1,s_2}\underset{=:G_{s_1,s_2}\left(\mathit{y}\right)}{\underbrace{\sum _{t=2}^M{\mathbb{1}}_1[y_{t-1}=s_1\wedge y_t=s_2]}},$$

(16)

$\mathit{w}=(\cdots ,w_{o,s},\cdots ,w_{s_1,s_2},...)$, $\mathit{G}=(\cdots ,G_{o,s},\cdots ,G_{s_1,s_2},...)$, and define

$$H(F\left(\mathit{y}\right),\mathit{G}\left(\mathit{y}\right))=F\left(\mathit{y}\right)+{\mathbb{1}}_{-\infty }[{\mathit{w}}^{\top }\mathit{G}\left(\mathit{y}\right)\notin [a,b]].$$

(17)

Note that $\mathit{G}$ contains all the sufficient statistics of F such that $F\left(\mathit{y}\right)={\mathit{w}}^{\top }\mathit{G}\left(\mathit{y}\right)$. Here, we could replace $a⩽{\mathit{w}}^{\top }\Psi \left(\mathit{y}\right)⩽b$ with any (non-linear) constraint on the sufficient statistics of the joint feature map $\Psi \left(\mathit{y}\right)$.

The corresponding computational complexity can be derived by considering an urn problem: one with $D·N$ and one with $N^2$ distinguishable urns and M indistinguishable balls. Here, D denotes the size of the dictionary for the observations $x_t$ in the input sequence $\mathit{x}$. Note that the dictionary of the input symbols can be large compared to other problem parameters. However, we can reduce D to the size of the vocabulary only occurring in the current input $\mathit{x}$. The first urn problem corresponds to the unary observation-state statistics $G_{o,s}\left(\mathit{y}\right)$, and the second corresponds to the pairwise statistics for the state transition $G_{s_1,s_2}\left(\mathit{y}\right)$. The resulting number of possible distributions of balls over the urns is given by

$$\underset{⩽M^{D·N}}{\underbrace{\left(\genfrac{}{}{0pt}{}{M+D·N-1}{M}\right)}}·\underset{⩽M^{N^2}}{\underbrace{\left(\genfrac{}{}{0pt}{}{M+N^2-1}{M}\right)}}⩽\underset{=R}{\underbrace{M^{D·N+N^2}}}.$$

(18)

Although the resulting complexity (due to $\nu =2$) being $O(M^{D·N+N^2+1}·N^2)$ is still a polynomial in the number of variables M, the degree is quite high, making it suitable for only short sequences. For practical use, we recommend the efficient approximation framework of Lagrangian relaxation and Dual Decomposition [5,16,79].

4.3.3. Constraints on Search Space

The constraints on the search space can be different from the constraints we can impose on the label counts. For example, we might want to exclude a set of K complete outputs $\{{\mathit{y}}^1,\cdots ,{\mathit{y}}^K\}$ from the feasible set $\mathcal{Y}$ by using an exclusion potential ${\mathbb{1}}_{-\infty }[\mathit{y}\in \{{\mathit{y}}^1,\cdots ,{\mathit{y}}^K\}]$.

For simplicity, we again consider a sequence-tagging example with pairwise dependencies. Given a set of K patterns to exclude, we can introduce auxiliary variables ${\mathit{l}}_m\in {\{0,1\}}^K$, where for each pattern ${\mathit{y}}^k$, we have a constraint ${\left({\mathit{l}}_m\right)}_k=max\{{\mathbb{1}}_1[y_m^k\ne y_m],{\left({\mathit{l}}_{m-1}\right)}_k\}$. More precisely, we modify the message computation in (7) with respect to the auxiliary variables by replacing the corresponding constraints ${\left({\mathit{l}}_m\right)}_k={\left({\mathit{l}}_{m-1}\right)}_k+{\mathbb{1}}_1[y_m^k\ne y_m]$ in the maximization over $\left\{{\mathit{l}}_m\right\}$ with the constraints ${\left({\mathit{l}}_m\right)}_k=max\{{\mathbb{1}}_1[y_m^k\ne y_m],{\left({\mathit{l}}_{m-1}\right)}_k\}$. Therefore, the maximal number of states for ${\mathit{l}}_m$ is given by $R=2^K$. The resulting complexity for finding an optimal solution over $\mathit{y}\in \mathcal{Y}\setminus \{{\mathit{y}}^1,\cdots ,{\mathit{y}}^K\}$ is of the order $O(2^K·M·N^2)$.

A related problem is finding a diverse k-best solution. Here, the goal is to produce the best solutions that are sufficiently different from each other according to a diversity function, e.g., a loss function such as the Hamming distance ${\Delta }_{\mathrm{HD}}$. More precisely, after computing the MAP solution ${\mathit{y}}^1$, we compute the second-best (diverse) output ${\mathit{y}}^2$ with ${\Delta }_{\mathrm{HD}}({\mathit{y}}^1,{\mathit{y}}^2)⩾m_1$. For the third-best solution, we then require ${\Delta }_{\mathrm{HD}}({\mathit{y}}^1,{\mathit{y}}^3)⩾m_2$ and ${\Delta }_{\mathrm{HD}}({\mathit{y}}^2,{\mathit{y}}^3)⩾m_2$, and so on. In other words, we search for an optimal output ${\mathit{y}}^K$ such that ${\Delta }_{\mathrm{HD}}({\mathit{y}}^k,{\mathit{y}}^K)⩾m_{K-1}$, $m_k\in \mathbb{N}$ for all $k\in \{1,\cdots ,K-1\}$.

For this purpose, we define auxiliary variables ${\mathit{l}}_m\in {\{0,\cdots ,M\}}^{K-1}$, where for each pattern ${\mathit{y}}^k$, we have a constraint ${\left({\mathit{l}}_m\right)}_k={\left({\mathit{l}}_{m-1}\right)}_k+{\mathbb{1}}_1[y_m^k\ne y_m]$, which computes the Hamming distance of a solution $\mathit{y}$ with respect to the pattern ${\mathit{y}}^k$. Therefore, we can define

$$H(F\left(\mathit{y}\right),G\left(\mathit{y}\right))=F\left(\mathit{y}\right)+{\mathbb{1}}_{-\infty }[\exists k\in \{1,\cdots ,K-1\}:G_k\left(\mathit{y}\right)<m_k]$$

(19)

where $\mathit{G}=(G_1,\cdots ,G_{K-1})$, and at the final stage (due to $\mathit{G}\left(\mathit{y}\right)={\mathit{l}}_M$), we have all the necessary information to evaluate the constraints with respect to the diversity function (here Hamming distance). The maximal number of states R for the auxiliary variables is upper bounded by $M^{K-1}$. Therefore, the resulting running time for finding the K-th diverse output sequence is of the order $O(M^K·N^{\tau +1})$.

Finally, we note that the concept of diverse K-best solutions can also be used during the training of SSVMs to speed up the convergence of a corresponding algorithm by generating diverse cutting planes or subgradients, as described in [63]. An appealing property of Algorithm 1 is that we obtain some of the necessary information for free as a side effect of the message passing.

5. Compact Representation of Loss Functions

We now further advance the task of loss-augmented inference (see Section 4.1) by presenting a list of popular dissimilarity measures that our algorithm can handle, which are summarized in Table 1. The measures are given in a compact representation $\Delta ({\mathit{y}}^{*},\mathit{y})=\eta \left(\mathit{G}\left(\mathit{y}\right)\right)$ based on the corresponding sufficient statistics encoded as a mapping $\mathit{G}$. Columns 2 and 3 show the form of $\mathit{G}(·)$ and $\eta$, respectively. Column 4 provides an upper bound R on the number of possible values of auxiliary variables that affect the resulting running time of Algorithm 1 (see Corollary 1).

Table 1. Compact representation of popular dissimilarity measures based on the corresponding sufficient statistics $\mathit{G}(·)$. The upper (and lower!) bounds on the number of states of the auxiliary variables R for the presented loss functions are shown in the last column.

Loss	$\mathit{G}\left(\mathit{y}\right)$	$\mathit{\eta }\left(\mathit{G}\right(·\left)\right)$	R
${\Delta }_{0/1}$	$(\mathit{TP},{\mathbb{1}}_1[\mathit{FP}>0])$	${\mathbb{1}}_1[max\{M-G_1,G_2\}>0]$	$M^2$
${\Delta }_{\mathit{HD}}$	${\sum }_{t=1}^M{\mathbb{1}}_1[y_t^{*}\ne y_t]$	G	M
${\Delta }_{\mathit{HL}}$	${\sum }_{t=1}^M{\mathbb{1}}_1[y_t^{*}\ne y_t]$	$G/M$	M
${\Delta }_{\mathit{WHD}}$	${\{\#(s_1,s_2)\}}_{s_1,s_2\in \{1,\cdots ,N\}}$	${\sum }_{s_1,s_2}\mathrm{weight}(s_1,s_2)·G_{s_1,s_2}$	$M^{N^2}$
${\Delta }_{\#\mathit{FP}}$	$\mathit{FP}$	G	M
${\Delta }_{\mathit{R}}$	$\mathit{TP}$	$1-G/|{\mathit{y}}^{*}|$	M
${\Delta }_{\mathit{P}}$	$(\mathit{TP},\mathit{FP})$	$1-\frac{G_1}{G_1+G_2}$	$M^2$
${\Delta }_{F_{\beta }}$	$(\mathit{TP},\mathit{FP})$	$1-\frac{(1+{\beta }^2)·G_1}{{\beta }^2·\left|{\mathit{y}}^{*}\right|+G_1+G_2}$	$M^2$
${\Delta }_{\cap /\cup }$	$(\mathit{TP},\mathit{FP})$	$1-\frac{G_1}{|{\mathit{y}}^{*}|+G_2}$	$M^2$
${\Delta }_{\mathit{LC}}$	${\sum }_{t=1}^M{y}_t$	$\left|G-{\sum }_{t=1}^M{y}_t^{*}\right|$	M
${\Delta }_{\#\mathit{CB}}$	$\#\mathit{CB}$	G	M
${\Delta }_{\mathit{CBR}}$	$(\#\mathit{CB},|\mathit{y}\left|\right)$	$G_1/G_2$	$M^2$
${\Delta }_{\mathit{BLEU}}$	$({\mathit{TP}}_1,{\mathit{FP}}_1,\cdots ,{\mathit{TP}}_K,{\mathit{FP}}_K)$	$1-\mathit{BP}(·)·exp\left(\frac{1}{K}{\sum }_{k=1}^{K}log{p}_k\right)$	$M^{2K}$
${\Delta }_{\mathit{ROUGE}-\mathit{K}}$	${\left\{\mathrm{count}(k,X)\right\}}_{k\in \mathrm{grams}\left(Ref\right)}$	$1-\frac{{\sum }_{S\in Ref}{\sum }_{k\in \mathrm{grams}\left(S\right)}min\{\mathrm{count}(k,S),G_k\}}{{\sum }_{S\in Ref}{\sum }_{k\in \mathrm{grams}\left(S\right)}\mathrm{count}(k,S)}$	$M^D$
${\Delta }_{\mathit{ROUGE}-\mathit{LCS}}$	${\left\{\mathit{LCS}(X,S)\right\}}_{S\in Ref}$	$1-\frac{1}{\left|Ref\right|}{\sum }_{S\in Ref}\frac{(1+{\beta }^2)P\left(G_S\right)·R\left(G_S\right)}{{\beta }^{2}P\left(G_S\right)+R\left(G_S\right)}$	$M^{2\left|Ref\right|}$

Table 1. Compact representation of popular dissimilarity measures based on the corresponding sufficient statistics $\mathit{G}(·)$. The upper (and lower!) bounds on the number of states of the auxiliary variables R for the presented loss functions are shown in the last column.

Loss	$\mathit{G}\left(\mathit{y}\right)$	$\mathit{\eta }\left(\mathit{G}\right(·\left)\right)$	R
${\Delta }_{0/1}$	$(\mathit{TP},{\mathbb{1}}_1[\mathit{FP}>0])$	${\mathbb{1}}_1[max\{M-G_1,G_2\}>0]$	$M^2$
${\Delta }_{\mathit{HD}}$	${\sum }_{t=1}^M{\mathbb{1}}_1[y_t^{*}\ne y_t]$	G	M
${\Delta }_{\mathit{HL}}$	${\sum }_{t=1}^M{\mathbb{1}}_1[y_t^{*}\ne y_t]$	$G/M$	M
${\Delta }_{\mathit{WHD}}$	${\{\#(s_1,s_2)\}}_{s_1,s_2\in \{1,\cdots ,N\}}$	${\sum }_{s_1,s_2}\mathrm{weight}(s_1,s_2)·G_{s_1,s_2}$	$M^{N^2}$
${\Delta }_{\#\mathit{FP}}$	$\mathit{FP}$	G	M
${\Delta }_{\mathit{R}}$	$\mathit{TP}$	$1-G/|{\mathit{y}}^{*}|$	M
${\Delta }_{\mathit{P}}$	$(\mathit{TP},\mathit{FP})$	$1-\frac{G_1}{G_1+G_2}$	$M^2$
${\Delta }_{F_{\beta }}$	$(\mathit{TP},\mathit{FP})$	$1-\frac{(1+{\beta }^2)·G_1}{{\beta }^2·\left|{\mathit{y}}^{*}\right|+G_1+G_2}$	$M^2$
${\Delta }_{\cap /\cup }$	$(\mathit{TP},\mathit{FP})$	$1-\frac{G_1}{|{\mathit{y}}^{*}|+G_2}$	$M^2$
${\Delta }_{\mathit{LC}}$	${\sum }_{t=1}^M{y}_t$	$\left|G-{\sum }_{t=1}^M{y}_t^{*}\right|$	M
${\Delta }_{\#\mathit{CB}}$	$\#\mathit{CB}$	G	M
${\Delta }_{\mathit{CBR}}$	$(\#\mathit{CB},|\mathit{y}\left|\right)$	$G_1/G_2$	$M^2$
${\Delta }_{\mathit{BLEU}}$	$({\mathit{TP}}_1,{\mathit{FP}}_1,\cdots ,{\mathit{TP}}_K,{\mathit{FP}}_K)$	$1-\mathit{BP}(·)·exp\left(\frac{1}{K}{\sum }_{k=1}^{K}log{p}_k\right)$	$M^{2K}$
${\Delta }_{\mathit{ROUGE}-\mathit{K}}$	${\left\{\mathrm{count}(k,X)\right\}}_{k\in \mathrm{grams}\left(Ref\right)}$	$1-\frac{{\sum }_{S\in Ref}{\sum }_{k\in \mathrm{grams}\left(S\right)}min\{\mathrm{count}(k,S),G_k\}}{{\sum }_{S\in Ref}{\sum }_{k\in \mathrm{grams}\left(S\right)}\mathrm{count}(k,S)}$	$M^D$
${\Delta }_{\mathit{ROUGE}-\mathit{LCS}}$	${\left\{\mathit{LCS}(X,S)\right\}}_{S\in Ref}$	$1-\frac{1}{\left|Ref\right|}{\sum }_{S\in Ref}\frac{(1+{\beta }^2)P\left(G_S\right)·R\left(G_S\right)}{{\beta }^{2}P\left(G_S\right)+R\left(G_S\right)}$	$M^{2\left|Ref\right|}$

Here, $\left|\mathit{y}\right|=M$ denotes the number of nodes in the output $\mathit{y}$. $\mathit{TP},\mathit{FP}$, and $\mathit{FN}$ are the numbers of true positives, false positives, and false negatives, respectively. The number of true positives for a prediction $\mathit{y}$ and a true output ${\mathit{y}}^{*}$ is defined as the number of common nodes with the same label. The number of false positives is determined by the number of nodes that are present in the output $\mathit{y}$ but missing (or with another label) in the true output ${\mathit{y}}^{*}$. Similarly, the number of false negatives corresponds to the number of nodes present in ${\mathit{y}}^{*}$ but missing (or with another label) in $\mathit{y}$. In particular, it holds that $|{\mathit{y}}^{*}| =\mathit{TP}+\mathit{FN}$.

We can see in Table 1 that each element of $\mathit{G}(·)$ is a sum of binary variables, which significantly reduces the image size of mapping $\mathit{G}(·)$. This occurs despite the exponential variety of the output space $\mathcal{Y}$. As a result, the image size grows only polynomially with the size of the outputs $\mathit{y}\in \mathcal{Y}$, and the number R provides an upper bound on the image size of $\mathit{G}(·)$.

Zero-One Loss (${\Delta }_{0/1}$)

This loss function takes on binary values $\{0,1\}$ and is the most uninformative since it requires a prediction to match the ground truth to 100% and provides no partial quantification of the prediction quality in the opposite case. Technically, this measure is not decomposable since it requires the numbers of $\mathit{FP}$ and $\mathit{FN}$ to be evaluated via

$${\Delta }_{0/1}({\mathit{y}}^{*},\mathit{y})={\mathbb{1}}_1[max\{\mathit{FP},\mathit{FN}\}>0].$$

(20)

Sometimes, we cannot compute the $\mathit{FN}$ (unlike $\mathit{FP}$) from the individual nodes of a prediction. Instead, we can count the $\mathit{TP}$ and compute the $\mathit{FN}$ using the relationship $|{\mathit{y}}^{*}|=\mathit{TP}+\mathit{FN}$. For example, if the outputs $\mathit{y}\in \mathcal{Y}$ are set with no ordering indication of the individual set elements, we need to know the whole set $\mathit{y}$ in order to be able to compute the $\mathit{FN}$. Therefore, computing $\mathit{FN}$ from a partially constructed output is not possible. We note, however, that in the case of the zero-one loss function, there is a faster inference approach, which involves modifying the prediction algorithm to also compute the second-best output and selecting the best result based on the value of the objective function.

Hamming Distance/Hamming Loss (${\Delta }_{\mathit{HD}},{\Delta }_{\mathit{HL}}$)

In the context of sequence learning, given a true output ${\mathit{y}}^{*}$ and a prediction $\mathit{y}$ of the same length, the Hamming distance measures the number of states on which the two sequences disagree:

$${\Delta }_{\mathit{HD}}({\mathit{y}}^{*},\mathit{y})=\sum _{t=1}^M{\mathbb{1}}_1[{\mathit{y}}_t^{*}\ne {\mathit{y}}_t].$$

(21)

By normalizing this value, we obtain the Hamming loss, which does not depend on the length of the sequences. Both measures are decomposable.

Weighted Hamming Distance (${\Delta }_{\mathit{WHD}}$)

For a given matrix $\mathrm{weight}\in {\mathbb{R}}^{N\times N}$, the weighted Hamming distance is defined as ${\Delta }_{\mathit{WHD}}({\mathit{y}}^{*},\mathit{y})={\sum }_{t=1}^M\mathrm{weight}(y_t^{*},y_t)$. However, keeping track of the accumulated sum of the weights until the current position t in a sequence, unlike for the Hamming distance, can be intractable. We can, however, use the following observation. It is sufficient to count the occurrences $({\mathit{y}}_t^{*},{\mathit{y}}_t)$ for each pair of states ${\mathit{y}}_t^{*},{\mathit{y}}_t\in \{1,\cdots ,N\}$ according to

$$\sum _{t=1}^M\mathrm{weight}(y_t^{*},y_t)=\sum _{s_1,s_2}\mathrm{weight}(s_1,s_2)\sum _{t=1}^M{\mathbb{1}}_1[y_t^{*}=s_1\wedge y_t=s_2].$$

(22)

In other words, each dimension of $\mathit{G}$ (denoted as $G_{s_1,s_2}$) corresponds to

$$G_{s_1,s_2}(\mathit{y};{\mathit{y}}^{*})=\sum _{t=1}^M{\mathbb{1}}_1[y_t^{*}=s_1\wedge y_t=s_2].$$

(23)

Here, we can upper bound the image size of $\mathit{G}(·)$ by considering an urn problem with $N^2$ distinguishable urns and M indistinguishable balls. The number of possible distributions of the balls over the urns is given by $\left(\genfrac{}{}{0pt}{}{M+N^2-1}{M}\right)⩽M^{N^2}$.

False Positives/Precision/Recall (${\Delta }_{\#\mathit{FP}},{\Delta }_{\mathit{P}},{\Delta }_{\mathit{R}}$)

False positives measure the discrepancy between outputs by counting the number of false positives in a prediction $\mathit{y}$ with respect to the true output ${\mathit{y}}^{*}$. This metric is often used in learning tasks such as natural language parsing due to its simplicity. Precision and recall are popular measures used in information retrieval. By subtracting the corresponding values from one, we can easily convert them to a loss function. Unlike precision given by $\mathit{TP}/(\mathit{TP}+\mathit{FP})$, recall effectively depends on only one parameter. Although it is originally parameterized by two parameters given as $\mathit{TP}/(\mathit{TP}+\mathit{FN})$, we can exploit the fact that the value $|{\mathit{y}}^{*}| =\mathit{TP}+\mathit{FN}$ is always known in advance during the inference, rendering recall a decomposable measure.

$F_{\beta }$-Loss (${\Delta }_{F_{\beta }}$)

The $F_{\beta =1}$-score is often used to evaluate performance in various natural language processing applications and is also suitable for many structured prediction tasks. It is originally defined as the harmonic mean of precision and recall

$$F_1=\frac{2\mathit{TP}}{2\mathit{TP}+\mathit{FP}+FN}.$$

(24)

However, since the value $|{\mathit{y}}^{*}| =\mathit{TP}+\mathit{FN}$ is always known in advance during the inference, the $F_{\beta }$-score effectively depends on only two parameters $(\mathit{TP},\mathit{FP})$. The corresponding loss function is defined as ${\Delta }_{F_{\beta }}=1-F_{\beta }$.

Intersection Over Union (${\Delta }_{\cap /\cup }$)

The Intersection-Over-Union loss is mostly used in image processing tasks such as image segmentation and object recognition and was used as a performance measure in the Pascal Visual Object Classes Challenge [80]. It is defined as $1-area({\mathit{y}}^{*}\cap \mathit{y})/area({\mathit{y}}^{*}\cup \mathit{y})$. We can easily interpret this value in cases where the outputs ${\mathit{y}}^{*}$, $\mathit{y}$ describe the bounding boxes of pixels. The more the overlap of two boxes, the smaller the loss value. We note that in the case of binary image segmentation, for example, we have a different interpretation of true and false positives. In particular, it holds that $\mathit{TP}+\mathit{FN}=\mathit{P}$, where $\mathit{P}$ is the number of positive entries in ${\mathit{y}}^{*}$. In terms of the contingency table, this yields

$${\Delta }_{\cap /\cup }=1-\frac{\mathit{TP}}{(\mathit{TP}+\mathit{FP}+\mathit{FN})}.$$

(25)

Since $|{\mathit{y}}^{*}|=\mathit{TP}+\mathit{FN}$, the value ${\Delta }_{\cap /\cup }$ effectively depends on only two parameters (instead of three). Moreover, unlike $F_{\beta }$-loss, ${\Delta }_{\cap /\cup }$ defines a proper distance metric on sets.

Label-Count Loss (${\Delta }_{\mathit{LC}}$)

The Label-Count loss is a performance measure used for the task of binary image segmentation in computer vision and is given by

$$\Delta ({\mathit{y}}^{*},\mathit{y})=\frac{1}{M}\left|\sum _{i=1}^M{y}_i-\sum _{i=1}^M{y}_i^{*}\right|.$$

(26)

This loss function prevents assigning low energy to segmentation labelings with substantially different areas compared to the ground truth.

Number/Rate of Crossing Brackets (${\Delta }_{\#CB}$, ${\Delta }_{\mathit{CBR}}$)

The number of Crossing Brackets ($\#CB$) is a measure used to evaluate performance in natural language parsing. It computes the average of how many constituents in one tree $\mathit{y}$ cross over constituent boundaries in the other tree ${\mathit{y}}^{*}$. The normalized version (by $\left|\mathit{y}\right|$) of this measure is called the Crossing Brackets (Recall) Rate. Since the value $\left|\mathit{y}\right|$ is not known in advance, the evaluation requires a further parameter for the size of $\mathit{y}$.

Bilingual Evaluation Understudy (${\Delta }_{\mathit{BLEU}}$)

Bilingual Evaluation Understudy, or BLEU for short [81], is a measure used to evaluate the quality of machine translations. It computes the geometric mean of the precision $p_k=T{P}_k/(T{P}_k+F{P}_k)$ of k-grams of various lengths (for $k=1,\cdots ,K$) between a hypothesis and a set of reference translations, multiplied by a factor $BP(·)$ to penalize short sentences according to

$${\Delta }_{\mathit{BLEU}}({\mathit{y}}^{*},\mathit{y})=1-\mathit{BP}\left(\mathit{y}\right)·exp\left(\frac{1}{K}\sum _{k=1}^{K}log{p}_k\right).$$

(27)

Note that K is a constant, rendering the term $M^{2K}$ a polynomial in M.

Recall-Oriented Understudy for Gisting Evaluation (${\Delta }_{\mathit{ROUGE}-\mathit{K}}$, ${\Delta }_{\mathit{ROUGE}-\mathit{LCS}}$)

The Recall-Oriented Understudy for Gisting Evaluation, or ROUGE for short [82], is a measure used to evaluate the quality of a summary by comparing it to other summaries created by humans. More precisely, for a given set of reference summaries $Ref$ and a summary candidate X, ROUGE-K computes the percentage of k-grams from $Ref$ that appear in X according to

$$\mathit{ROUGE}-\mathit{K}(X,Ref)=\frac{{\sum }_{S\in Ref}{\sum }_{k\in \mathrm{k}-\mathrm{grams}\left(S\right)}min\{\mathrm{count}(k,X),\mathrm{count}(k,S)\}}{{\sum }_{S\in Ref}{\sum }_{k\in \mathrm{k}-\mathrm{grams}\left(S\right)}\mathrm{count}(k,S)}$$

(28)

where $\mathrm{count}(k,S)$ provides the number of occurrences of a k-gram k in a summary S. We can estimate an upper bound R on the image size of $\mathit{G}(·)$ similarly to the derivation for the weighted Hamming distance above as $M^D$, where $D:=\left|\mathrm{grams}\left(\mathrm{Ref}\right)\right|$ is the dimensionality of $\mathit{G}(·)$. This represents the number of unique k-grams occurring in the reference summaries. Note that we do not need to count grams that do not occur in the references.

Another version, ROUGE-LCS, is based on the concept of the longest common subsequence (LCS). More precisely, for two summaries X and Y, we first compute $LCS(X,Y)$, which is the length of the LCS. We then use this value to define precision and recall measures given by $LCS(X,Y)/\left|X\right|$ and $LCS(X,Y)/\left|Y\right|$, respectively. These measures are used to evaluate the corresponding F-measure:

$${\Delta }_{\mathit{ROUGE}-\mathit{LCS}}=1-\frac{1}{\left|Ref\right|}\sum _{S\in Ref}\frac{(1+{\beta }^2)P_{LCS(X,S)}·R_{LCS(X,S)}}{{\beta }^2{P}_{LCS(X,S)}+R_{LCS(X,S)}}.$$

(29)

In other words, each dimension in $\mathit{G}(·)$ is indexed by an $S\in Ref$. ROUGE-LCS (unlike ROUGE-K) is non-decomposable.

6. Validation of Theoretical Time Complexity

To demonstrate feasibility, we evaluate the performance of our algorithm on several application tasks: part-of-speech tagging [83], base-NP chunking [84], and constituency parsing [39,46]. More precisely, we consider the task of loss-augmented inference (see Section 4.1) for margin and slack scaling with different loss functions. The run-times for the task of diverse k-best MAP inference (Section 4.3) and for evaluating the structured generalization bounds (see Section 4.2) are identical to the run-time for the loss-augmented inference with slack scaling. We omit the corresponding plots due to redundancy. In all the experiments, we used Penn English Treebank-3 [85] as a benchmark data set, which provides a large corpus of annotated sentences from the Wall Street Journal. For better visualization, we restrict our experiments to sentences containing at most 40 words. The resulting time performance is shown in Figure 6. We can see that different loss functions result in different computation costs, as indicated by the number of values for the auxiliary variables shown in the last column of Table 1. In particular, the shapes of the curves are consistent with the upper bound provided in Theorem 1, which reflects the polynomial degree of the overall dependency with respect to the graph size M. The difference between margin and slack scaling is due to the fact that in the case of a decomposable loss function $\mathit{G}$, the corresponding loss terms can be folded into the factors of the compatibility function F, allowing for the use of conventional message passing.

7. Summary of Contributions

In this paper, we provided a set of formal contributions by generalizing the idea of constrained message passing introduced in [69]. Our improved framework, which is presented in this paper, includes the following key elements:

• Abstract definition of the target problem (see Problem 1);
• Constrained message-passing algorithm on clique trees (see Algorithm 1);
• Formal statements to ensure theoretical properties such as correctness and efficiency (see Theorem 1, Proposition 2, Corollary 1, Theorem 2).

We emphasize that the idea of using auxiliary variables to constrain message passing is not novel per se and was originally proposed in [69]. Our first contribution concerns the difference in the definition of the target problem, which broadens the range of admissible applications. More precisely, the target objective H in the previous publication relates the energy of the model F and the vector-valued mapping $\mathit{G}$, which describes the sufficient statistics of the global terms in a rather restricted form according to

$$H\left(F\right(·),\mathit{G}(·\left)\right):=F(·)\odot \eta \left(\mathit{G}\right(·\left)\right),\odot \in \{+,·\}$$

where $\eta$ is a non-negative function on $\mathit{G}$. Although this is sufficient for the purposes of [69], there are other computational problems that could benefit from the same message-passing idea but cannot be addressed by it in the above form. Therefore, we relax the form of the function H in our target problem (Problem 1) to allow much finer interactions between F and $\mathit{G}$. Specifically, we allow H to be any function on the arguments $(F,\mathit{G})$, which is restricted only by the requirement that it is non-decreasing in the first argument. Without this assumption, the proposed algorithm (Algorithm 1) is not guaranteed to provide an optimal solution. Therefore, all related theorems, algorithms, and the corresponding proofs in [69] must be adjusted accordingly. Conceptually, we can reuse the computational core idea, including the introduction of auxiliary variables, without any changes. The only difference in the new version of the algorithm is how the statistics gathered during message passing are evaluated at the end to ensure the optimality of the solution according to the new requirement on the function H (see line 8 in Algorithm 1). In order to motivate the significance and practical usability of this extension, we demonstrated that mappings F and $\mathit{G}$ can interact by means of the function H in a highly non-trivial way, thereby exceeding the range of problem formulations in [69]. In particular, we showed how the generalization bounds in structured prediction can be evaluated using our approach (see Section 4.2). To the best of our knowledge, there are no existing methods for finding an optimal solution for this task.

Our second contribution involves a new graph transformation technique via node cloning (illustrated in Figure 5), which significantly enhances the asymptotic bounds on the computational complexity of Algorithm 1. Previously, in [69], the polynomial running time was guaranteed only if the maximal node degree $\nu$ in a corresponding cluster graph was bounded by a (small) constant. The proposed transformation effectively removes this limitation and ensures polynomial running time regardless of the graph structure. We emphasize this result in Corollary 1. Note that the term $R^{\nu -1}$ in Theorem 1 in [69] has been replaced with $R^2$ in Corollary 1 of the current paper, thereby reducing the computational complexity of the resulting message-passing algorithm. This reduction is significant, as $\nu$ can be dependent on the graph size in the worst case (see Figure 4).

As our third contribution, we investigate the important question of how the parameter R, which describes the maximum number of states for auxiliary variables, grows with the size of the graph, as measured by the total number of variables M in the corresponding MRF. As a result, we identify a sufficient condition on $\mathit{G}$ that guarantees the overall polynomial run-time of Algorithm 1 in relation to the graph size in our target problem (Problem 1; see Theorem 2).

8. Related Works

Several previous works have addressed the problem of exact MAP inference with global factors in the context of SSVMs when optimizing for non-decomposable loss functions. Joachims [86] proposed an algorithm for a set of multivariate losses, including the $F_{\beta }$-loss. However, the presented idea applies only to a simple case, where the corresponding mapping F in (2) decomposes into non-overlapping components (e.g., unary potentials).

Similar ideas based on the introduction of auxiliary variables have been proposed [87,88] to modify the belief propagation algorithm according to a special form of high-order potentials. Specifically, for the case of binary-valued variables $y_i\in \{0,1\}$, the authors focus on the univariate cardinality potentials$\eta \left(\mathit{G}\left(\mathit{y}\right)\right)=\eta \left({\sum }_i{y}_i\right)$. For the tasks of sequence tagging and constituency parsing, ref. [45,46] propose an exact inference algorithm for the slack scaling formulation that focuses on the univariate dissimilarity measures $\mathit{G}\left(\mathit{y}\right)={\sum }_t{\mathbb{1}}_1[y_t^{*}\ne y_t]$ and $\mathit{G}\left(\mathit{y}\right)=\#\mathit{FP}\left(\mathit{y}\right)$ (see Table 1 for details). In [69] the authors extrapolate this idea and provide a unified strategy to tackle multivariate and non-decomposable loss functions.

In the current paper, we build on the results in [69] and generalize the target problem (Problem 1) by increasing the range of admissible applications. More precisely, we replace the binary operation $\odot :\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ corresponding to either a summation or multiplication in the previous objective $F\left(\mathit{y}\right)\odot \eta \left(\mathit{G}\right(\mathit{y}\left)\right)$ with a function $H:\mathbb{R}\times {\mathbb{R}}^P\to \mathbb{R}$, thereby allowing for more subtle interactions between the energy of the core model F and the sufficient statistics $\mathit{G}$ according to $H\left(F\right(\mathit{y}),\mathit{G}(\mathit{y}\left)\right)$. The increased flexibility, however, must be further restricted in order for a corresponding solution to be optimal. We have found that it is sufficient to impose a requirement on H to be non-decreasing in the first argument. Note that for the special case of the objective in [69], this requirement automatically holds.

Furthermore, the previous works could only guarantee polynomial run-time in cases where (a) the core model can be represented by a tree-shaped factor graph, and (b) the maximal degree of a variable node in the factor graph is bounded by a constant. The former excludes problems with cyclic dependencies and the latter rejects graphs with star-like shapes. The corresponding idea for clique trees can handle cycles but suffers from a similar restriction on the maximal node degree being bounded. Here, we solve this problem by applying the graph transformation proposed in Section 3.2, effectively reducing the maximal node degree to $\nu =3$. We note that a similar idea can be applied to factor graphs by replicating variable nodes and introducing constant factors. As a result, we improve the guarantee on the computational complexity by reducing the potentially unbounded parameter $\nu$ in the upper bound $O(M·N^{\tau +1}·R^{\nu -1})$ to $\nu ⩽3$.

In future work, we will consider applying our framework to the explanation task based on the technique of layer-wise relevance propagation [89] in graph neural networks, following in the spirit of [90,91].

9. Conclusions

Despite the high diversity in the range of existing applications, a considerable number of the underlying MAP problems share the unifying property that the information on the global variable interactions imposed by either a global factor or a global constraint can be locally propagated through the network by means of dynamic programming. By extending previous works, we presented a theoretical framework for efficient exact inference on models involving global factors with decomposable internal structures. At the heart of the presented framework is a constrained message-passing algorithm that always finds an optimal solution for our target problem in polynomial time. In particular, the performance of our approach does not explicitly depend on the graph form but rather on intrinsic properties such as the treewidth and the number of states of the auxiliary variables defined by the sufficient statistics of global interactions. The overall computational procedure is provably exact, and it has lower asymptotic bounds on the computational time complexity compared to previous works.