Attacking Tropical Stickel Protocol by MILP and Heuristic Optimization Techniques

Abstract

Known attacks on the tropical implementation of Stickel protocol involve finding minimal covers for a certain covering problem, and this leads to an exponential growth in the worst case time required to recover the secret key as the used polynomial degree increases. The computational inefficiency of this attack is also observed in practice, unless the number of explored covers is limited, on the expense of the success rate of the attack. Consequently, it can be argued that Alice and Bob can still repel these attacks on tropical Stickel protocol by utilizing very high polynomial degrees, a feasible approach due to the efficiency of tropical operations. The same is true for the implementation of Stickel protocol over some other semirings with idempotent addition (such as the max–min or digital semiring). In this paper, we propose alternative methods to attack the Stickel protocols that avoid solving the covering problem. These methods involve framing the attacks as a mixed integer linear programming (MILP) problem or applying certain heuristic global optimization techniques. We also include a number of numerical experiments to analyze the success rate and the time required to execute the suggested attacks in practice.

1. Introduction

A key exchange protocol is a process where two parties, commonly referred to as Alice and Bob, collaboratively generate a shared secret key using public information and messages exchanged over a public channel. The security of a protocol is determined by its ability to prevent an attacker from easily recovering the shared secret key using these public information and intercepted messages, typically by ensuring that the attacker must solve a problem that is computationally hard to succeed in practice. $NP$-hard problems or problems with exponential worst case complexity are natural candidates for these (although $NP$-hardness or exponential worst case complexity are not enough to guarantee the security of protocols). Such protocols often rely on various algebraic tools to achieve the desired security properties.

Polynomials over the tropical (max-plus) semiring are one of the recent tools utilized in key exchange protocols, appearing in the tropical implementation of the Stickel protocol proposed by Grigoriev and Shpilrain [1]. This new implementation followed Shpilrain’s successful attack [2] on the initial Stickel protocol [3] and has become one of the most popular key exchange protocols utilizing tropical operations. The rationale behind suggesting a tropical implementation of the protocol was to avoid obvious attacks involving linear algebra and matrix inverses, which were effective against the original protocol. The Stickel protocol can be similarly implemented over any semiring, and its implementation over max–min and max-T semirings (where the symbol T stands for arbitrary T-norm [4]) is analyzed in [5]. The survey in [6] argues for broadening semiring choices beyond the tropical semiring and reviews the main hard problems in semiring-based cryptography.

Kotov and Ushakov [7] later suggested an attack on the tropical Stickel protocol by transforming the underlying problem into finding a special solution to the protocol’s associated system of equations of the form $A\otimes x=b$, the complete solution set to which can be described using solution to a certain covering problem. The attacker still faces a significant challenge: solving the problem to find a solution to the covering problem that satisfies certain conditions. To find such a cover, the attacker potentially needs to check all the minimal covers and find a cover that actually produces the required special solution to $A\otimes x=b$. Therefore, this approach is less effective when Alice and Bob use high-degree polynomials, which can be efficiently managed by Alice and Bob with minimal computational resources due to the efficient nature of tropical operations. An analogue of the Kotov–Ushakov attack against the max–min and, more generally, max-T implementations of the Stickel protocol can be similarly proposed [5]. However, it encounters a similar challenge of finding a minimal solution with special properties, resulting in an exponential increase in the worst case execution time.

The main idea of this paper is to introduce alternative attack strategies that avoid solving the covering problem encountered in a conventional Kotov–Ushakov attack. Specifically, we propose an attack where we instead find a solution x that minimizes the protocol’s associated objective function ${\sum }_i{({(A\otimes x)}_i-b_i)}^2$ using a heuristic optimization technique. We will compare this with a different approach where some of the known attacks are formulated as mixed integer linear programs, allowing the shared key to be recovered using an MILP solver.

This paper is organized as follows: Section 2 covers preliminaries and basic definitions, particularly those related to the matrix algebra over the tropical and max–min semirings, as well as the targeted key exchange protocols based on these semirings. In Section 3, we present our alternative attacks, provide numerical implementations demonstrating their performance, and compare them with a typical Kotov–Ushakov attack. In Section 4 and Section 5, we discuss how these proposed attacks can also target a recent implementation of Stickel protocol over a newly introduced semiring known as the “digital semiring” [8] and a recently proposed tropical digital signature protocol [9], respectively. Section 6 is dedicated to conclusions and discussion. Our code implementations have been made available on GitHub: https://github.com/suliman1n/Attacking-Tropical-Stickel-Protocol-by-MILP-and-Heuristic-Optimization-Techniques (accessed on 12 September 2025) and were developed using MATLAB R2023b.

2. Preliminaries

In this section, we are going to introduce the matrix algebra over the tropical and max–min semirings, followed by the Stickel protocol over these semirings and two versions of the Kotov–Ushakov attack. Note that we use the standard notation $\left[m\right]=\{1,\dots ,m\}$ and $\left[n\right]=\{1,\dots ,n\}$ for most common index sets.

Definition 1

(Matrix Algebra over Semirings [10]). We define the tropical semiring as ${\mathbb{R}}_{\max }=(\mathbb{R}\cup \{-\infty \},\oplus ,\otimes ),$ and the max–min semiring as ${\mathbb{R}}_{\max ,\min }=(\mathbb{R}\cup \{-\infty \}\cup \{\infty \},\oplus ,\otimes ),$ where the arithmetical operations are defined by $x\oplus y=\max (x,y)$ and $x\otimes y=x+y$ for all $x,y\in {\mathbb{R}}_{\max }$ in the tropical case, and by $x\oplus y=\max (x,y)$ and $x\otimes y=\min (x,y)$ for all $x,y\in {\mathbb{R}}_{\max ,\min }$ for the max-min case. When addressing both semirings at the same time or any semiring in general, we will use the symbol ${\mathbb{R}}_T$ (also reminiscent of max-T semirings, of which the max–min semiring and the non-positive part of the tropical semiring are special cases).

The arithmetic operations over any semiring are naturally extended to include matrices and vectors. In particular, the operation $A\otimes \alpha =\alpha \otimes A,$ where $\alpha \in {\mathbb{R}}_T,A\in {\mathbb{R}}_T^{m\times n}$ and ${\left(A\right)}_{ij}=a_{ij}$ for $i\in \left[m\right]$ and $j\in \left[n\right]$, is defined by

$${(A\otimes \alpha )}_{ij}={(\alpha \otimes A)}_{ij}=\alpha \otimes a_{ij}\forall i\in \left[m\right]\mathrm{and}\forall j\in \left[n\right].$$

The matrix addition $A\oplus B$ of two matrices $A\in {\mathbb{R}}_T^{m\times n}$ and $B\in {\mathbb{R}}_T^{m\times n}$, where ${\left(A\right)}_{ij}=a_{ij}$ and ${\left(B\right)}_{ij}=b_{ij}$ for $i\in \left[m\right]$ and $j\in \left[n\right]$, is defined by

$${(A\oplus B)}_{ij}=a_{ij}\oplus b_{ij}\forall i\in \left[m\right]\mathit{and}\forall j\in \left[n\right].$$

The matrix multiplication of two matrices is also similar to the “traditional” algebra. Namely, we define $A\otimes B$ for two matrices, where $A\in {\mathbb{R}}_T^{m\times p}$ and $B\in {\mathbb{R}}_T^{p\times n}$, as follows:

$${(A\otimes B)}_{ij}=\underset{k=1}{\overset{p}{⨁}}{a}_{ik}\otimes b_{kj}=\left(a_{i1}\otimes b_{1j}\oplus a_{i2}\otimes b_{2j}\oplus \dots \oplus a_{ip}\otimes b_{pj}\right)\forall i\in \left[m\right]\mathit{and}\forall j\in \left[n\right].$$

The arithmetics of the max-plus and max-min semirings are summarized in

Table 1. Summary of semiring operations.

Semiring	Ground Set	⊕	⊗	Zero Element	Identity Element
Tropical	$\mathbb{R}\cup \{-\infty \}$	$\max (a,b)$	$a+b$	$-\infty$	0
Max–min	$\mathbb{R}\cup \{-\infty \}\cup \{+\infty \}$	$\max (a,b)$	$\min (a,b)$	$-\infty$	∞

below.

Note that, despite introducing this arithmetic, we will also quite often utilize the usual arithmetical operations to introduce concepts and explain arguments, mostly since the optimization methods that we are going to exploit are based on the usual arithmetic.

Definition 2

(Matrix Powers). For $M\in {\mathbb{R}}_T^{n\times n}$, the n-th power of M is denoted by $M^{\otimes n}$, and is equal to

$$M^{\otimes n}=\underset{n\mathit{times}}{\underbrace{M\otimes M\otimes \dots \otimes M}}$$

By definition, any square matrix to the power 0 is the identity.

Definition 3

(Identity Matrix). The identity matrix $I\in {\mathbb{R}}_T^{n\times n}$ is of the form ${\left(I\right)}_{ij}={\delta }_{ij}$ where

$${\delta }_{ij}=\left\{\begin{array}{cc}0\mathit{for}\mathit{tropical}\mathit{case},\mathit{or}\infty \mathit{for}\mathit{max}-\mathit{min}\mathit{case}\hfill & \mathit{if}i=j\hfill \\ -\infty \hfill & \mathit{otherwise}\hfill \end{array}\right.$$

Note that the identity matrix can be defined also for a general semiring: one sets the diagonal entries equal to the semiring unity and the off-diagonal entries to the semiring zero [10].

Subsequently, we define the matrix polynomials.

Definition 4

(Matrix Polynomials). Matrix polynomial is a function of the form

$$A\mapsto p\left(A\right)=\underset{k=0}{\overset{d}{⨁}}{a}_k\otimes A^{\otimes k}.$$

where $a_k\in {\mathbb{R}}_T$ for $k=0,1,\dots ,d$. Here, $A\in {\mathbb{R}}_T^{n\times n}$ is a square matrix of any dimension n.

Any two matrix polynomials of the same matrix over any semiring commute just like in the classical algebra [10], and this fact was utilized by Grigoriev and Shpilrain [1] to construct a tropical implementation of the Stickel protocol (Protocol 1). Quite obviously, this protocol can be implemented over any semiring (and in particular, over the max–min semiring).

Protocol 1

(Stickel Protocol over Semirings).

• Alice and Bob agree on public matrices $A,B,W\in {\mathbb{R}}_T^{n\times n}$.
• Alice chooses two random tropical polynomials, $p_1\left(x\right)$ and $p_2\left(x\right)$, and sends $U=p_1\left(A\right)\otimes W\otimes p_2\left(B\right)$ to Bob.
• Bob chooses two random tropical polynomials, $q_1\left(x\right)$ and $q_2\left(x\right)$, and sends $V=q_1\left(A\right)\otimes W\otimes q_2\left(B\right)$ to Alice.
• Alice computes her secret key using a public key V obtained from Bob, which is $K_a=p_1\left(A\right)\otimes V\otimes p_2\left(B\right)$.
• Bob also computes his secret key using Alice’s public key U, which is $K_b=q_1\left(A\right)\otimes U\otimes q_2\left(B\right)$.

The two parties end up with an identical key in both protocols due to the commutativity of polynomials of the same matrix. Formally, we have $K_a=p_1\left(A\right)\otimes V\otimes p_2\left(B\right)$ = $p_1\left(A\right)\otimes q_1\left(A\right)\otimes W\otimes q_2\left(B\right)\otimes p_2\left(B\right)$ = $q_1\left(A\right)\otimes p_1\left(A\right)\otimes W\otimes p_2\left(B\right)\otimes q_2\left(B\right)$ = $q_1\left(A\right)\otimes U\otimes q_2\left(B\right)=K_b$.

An attack against Protocol 1 over the tropical semiring was published by Kotov and Ushakov [7], and an analog of this attack against Protocol 1 over max–min semiring (and, more generally, max-T semiring with continuous T-norm) was discussed in [5]. In the next section, we will compare their performance with the optimization methods proposed in the present paper. The presented attacks break Protocol 1 by solving the following problem

Problem 1.

Given the public matrices U and W where $U=p_1\left(A\right)\otimes W\otimes p_2\left(B\right)$ for some unknown $p_1\left(A\right)$ and $p_2\left(B\right)$, find $p_1^{\prime }\left(A\right)$ and $p_2^{\prime }\left(B\right)$ such that $U=p_1^{\prime }\left(A\right)\otimes W\otimes p_2^{\prime }\left(B\right)$.

or the following problem for the attack presented in Section 3.3

Problem 2.

Given the public matrices U and W where $U=p_1\left(A\right)\otimes W\otimes p_2\left(B\right)$ for some unknown $p_1\left(A\right)$ and $p_2\left(B\right)$, find X and Y such that X commutes with A, Y commutes with B, and $X\otimes W\otimes Y=U$.

Solving these problems is sufficient but not necessary to compromise the protocol. For example, the attack presented in [11] offers a more efficient approach against this particular version of the protocol. This attack applies under the two conditions detailed in [12]. Consequently, variants of the Stickel protocol that employ broader classes of commuting matrices (beyond polynomials of some public matrices) or alternative semirings, other than the tropical semiring, may not be vulnerable to this attack. Currently, no such variant of the tropical Stickel protocol is known to us. In such generalized settings, solving the aforementioned problems may be the only viable approach for attacking the Stickel protocol, which is why we still consider them relevant. In [13], the authors proposed a non-Stickel-type protocol based on a tropical multiple-exponentiation problem and showed that known attacks do not directly apply to this construction.

We now turn to the specific goal of the upcoming attacks. The objectives of the attacks is to find the polynomial coefficients $x_{\alpha },y_{\beta }$ for all $\alpha ,\beta \in \{0,\dots D\}$ where D is the maximum polynomial degree used in the protocols, and hence construct $X={⨁}_{\alpha =0}^D\left(x_{\alpha }\otimes A^{\otimes \alpha }\right)$ and $Y={⨁}_{\beta =0}^D\left(y_{\beta }\otimes B^{\otimes \beta }\right)$ that satisfy $X\otimes W\otimes Y=U$. Thus, the attacks aim to recover the shared secret key, by turning $X\otimes W\otimes Y=U$ into the form of a system of linear equations of the shape $A\otimes x=b$ and search for a solution that satisfies a special structure among all possible solutions. Thus, these attacks encounter the problem of finding all minimal solutions of a linear system of the shape $A\otimes x=b$, which is easy to solve when Alice and Bob use low-degree polynomials, as demonstrated numerically in [7,14,15] for the tropical case, or in [5] for the max–min case. However, it becomes significantly more challenging for higher-degree polynomials due to the exponential increase in the number of the minimal solutions of the system. The full details of the Kotov–Ushakov attack are described below.

We are aiming to find two matrices X and Y, where they are expressed as

$$\begin{array}{cc}\hfill X& =\underset{\alpha =0}{\overset{D}{⨁}}\left(x_{\alpha }\otimes A^{\otimes \alpha }\right)\hfill \\ \hfill Y& =\underset{\beta =0}{\overset{D}{⨁}}\left(y_{\beta }\otimes B^{\otimes \beta }\right),\hfill \end{array}$$

such that D is sufficiently large to exceed the maximal degree of any polynomial that Alice and Bob might use. Then, we substitute these expressions into $X\otimes W\otimes Y=U$ to obtain

$$U=\underset{\alpha =0}{\overset{D}{⨁}}\left(x_{\alpha }\otimes A^{\otimes \alpha }\right)\otimes W\otimes \underset{\beta =0}{\overset{D}{⨁}}\left(y_{\beta }\otimes B^{\otimes \beta }\right).$$

Combining the summations, we obtain

$$U=\underset{\alpha ,\beta =0}{\overset{D}{⨁}}\left(x_{\alpha }\otimes A^{\otimes \alpha }\right)\otimes W\otimes \left(y_{\beta }\otimes B^{\otimes \beta }\right).$$

Rearranging those using the distributivity law will give

$$\underset{\alpha ,\beta =0}{\overset{D}{⨁}}{x}_{\alpha }\otimes y_{\beta }\otimes \left(A^{\otimes \alpha }\otimes W\otimes B^{\otimes \beta }\right)=U.$$

We then denote $R^{\alpha \beta }=A^{\otimes \alpha }\otimes W\otimes B^{\otimes \beta }$ and therefore we can write

$$\underset{\alpha ,\beta =0}{\overset{D}{⨁}}{x}_{\alpha }\otimes y_{\beta }\otimes {\left(R^{\alpha \beta }\right)}_{\gamma \delta }=U_{\gamma \delta }\forall \gamma ,\delta \in \left[n\right]\times \left[n\right].$$

(1)

If we additionally denote $z_{\alpha \beta }=x_{\alpha }\otimes y_{\beta }$, we have

$$\underset{\alpha ,\beta =0}{\overset{D}{⨁}}{z}_{\alpha \beta }\otimes {\left(R^{\alpha \beta }\right)}_{\gamma \delta }=U_{\gamma \delta }\forall \gamma ,\delta \in \left[n\right]\times \left[n\right].$$

(2)

We have arrived at a system of linear equations of the shape $A\otimes x=b$ with coefficients ${\left(R^{\alpha \beta }\right)}_{\gamma \delta }$ and unknowns $z_{\alpha \beta }$.

We now need to scan all solutions to this system, and obtain the solution that satisfies $z_{\alpha \beta }=x_{\alpha }\otimes y_{\beta }$ for some $x_{\alpha },y_{\beta }\in \mathbb{N}\forall \alpha ,\beta \in \{0,1,\dots ,D\}$. Thus, using the theory of $A\otimes x=b$ solvability, we need to find the greatest solution, as well as all minimal solutions. For each minimal solution, we need to search for a vector $\left(z_{\alpha \beta }\right)$ in the interval between the minimal solution and the greatest solution that solves $z_{\alpha \beta }=x_{\alpha }\otimes y_{\beta }$ for some $x_{\alpha },y_{\beta }$.

Note that, for the tropical case, a minimal solution can be found by finding a minimal cover (i.e., the minimal number of variables that satisfy all the equations in the system), and the other variables are set to $-\infty$. The following algorithm captures this process.

For the max–min case, we similarly need to compute the greatest solution c (using Lemma 3.2 in [16]) and all minimal solutions $d^{\left(i\right)}$’s (using Section 3.3 in [17] or Chapter 3 in [18]), and search for the required solution. The following algorithm captures this process.

Note that system (4) can be transformed into a problem of mixed-integer linear programming as shown in [5].

These attacks succeed under the condition that the attacker is using D that exceeds the greatest polynomial degrees used by Alice and Bob, because, in this case, these attacks produce X and Y that satisfy $X\otimes W\otimes Y=U$. The proof can be found in [5,15]. However, they exhibit exponential growth in computational time relative to the used polynomial degree in the protocol. Numerical experiments showing the time taken by these attacks to compromise the tropical implementation of Protocol 1 can be found in [7,14,15], and for the max–min implementation, see [5].

Table 2. Comparison of runtimes for Algorithm 1 (tropical) and Algorithm 2 (max–min).

Polynomial Degree D	Algorithm 1 Time (s)	Algorithm 2 Time (s)
3	<$0.01$	0.04
5	<$0.01$	2.9
9	<$0.01$	12,204
30	223	N/A
50	2640	N/A

summarizes a representative subset of these runtime results.

The most computationally intensive component of the attacks described above is the enumeration of all minimal covers. This problem is fundamentally equivalent to the hypergraph traversal hitting sets enumeration, a widely studied topic in various fields such as combinatorics and optimization, To formalize this connection in the tropical case (Algorithm 1), we firstly present some relevant definitions.

Definition 5

(Hypergraph). A hypergraph $H=(V,\mathcal{E})$ consists of a vertex set V and a set of hyperedges $\mathcal{E}$, where each hyperedge $E\in \mathcal{E}$ is a subset of V.

Definition 6

(Hitting set (e.g., ref. [19])). A hitting set for a hypergraph $H=(V,\mathcal{E})$ is a subset $K\subseteq V$ such that $K\cap E\ne \varnothing$ for every $E\in \mathcal{E}$. A hitting set is minimal if no proper subset of K is a hitting set.

The enumeration process of all minimal covers of $\left[n\right]\times \left[n\right]$ by the computed sets $S_{\alpha \beta }$ in Algorithm 1 is equivalent to the process of enumerating all minimal hitting sets of the hypergraph $H=\left(\{0,\dots ,D\}\times \{0,\dots ,D\},\{G_{11},G_{12},\dots ,G_{nn}\}\right)$ where $G_{\gamma \delta }=\{(\alpha ,\beta )\in \{0,\dots ,D\}\times \{0,\dots ,D\}:{⨁}_{\alpha ,\beta =0}^D{c}_{\alpha \beta }\otimes {\left(R^{\alpha \beta }\right)}_{\gamma \delta }=U_{\gamma \delta }\}$. This is because we know that a minimal cover $\mathcal{C}\subseteq \{0,\dots ,D\}\times \{0,\dots ,D\}$ in Algorithm  1 satisfies ${\bigcup }_{(\alpha ,\beta )\in \mathcal{C}}{S}_{\alpha \beta }=\left[n\right]\times \left[n\right]$. This is equivalent to $\mathcal{C}$ intersecting every hyperedge $G_{\gamma \delta }$ (i.e., $\mathcal{C}$ is a hitting set for H). Minimality of $\mathcal{C}$ as a hitting set then follows since removing any $(\alpha ,\beta )\in \mathcal{C}$ would leave some $G_{\gamma \delta }$ unhit. Similarly, given $H=\left(\{0,\dots ,D\}\times \{0,\dots ,D\},\{G_{11},G_{12},\dots ,G_{nn}\}\right)$, we know that a minimal hitting set $\mathcal{C}\subseteq \{0,\dots ,D\}\times \{0,\dots ,D\}$ intersects every $G_{\gamma \delta }$. By defining $S_{\alpha \beta }=\{(\gamma ,\delta )\in \left[n\right]\times \left[n\right]:(\alpha ,\beta )\in G_{\gamma \delta }\}$, the union ${\bigcup }_{(\alpha ,\beta )\in \mathcal{C}}{S}_{\alpha \beta }$ covers $\left[n\right]\times \left[n\right]$. Minimality of $\mathcal{C}$ as a cover follows since no smaller subset of $\mathcal{C}$ can cover $\left[n\right]\times \left[n\right]$. This means there is a one-to-one correspondence between the enumerated minimal covers in Algorithm 1 and the minimal hitting sets of H.

Algorithm 1 Tropical Kotov–Ushakov attack [7]

Input: Public matrices $A,B,W$, transmitted message U, maximum polynomial degree D Output: Coefficients $x_{\alpha },y_{\beta }$. 1: Compute $$\begin{array}{c}{c}_{\alpha \beta }=\underset{\gamma ,\delta \in \left[n\right]}{\min }\left(U_{\gamma \delta }-R_{\gamma \delta }^{\alpha \beta }\right)\\ S_{\alpha \beta }=arg\underset{\gamma ,\delta \in \left[n\right]}{\min }\left(U_{\gamma \delta }-R_{\gamma \delta }^{\alpha \beta }\right).\end{array}$$ 2: Among all minimal covers of $\left[n\right]\times \left[n\right]$ by $S_{\alpha \beta }$, that is, all minimal subsets $\mathcal{C}\subseteq \{0,\dots ,D\}\times \{0,\dots ,D\}$ such that $$\bigcup _{(\alpha ,\beta )\in \mathcal{C}}{S}_{\alpha \beta }=\left[n\right]\times \left[n\right],$$ find a cover for which the system $$\begin{array}{c}{x}_{\alpha }+y_{\beta }=c_{\alpha \beta },\mathrm{if}(\alpha ,\beta )\in \mathcal{C},\\ x_{\alpha }+y_{\beta }⩽c_{\alpha \beta },\mathrm{if}\mathrm{otherwise}.\end{array}$$ (3) is solvable. 3: return $(x_{\alpha },y_{\beta })$.

Algorithm 2 Max–min Kotov–Ushakov attack [5]

Input: Public matrices $A,B,W$, transmitted message U, maximum polynomial degree D. Output: Coefficients $x_{\alpha },y_{\beta }$. 1: Compute the maximum solution c of system (2) as $$\begin{array}{cc}\hfill c_{\alpha \beta }& =\underset{\gamma ,\delta \in \left[n\right]}{\min }\left(U_{\gamma \delta }:R_{\gamma \delta }^{\alpha \beta }>U_{\gamma \delta }\right)\forall \alpha ,\beta \in \{0,\dots ,D\}\hfill \end{array}$$ 2: Compute all minimal solutions $d^{\left(i\right)}$ of system (2). 3: Find a minimal solution $d^{\left(i\right)}$ with components $d_{\alpha \beta }^{\left(i\right)}$ for which the system $$\begin{array}{c}\hfill d_{\alpha \beta }^{\left(i\right)}\le x_{\alpha }\otimes y_{\beta }\le c_{\alpha \beta }\forall \alpha ,\beta \in \{0,\dots ,D\}\end{array}$$ (4) is solvable. 4: return $(x_{\alpha },y_{\beta })$.

From this perspective, we know that a hypergraph can have exponentially many minimal hitting sets, so a polynomial-time algorithm for the enumeration process in the above attacks is not possible, but it can be achieved in incremental quasi-polynomial time [19]. This also implies the exponential worst case complexity of the Kotov–Ushakov attacks (Algorithms 1 and 2). Another closely related problem is finding the smallest hitting set, which is known to be NP-hard [19], although the Kotov–Ushakov attacks are not aimed precisely at this problem. Nevertheless, their exponential worst-case complexity presents a major drawback. To address this, we next study the application of some well-known optimization techniques.

3. Attacks Using Optimization

In this section, we explore more efficient approaches to attacking the tropical and max-min implementations of Protocol 1 that avoid the minimal covering problem and the associated exponential complexity, which are evident in Algorithms 1 and 2. For all experiments, we use a matrix dimension of 10, which is the default parameter suggested in [1,7]. This choice allows us to compare the performance of the optimization methods discussed in this paper with the performance of Algorithms 1 and 2. To read this section, the basic knowledge of semiring algebra (see Definitions 1–4) as well as the knowledge of above mentioned Problems and Protocol 1 will be required from the reader. See also Table 1 for a summary of tropical and max–min arithmetics.

3.1. Simulated Annealing

Both Algorithms 1 and 2 aim to find all minimal solutions that satisfy all equations in system (2). In this approach, we aim to find a solution that minimizes the Euclidean distance between the left hand side and the right hand side of the system. Formally, we solve the following:

$$\underset{x_{\alpha },y_{\beta }}{\min }\sum _{(\gamma ,\delta )\in \left[n\right]\times \left[n\right]}{f}_{\gamma \delta }^2$$

where

$$f_{\gamma \delta }=\underset{\begin{array}{c}\alpha \in \{0,1,\dots ,D\}\\ \beta \in \{0,1,\dots ,D\}\end{array}}{\max }(x_{\alpha }\otimes y_{\beta }\otimes R_{\gamma \delta }^{\alpha \beta })-U_{\gamma \delta }$$

(5)

This objective function is complex with numerous local minima. However, the simulated annealing algorithm (see, e.g., ref. [20]), when initialized with a sufficiently high temperature parameter, effectively navigates these local minima and converges to the global minimum, where the objective function equals zero. The tropical and max–min objective functions are defined, respectively, in (6) and (7).

We now formally outline how the tropical Stickel protocol (Protocol 1 with ${\mathbb{R}}_T={\mathbb{R}}_{\max }$) is attacked using the simulated annealing method; see Algorithm 3.

To ensure the simulated annealing algorithm escapes local minima, the initial temperature has to be sufficiently large to allow the acceptance of worse points. A practical method for determining this initial temperature is to set it based on the sample variance of multiple randomly evaluated points (e.g., ref. [21]). This captures the variability of the objective function, reducing the risk of getting stuck in local minima.

The performance of simulated annealing is also highly sensitive to the initial point. An optimal initial point can facilitate a quicker convergence to the global minimum. However, in our implementation, we started with a random point, as it seems the high initial temperature helps to mitigate the potential drawback of this non-optimal initialization.

Furthermore, as Alice and Bob increase the range of entries for public matrices and polynomial coefficients, the objective function becomes more complex. Kotov–Ushakov attack (Algorithm 1) is not impacted by this, as it relies on solving a minimal covering problem that is independent of the individual entries (i.e., finding minimal covers using $S_{\alpha \beta }$’s, which are independent of the used entries). We will therefore also examine how Algorithm 3 performs under such conditions. Figure 1 shows the time taken in seconds to compromise Protocol 1 using Algorithm 3 for different degrees and entry ranges. All numerical experiments were executed on Windows 11 64-bit, with an Intel(R) Core(TM) i7-9750H CPU @ 2.60 GHz and 16.0 GB RAM.

Algorithm 3 Attacking tropical Stickel protocol using simulated annealing

Input: Public matrices $A,B,W$, transmitted message U, maximum polynomial degree D Output: Matrices $X,Y$. 1: Compute $T_{\alpha \beta }=A^{\otimes \alpha }\otimes W\otimes B^{\otimes \beta }-U$ for all $0\le \alpha ,\beta \le D$. 2: Define objective $$F(x,y)=\sum _{\gamma ,\delta }{\left(\underset{\alpha ,\beta }{\max }\left(x_{\alpha }+y_{\beta }+T_{\alpha \beta }^{\gamma \delta }\right)\right)}^2.$$ (6) 3: Initialize temperature T and choose a random starting point $(x^c,y^c)$. 4: repeat 5:    Set trial counter $k\leftarrow k+1$ (initialize $k\leftarrow 0$ before the loop). 6:    Update the temperature: $T_k\leftarrow T\times 0.{95}^k$. 7:    Select a new candidate point $(x^{test},y^{test})$ from the neighbourhood of $(x^c,y^c)$. 8:    Compute $\Delta \leftarrow F(x^{test},y^{test})-F(x^c,y^c)$. 9:    if ${\displaystyle exp\left(-\frac{\Delta }{T_k}\right)>Random[0,1)}$ then 10:    Accept the candidate: $(x^c,y^c)\leftarrow (x^{test},y^{test})$. 11: until  $F(x^c,y^c)=0$ 12: Let $(\overline{x},\overline{y})=(x^c,y^c)$. 13: Construct $$X=\underset{\alpha =0}{\overset{D}{⨁}}({\overline{x}}_{\alpha }\otimes A^{\otimes \alpha })\mathrm{and}Y=\underset{\beta =0}{\overset{D}{⨁}}({\overline{y}}_{\beta }\otimes B^{\otimes \beta }).$$ 14: return $(X,Y)$.

This attack achieved a perfect success rate and is significantly faster than Algorithm 1, averaging about 30 times the speed for a polynomial degree of 50 (refer to [14] for detailed experimental results of Algorithm 1). Note that the attack still performs well for higher entry ranges, but it is more likely that we encounter some samples that take significantly longer than average to converge. This is probably caused by the increased complexity of the objective function and how optimal the probabilistic selection of the next neighboring point in the simulated annealing algorithm is, as well as the number of iterations performed until convergence.

For the max–min implementation of Protocol 1, the simulated annealing algorithm often struggles to reach the zero of the objective function, frequently getting stuck in local minima. Therefore, we have to utilizes the lowest local minimum obtained to attempt to recover the secret key; see Step 4 in Algorithm 4.

Algorithm 4 Attacking max–min Stickel protocol using simulated annealing

Input: Public matrices $A,B,W$, transmitted message U, maximum polynomial degree D Output: Matrices $X,Y$. 1: Compute $R_{\alpha \beta }=A^{\otimes \alpha }\otimes W\otimes B^{\otimes \beta }$ for all $0\le \alpha ,\beta \le D$. 2: Define objective $$F(x,y)=\sum _{\gamma ,\delta }{\left(\underset{\alpha ,\beta }{\max }(x_{\alpha }+y_{\beta }+R_{\alpha \beta }^{\gamma \delta }-U_{\gamma \delta })\right)}^2.$$ (7) 3: Initialize temperature T and choose a random starting point $(x^c,y^c)$. 4: repeat 5:    Set trial counter $k\leftarrow k+1$ (initialize $k\leftarrow 0$ before the loop). 6:    Update the temperature: $T_k\leftarrow T\times 0.{95}^k$. 7:    Select a new candidate point $(x^{test},y^{test})$ from the neighbourhood of $(x^c,y^c)$. 8:    Compute $\Delta \leftarrow F(x^{test},y^{test})-F(x^c,y^c)$. 9:    if ${\displaystyle exp\left(-\frac{\Delta }{T_k}\right)>Random[0,1)}$ then 10:    Accept the candidate: $(x^c,y^c)\leftarrow (x^{test},y^{test})$. 11: until $F(x^c,y^c)$ does not change after N loops 12: Let $(\overline{x},\overline{y})=(x^c,y^c)$. 13: Construct $$X=\underset{\alpha =0}{\overset{D}{⨁}}({\overline{x}}_{\alpha }\otimes A^{\otimes \alpha })\mathrm{and}Y=\underset{\beta =0}{\overset{D}{⨁}}({\overline{y}}_{\beta }\otimes B^{\otimes \beta }).$$ 14: return $(X,Y)$.

In the experiments, we set $N=300$. Although this attack does not achieve a perfect success rate, it frequently recovers the majority of the entries of the secret key. The average number of recovered entries and the average execution time are respectively illustrated in Figure 2 and Figure 3. The flowchart of the attacks based on the simulated annealing is shown in Figure 4.

Note that this attack is significantly faster than Algorithm 2 (for detailed experimental results of Algorithm 2, refer to [5]). However, as shown experimentally, it does not guarantee the successful recovery of the entire secret key. Furthermore, the algorithm maintains consistent performance with higher entry ranges, largely due to the appropriate adjustment of the initial temperature.

3.2. Kotov-Ushakov Attack Using MILP Solver

We now propose an attack that recovers the secret key by solving a mixed integer linear program (MILP), following an observation by [22]. Specifically, we start by transforming system (1) in the Kotov–Ushakov attack into a linear system by converting the disjunctive constraints into linear constraints by using Boolean variables and a big parameter. This approach allows us to avoid dealing with system (2) and the associated challenge of enumerating all minimal solutions. Then we solve this system of inequalities using the Gurobi solver [23] (but we could use any other available MILP solver instead) employing the default parameters of this solver. The tropical and max-–min encodings of system (1) are presented in (8)–(10), respectively. See Algorithms 5 and 6 for a detailed description.

Algorithm 5 Kotov–Ushakov MILP attack on tropical Stickel protocol

Input: Public matrices $A,B,W$, transmitted message U, maximum polynomial degree D Output: Matrices $X,Y$. 1: Compute $T_{\alpha \beta }=A^{\otimes \alpha }\otimes W\otimes B^{\otimes \beta }-U$ for all $0\le \alpha ,\beta \le D$. 2: Find $x,y$ and z that satisfy the following system where M is a big enough number, $\alpha$ and $\beta$ range from 0 to D, and $\gamma$ and $\delta$ range from 1 to n: $$\begin{array}{l}{x}_{\alpha }+y_{\beta }+T_{\gamma \delta }^{\alpha \beta }\le 0\forall \alpha ,\beta ,\gamma ,\delta ,\hfill \\ x_{\alpha }+y_{\beta }+T_{\gamma \delta }^{\alpha \beta }+(1-z_{\alpha \beta \gamma \delta })M\ge 0\forall \alpha ,\beta ,\gamma ,\delta ,\\ z_{\alpha \beta \gamma \delta }\in \{0,1\}\forall \alpha ,\beta ,\gamma ,\delta ,\\ \sum _{(\alpha ,\beta )}{z}_{\alpha \beta \gamma \delta }=1\forall \gamma ,\delta .\end{array}$$ (8) 3: Using these x and y construct $$X=\underset{\alpha =0}{\overset{D}{⨁}}(x_{\alpha }\otimes A^{\otimes \alpha })\mathrm{and}Y=\underset{\beta =0}{\overset{D}{⨁}}(y_{\beta }\otimes B^{\otimes \beta }).$$ 4: return $(X,Y)$.

The parameter M acts as a tunable variable whose value can be adjusted to ensure the correct and efficient solution of the MILP. In practice, we used a value of M that exceeded 1000 multiplied by the biggest possible entry of $A,B$, and W. Note that the number of variables in system (8) increases both with the matrix dimension and the polynomial degree used in the protocol. Specifically, the number of variables would be $2(D+1)+n^2{(D+1)}^2$. Also, the number of equations in this system is $2n^2{(D+1)}^2+n^2$. Figure 5 illustrates the time taken by Algorithm 5 when applied to the tropical Stickel protocol.

The attack on the max-min version of Protocol 1 can be similarly described: see Algorithm 6.

Note that the number of variables in this system similarly increases with both the matrix dimension and the polynomial degree used in the protocol. Specifically, the number of variables is $2(D+1)+n^2{(D+1)}^2+3n^2{(D+1)}^2$. Also, the number of equations in this system is $7n^2{(D+1)}^2+n^2$. The time taken by Algorithm 6 when applied to the max–min Stickel protocol is illustrated in Figure 6.

We observe that the computational time required for this approach is worse than that of the tropical case (Figure 5).

Therefore, both Algorithms 5 and 6 require significantly more time even for lower polynomial degrees compared to the tropical and max–min Kotov–Ushakov attacks (Algorithms 1 and 2). This is likely due to the high number of variables involved in the linear system. Consequently, these attacks do not provide any significant advantage over the previously described Kotov–Ushakov attacks.

Algorithm 6 Kotov–Ushakov MILP attack on max–min Stickel protocol

Input: Public matrices $A,B,W$, transmitted message U, maximum polynomial degree D Output: Matrices $X,Y$. 1: Compute $R_{\alpha \beta }=A^{\otimes \alpha }\otimes W\otimes B^{\otimes \beta }$ for all $0\le \alpha ,\beta \le D$. 2: Find $x,y$ and z that satisfy the following system where M is a big enough number, $\alpha$ and $\beta$ range from 0 to D, and $\gamma$ and $\delta$ range from 1 to n: $$\begin{array}{l}{x}_{\alpha }-(1-z_{\alpha \beta \gamma \delta }^{\left(1\right)})M\le U_{\gamma \delta }\\ y_{\beta }-(1-z_{\alpha \beta \gamma \delta }^{\left(2\right)})M\le U_{\gamma \delta }\\ R_{\gamma \delta }^{\alpha \beta }-(1-z_{\alpha \beta \gamma \delta }^{\left(3\right)})M\le U_{\gamma \delta }\\ z_{\alpha \beta \gamma \delta }^{\left(i\right)}\in \{0,1\}\mathrm{and}\sum _{i=1}^3{z}_{\alpha \beta \gamma \delta }^{\left(i\right)}=1\end{array}$$ (9) $$\begin{array}{l}{x}_{\alpha }+(1-z_{\alpha \beta \gamma \delta })M\ge U_{\gamma \delta }\\ y_{\beta }+(1-z_{\alpha \beta \gamma \delta })M\ge U_{\gamma \delta }\\ R_{\gamma \delta }^{\alpha \beta }+(1-z_{\alpha \beta \gamma \delta })M\ge U_{\gamma \delta }\\ z_{\alpha \beta \gamma \delta }\in \{0,1\}\mathrm{and}\sum _{(\alpha ,\beta )}{z}_{\alpha \beta \gamma \delta }=1\end{array}$$ (10) 3: Solve the MILP, and construct $$X=\underset{\alpha =0}{\overset{D}{⨁}}(x_{\alpha }\otimes A^{\otimes \alpha })\mathrm{and}Y=\underset{\beta =0}{\overset{D}{⨁}}(y_{\beta }\otimes B^{\otimes \beta }).$$ 4: return $(X,Y)$.

3.3. Shpilrain Attack Using MILP Solver

We now propose an alternative method to formulate the MILP to attack the tropical and max-min implementations of Protocol 1. Specifically, we introduce the tropical and max-min versions of the Shpilrain attack [2], where our objective is to find X and Y such that

$$\left\{\begin{array}{c}X\otimes A=T\hfill \\ A\otimes X=T\hfill \\ Y\otimes B=R\hfill \\ B\otimes Y=R\hfill \\ X\otimes W\otimes Y=U\hfill \end{array}\right.$$

(11)

where matrices T and R are composed of newly introduced auxiliary variables $t_{ij},r_{ij}$ for $(i,j)\in \left[n\right]\times \left[n\right]$. Then, the MILP can similarly be formulated by converting the disjunctive constraints into linear constraints with Boolean variables. In particular, for the first equation of (11), with $a_{ij}$ being the entries of A, we have

$$\underset{k\in \left[n\right]}{\max }(x_{ik}\otimes a_{kj})=t_{ij}\forall (i,j)\in \left[n\right]\times \left[n\right],$$

which can be represented as the following set of inequalities

$$x_{ik}\otimes a_{kj}\le t_{ij}\forall i,j,k\in \left[n\right],$$

and with M being a sufficiently large number

$$x_{ik}\otimes a_{kj}+(1-z_{kij})M\ge t_{ij}\forall i,j,k\in \left[n\right],$$

$$\sum _k{z}_{kij}=1,z_{kij}\in \{0,1\}\forall i,j,k\in \left[n\right].$$

The rest of inequalities can similarly be formulated using the other equations in (11), and then we solve the system using MILP solver. The tropical and max–min versions of the attack are described below in Algorithms 7 and 8. We observe that the number of variables in the system increases only with the matrix dimension, but not the polynomial degree used in the protocol. Specifically, for the tropical case, the number of variables in this system is $4n^2+4n^3+n^4$, and the number of equations is $5n^2+8n^3+2n^4$. For the max–min case, the number of variables is $4n^2+12n^3+4n^4$, and the number of equations is $5n^2+20n^3+7n^4$. The tropical and max–min encodings of system (11) are displayed in (12)–(16) and (17)–(21), respectively.

Algorithm 7 MILP Shpilrain attack on tropical Stickel protocol

Input: Public matrices $A,B,W$, transmitted message U Output: Matrices $X,Y$. 1: Represent (11) (over the tropical semiring) by the following system: $$\begin{array}{l}{x}_{ik}+a_{kj}\le t_{ij}\forall i,j,k\in \left[n\right],\\ x_{ik}+a_{kj}+(1-z_{1kij})M\ge t_{ij}\forall i,j,k\in \left[n\right],\\ z_{1kij}\in \{0,1\},\forall i,j,k\in \left[n\right],\\ \sum _k{z}_{1kij}=1\forall i,j\in \left[n\right],\end{array}$$ (12) $$\begin{array}{l}{a}_{ik}+x_{kj}\le t_{ij}\forall i,j,k\in \left[n\right],\\ a_{ik}+x_{kj}+(1-z_{2kij})M\ge t_{ij}\forall i,j,k\in \left[n\right],\\ z_{2kij}\in \{0,1\},\forall i,j,k\in \left[n\right],\\ \sum _k{z}_{2kij}=1\forall i,j\in \left[n\right],\end{array}$$ (13) $$\begin{array}{l}{y}_{ik}+b_{kj}\le r_{ij}\forall i,j,k\in \left[n\right],\\ y_{ik}+b_{kj}+(1-z_{3kij})M\ge r_{ij}\forall i,j,k\in \left[n\right],\\ z_{3kij}\in \{0,1\}\forall i,j,k\in \left[n\right],\\ \sum _k{z}_{3kij}=1\forall i,j\in \left[n\right],\end{array}$$ (14) $$\begin{array}{l}{b}_{ik}+y_{kj}\le r_{ij}\forall i,j,k\in \left[n\right],\\ b_{ik}+y_{kj}+(1-z_{4kij})M\ge r_{ij}\forall i,j,k\in \left[n\right],\\ z_{4kij}\in \{0,1\},i,j,k\in \left[n\right]\\ \sum _k{z}_{4kij}=1\forall i,j\in \left[n\right],\end{array}$$ (15) $$\begin{array}{l}{x}_{ik}+w_{kl}+y_{lj}\le u_{ij}\forall i,j,k,l\in \left[n\right],\\ x_{ik}+w_{kl}+y_{lj}+(1-z_{5klij})M\ge u_{ij}\forall i,j,k,l\in \left[n\right],\\ z_{5klij}\in \{0,1\},\\ \sum _{k,l}{z}_{5klij}=1\forall i,j\in \left[n\right],\end{array}$$ (16) where $a_{ij},b_{ij},w_{ij}$ are, respectively, the entries of the public matrices $A,B,W$, and $x_{ij},y_{ij}$ are the variables of the system. 2: Solve the MILP, and construct $X=\left(x_{ij}\right)$ and $Y=\left(y_{ij}\right)$. 3: return $(X,Y)$.

Algorithm 8 MILP Shpilrain attack on max–min Stickel protocol

Input: Public matrices $A,B,W$, transmitted message U Output: Matrices $X,Y$. 1: Represent (11) (over the max–min semiring) by the following system $$\begin{array}{l}{x}_{ik}-(1-z_{1kij}^{\left(1\right)})M\le t_{ij}\forall i,j,k\in \left[n\right],\\ a_{kj}-(1-z_{1kij}^{\left(2\right)})M\le t_{ij}\forall i,j,k\in \left[n\right],\\ z_{1kij}^{\left(1\right)}+z_{1kij}^{\left(2\right)}=1\forall i,j,k\in \left[n\right],\\ x_{ik}+(1-z_{1kij}^{\left(3\right)})M\ge t_{ij}\forall i,j,k\in \left[n\right],\\ a_{kj}+(1-z_{1kij}^{\left(3\right)})M\ge t_{ij}\forall i,j,k\in \left[n\right],\\ z_{1kij}^{\left(1\right)},z_{1kij}^{\left(2\right)},z_{1kij}^{\left(3\right)}\in \{0,1\}\forall i,j,k\in \left[n\right]\\ \sum _k{z}_{1kij}^{\left(3\right)}=1\forall i,j\in \left[n\right],\end{array}$$ (17) $$\begin{array}{l}{a}_{ik}-(1-z_{2kij}^{\left(1\right)})M\le t_{ij}\forall i,j,k\in \left[n\right],\\ x_{kj}-(1-z_{2kij}^{\left(2\right)})M\le t_{ij}\forall i,j,k\in \left[n\right],\\ z_{2kij}^{\left(1\right)}+z_{2kij}^{\left(2\right)}=1\forall i,j,k\in \left[n\right],\\ a_{ik}+(1-z_{2kij}^{\left(3\right)})M\ge t_{ij}\forall i,j,k\in \left[n\right],\\ x_{kj}+(1-z_{2kij}^{\left(3\right)})M\ge t_{ij}\forall i,j,k\in \left[n\right],\\ z_{2kij}^{\left(1\right)},z_{2kij}^{\left(2\right)},z_{2kij}^{\left(3\right)}\in \{0,1\}\forall i,j,k\in \left[n\right]\\ \sum _k{z}_{2kij}^{\left(3\right)}=1\forall i,j\in \left[n\right],\end{array}$$ (18) $$\begin{array}{l}{y}_{ik}-(1-z_{3kij}^{\left(1\right)})M\le r_{ij}\forall i,j,k\in \left[n\right],\\ b_{kj}-(1-z_{3kij}^{\left(2\right)})M\le r_{ij}\forall i,j,k\in \left[n\right],\\ z_{3kij}^{\left(1\right)}+z_{3kij}^{\left(2\right)}=1\forall i,j,k\in \left[n\right],\\ y_{ik}+(1-z_{3kij}^{\left(3\right)})M\ge r_{ij}\forall i,j,k\in \left[n\right],\\ b_{kj}+(1-z_{3kij}^{\left(3\right)})M\ge r_{ij}\forall i,j,k\in \left[n\right],\\ z_{3kij}^{\left(1\right)},z_{3kij}^{\left(2\right)},z_{3kij}^{\left(3\right)}\in \{0,1\}\forall i,j,k\in \left[n\right]\\ \sum _k{z}_{3kij}^{\left(3\right)}=1\forall i,j\in \left[n\right],\end{array}$$ (19) $$\begin{array}{l}{b}_{ik}-(1-z_{4kij}^{\left(1\right)})M\le r_{ij}\forall i,j,k\in \left[n\right],\\ y_{kj}-(1-z_{4kij}^{\left(2\right)})M\le r_{ij}\forall i,j,k\in \left[n\right],\\ z_{4kij}^{\left(1\right)}+z_{4kij}^{\left(2\right)}=1\forall i,j,k\in \left[n\right],\\ b_{ik}+(1-z_{4kij}^{\left(3\right)})M\ge r_{ij}\forall i,j,k\in \left[n\right],\\ y_{kj}+(1-z_{4kij}^{\left(3\right)})M\ge r_{ij}\forall i,j,k\in \left[n\right],\\ z_{4kij}^{\left(1\right)},z_{4kij}^{\left(2\right)},z_{4kij}^{\left(3\right)}\in \{0,1\}\forall i,j,k\in \left[n\right]\\ \sum _k{z}_{4kij}^{\left(3\right)}=1\forall i,j\in \left[n\right],\end{array}$$ (20) $$\begin{array}{l}{x}_{ik}-(1-z_{5klij}^{\left(1\right)})M\le u_{ij}\forall i,j,k,l\in \left[n\right],\\ w_{kl}-(1-z_{5klij}^{\left(2\right)})M\le u_{ij}\forall i,j,k,l\in \left[n\right],\\ y_{lj}-(1-z_{5klij}^{\left(3\right)})M\le u_{ij}\forall i,j,k,l\in \left[n\right],\\ z_{5klij}^{\left(1\right)}+z_{5klij}^{\left(2\right)}+z_{5klij}^{\left(3\right)}=1\forall i,j,k,l\in \left[n\right],\\ x_{ik}+(1-z_{5klij}^{\left(4\right)})M\ge u_{ij}\forall i,j,k,l\in \left[n\right],\\ w_{kl}+(1-z_{5klij}^{\left(4\right)})M\ge u_{ij}\forall i,j,k,l\in \left[n\right],\\ y_{lj}+(1-z_{5klij}^{\left(4\right)})M\ge u_{ij}\forall i,j,k,l\in \left[n\right],\\ z_{5klij}^{\left(1\right)},z_{5klij}^{\left(2\right)},z_{5klij}^{\left(3\right)},z_{5klij}^{\left(4\right)}\in \{0,1\},\\ \sum _{k,l}{z}_{5klij}^{\left(4\right)}=1\forall i,j\in \left[n\right],\end{array}$$ (21) where $a_{ij},b_{ij},w_{ij}$ are, respectively, the entries of the public matrices $A,B,W$, and $x_{ij},y_{ij}$ are the variables of the system. 2: Solve the MILP, and construct $X=\left(x_{ij}\right)$ and $Y=\left(y_{ij}\right)$. 3: return $(X,Y)$.

Note that a distinct advantage of these attacks is that they are independent of the polynomial degree used in the protocol. Therefore, Alice and Bob cannot improve the protocol’s resistance against these attacks by increasing the polynomial degree, a way that is very effective against Kotov–Ushakov attack and its max–min analog (Algorithms 1 and 2). In other words, the limitations for MILP Shpilrain attacks fully depend on the MILP techniques being used, but it is inevitable that the memory usage blows up as the matrix dimensions increase, due to the high number of equations and hence variables involved in the linear program. Figure 7 shows the time taken by Algorithm 7 for different polynomial degrees.

As illustrated in Figure 7, this attack is much faster than Algorithm 1 and maintains consistent computational efficiency across varying polynomial degrees. It is worth noting that for larger matrix dimensions, such as $n=10$ or higher, the Gurobi solver may encounter challenges in directly solving the system in some trials. Fine-tuning of the solver parameters is required to solve the system in such cases. The time taken by Algorithm 8 for different polynomial degrees is shown in Figure 8. Note that due to the higher number of equations and variables in the max–min case compared with the tropical case, the memory required for encoding the linear program for a dimension higher than 8 would exceed the available memory threshold.

We now summarize the performance of the suggested attacks in

Table 3. Comparative performance of the algorithms.

Algorithm	Semiring	Computational Efficiency	Memory Use	Empirical Success
1 Kotov–Ushakov (Algorithm 1)	Tropical	Inefficient	Low	$100\%$
2 Kotov–Ushakov (Algorithm 2)	Max–min	Inefficient	Low	$100\%$
3 Simulated Annealing (Algorithm 3)	Tropical	Efficient (most cases)	Low	$100\%$
4 Simulated Annealing (Algorithm 4)	Max–min	Efficient (most cases)	Low	<$100\%$
5 MILP Kotov–Ushakov (Algorithm 5)	Tropical	Inefficient	Low	$100\%$
6 MILP Kotov–Ushakov (Algorithm 6)	Max–min	Inefficient	Low	$100\%$
7 MILP Shpilrain (Algorithm 7)	Tropical	Efficient	High	$100\%$
8 MILP Shpilrain (Algorithm 8)	Max–min	Efficient	High	$100\%$

. Here, note that our conclusion on the computational efficiency is based on the numerical experiments (see Figure 1, Figure 3 and Figure 5, Figure 6, Figure 7 and Figure 8). The required assumptions and some other notes are summarized in

Table 4. Constraints, assumption, and notes.

Algorithm	Constraints, Assumptions, and Notes
Kotov–Ushakov (Algorithms 1 and 2)	Public matrices $A,B,W$ and transmitted U are assumed known. Require D to be larger than the actual maximum degree of polynomials used by Alice and Bob.
Simulated Annealing (Algorithms 3 and 4)	Public matrices $A,B,W$ and transmitted U are assumed known. Require D to be larger than the actual maximum degree of polynomials used by Alice and Bob. Sufficiently large initial temperature parameter. Larger range of entries and coefficients increases the observed execution time. Stopping criteria: the objective function reaches 0 (tropical); the objective function does not change after a specified number of loops (max–min).
MILP Kotov–Ushakov (Algorithms 5 and 6)	Public matrices $A,B,W$ and transmitted U are assumed known. Require D larger than the actual maximum degree of polynomials used by Alice and Bob. Large value of the parameter M must be chosen as it affects correctness and numerical stability. Use a MILP solver.
MILP Shpilrain (Algorithms 7 and 8)	Public matrices $A,B,W$ and transmitted U are assumed known. Large value of the parameter M must be chosen as it affects correctness and numerical stability. Independent of the actual maximum degree of polynomials used by Alice and Bob. Use a MILP solver. Require substantial memory.

.

4. Attacking Stickel’s Protocol over Digital Semiring

A recent implementation of Stickel protocol (Protocol 1) was introduced by [8], which employs a newly defined semiring referred to by the authors as the “digital semiring”. The authors claim that this new implementation of Stickel protocol resists the known attacks such as the Kotov–Ushakov attack. Let us discuss how the methods outlined in this paper as well as those in [5] can be applied in this new situation.

The digital semiring of [8], which we here denote by $\mathbb{N}(\vee ,\wedge )$, is defined over the set of natural numbers $\mathbb{N}$ with adjoined $+\infty$, and is based on an unconventional order relation defined by

$$a⪯b\iff \left\{\begin{array}{cc}\left(a\right)\le \left(b\right),\hfill & \mathrm{if}\left(a\right)\ne \left(b\right),\hfill \\ a\le b,\hfill & \mathrm{if}\left(a\right)=\left(b\right),\hfill \end{array}\right.$$

(22)

where $\left(a\right)$ denotes the sum of digits of $a\in \mathbb{N}$. It is understood that the sum of digits of $+\infty$ is $+\infty$, so this is the greatest element of the semiring. Based on this order relation, we then define the new addition $a\oplus b$ as the greatest element (also denoted as $a\vee b$) among $a,b$ with respect to this order relation, and $a\otimes b$ as the smallest element (also denoted as $a\wedge b$) among $a,b$ with respect to this order relation.

For the practical purposes of software implementation, Alice and Bob are always limited by a big enough number M, and therefore they would actually be using a semiring of the form ${\mathbb{N}}_M(\vee ,\wedge )$ similarly defined using (22) over the natural numbers not exceeding M. However, it then can be shown that this semiring ${\mathbb{N}}_M(\vee ,\wedge )$ is isomorphic to the semiring ${\mathbb{N}}_M(\max ,\min )$, which is the set of natural numbers not exceeding M for which the operations are defined by $a\oplus b=\max (a,b)$ and $a\otimes b=\min (a,b)$. Indeed, the isomorphism is given by the mapping $f:{\mathbb{N}}_M(\vee ,\wedge )\mapsto {\mathbb{N}}_M(\max ,\min )$, for which

$$f\left(a\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}a=0,\hfill \\ \sum _{i=1}^{\left(a\right)-1}{|\left[i\right]}_{\le M}{|+|\left[\left(a\right)\right]}_{\le a}|,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(23)

where ${\left[i\right]}_{\le a}$, for natural $a,i$ such that $0\le i,a\le M$, denotes the set of natural numbers whose sum of digits is equal to i and which do not exceed a, and $|{\left[i\right]}_{\le a}|$ denotes the number of elements in this set.

Consequently, the attacks on the max–min semiring implementation of Stickel protocol discussed in this paper are equally applicable to the digital semiring implementation, due to the known limitations of Alice and Bob and the isomorphism given by (23). This also includes the guaranteed attack described in [5] (the max–min version of Kotov–Ushakov attack). Thus, the attacker only needs to take one additional step to exploit this isomorphism. A possible approach for such exploitation is to group the elements of the digital semiring by their digit sums, arranging the groups and the numbers within each group in ascending order. Each element in the digital semiring is then mapped to a corresponding element in the max–min semiring with the natural order from smallest to largest. The resulting algorithm has complexity at most $O\left(M{log}_{10}M\right)$ since we have to go through each number and compute the sum of its digits (which has complexity not exceeding $O\left({log}_{10}M\right)$).

Figure 9 illustrates the computational time needed to execute it for different maximum values M.

As shown in Figure 9, the computational time required for this isomorphism mapping is relatively minor, but it obviously increases as Alice and Bob agree on higher ranges. However, it can be argued that they cannot extend these ranges indefinitely due to the risk of potential numerical instability. Thus, while attacking the Stickel protocol over the digital semiring involves this additional computational overhead, it is a one-time setup and does not affect the computational time during individual attack sessions since it should only be pre-computed once. Therefore, to keep the paper more concise, we have not included numerical experiments for attacking the Stickel protocol over the digital semiring, as these would be identical to the experiments on attacking the Stickel protocol over the max–min semiring described in the previous section and in [5]. We also note that a different attack on the Stickel protocol over digital semiring has been recently published in [24], which develops a branch and bound approach and exploits the structure of the circulant matrices involved in the protocol.

5. Forging the Tropical Signatures

A digital signature protocol based on the hardness of tropical polynomial factorization was proposed in [9]. Several heuristics to attack this protocol have been proposed in [25,26]. These heuristics primarily focus on generating a valid forged signature from a previously legitimate signature. To counter these attacks, along with other trivial forgeries, a revised version of the protocol has also been introduced. In this section, we present new attacks that directly target the public key, which also apply to the revised version, as the public key is unchanged. In what follows, we present the protocol and how it can be attacked. To read this section, the basic knowledge of semiring algebra (see Definitions 1–4) will be required from the reader, but only the tropical semiring ${\mathbb{R}}_{\max }$ will be used (see Table 1 for a concise summary).

Protocol 2

(The tropical digital signatures [9]).

Private Key: Two tropical polynomials $X,Y$, with integer coefficients from $[0,r]$ and the sum of their degrees is $2d$.

Public Key: r and d, and the multiplication of the two secret polynomials $M=X\otimes Y$.

Signing:

• Compute the hash of the message, and use it to form the tropical polynomial H using a known deterministic procedure.
• Select random private polynomials $U,V$ such that $deg\left(U\right)=deg\left(Y\right)$ and $deg\left(V\right)=deg\left(X\right)$, with coefficients in $[0,r]$, and let $N=U\otimes V$.
• The signature is the tuple $(H,H\otimes X\otimes U,H\otimes Y\otimes V,N)$.

Verification:

• Compute H as in the first step of signing, and verify it.
• Verify that $deg(H\otimes X\otimes U)=deg(H\otimes Y\otimes V)=3d$ and $deg\left(N\right)=2d$.
• Verify that neither $H\otimes X\otimes U$ nor $H\otimes Y\otimes V$ is a tropical constant multiple of $H\otimes M$ or $H\otimes N$.
• Verify that coefficients of $H\otimes X\otimes U$ and $H\otimes Y\otimes V$ are within $[0,3r]$ and those of N are within $[0,2r]$.
• Compute $W=(H\otimes X\otimes U)\otimes (H\otimes Y\otimes V)$, and accept the signature if and only if $W=H\otimes H\otimes M\otimes N$.

The security of this protocol relies on the hardness of tropical polynomial factorization, which was shown to be NP-hard [27]. This problem can be formulated as follows:

Problem 3

(Tropical Polynomial Factorization). Given a tropical polynomial $M=X\otimes Y$, find X and Y.

At first glance, it might seem straightforward to factor M using the tropical fundamental theorem of algebra [28], which states that any tropical polynomial can be easily factored into exactly linear polynomials. Let us explore this theorem formally.

Theorem 1

(Tropical fundamental theorem of algebra [28]). Any tropical polynomial of degree n

$$M\left(t\right)=\underset{i=0}{\overset{n}{⨁}}(m_i\otimes t^{\otimes i})$$

can be efficiently factored into linear factors. Specifically, there exists a constant c and roots $r_1,r_2,\dots ,r_n$ such that

$$M\left(t\right)=c\otimes \left(\underset{i=1}{\overset{n}{⨂}}(t\oplus r_i)\right),$$

The roots $r_i$ are the points where the piecewise-linear function $M\left(t\right)$ changes slope. This factorization provides a canonical form of $M\left(t\right)$ as a function.

Note that the factorization from this theorem is a functional factorization, meaning $M\left(t\right)$ holds for all t as a function. However, it does not necessarily preserve the original coefficient sequence $(m_0,m_1,\dots ,m_n)$ of M. That is, the string of coefficients obtained from this factorization is a canonical (most reduced) form of the tropical polynomial. However, this canonical form, while equivalent to the original polynomial as a function, does not necessarily preserve the initial polynomial’s sequence of coefficients.

In contrast, a sequence-based factorization requires finding X and Y such that their polynomial multiplication matches the original coefficients of M, where the coefficients $m_k$ of M are as follows:

$$m_k=\underset{(i,j):i+j=k}{⨁}(x_i\otimes y_j)=\underset{i+j=l}{\max }(x_i+y_j),k=0,1,\dots ,n.$$

Therefore, the security of Problem 3 relies on factoring M as a sequence (i.e., string of numbers), a problem shown to be NP-hard. Factoring M as a function does not generally preserve the original sequence, which most likely causes the original sequence recovery to fail. That is, a function-based factorization yield factors that satisfy the same maximum operations but do not necessarily reconstruct the original sequence of coefficients. In contrast, a sequence-based factorization requires that multiplying the factors exactly reproduces the original coefficients of M. As such, it is required for the attacks on Problem 3 to target a “sequence-based” factoring of M, where the multiplication of the factors exactly recovers the original coefficients of M.

Note that there are possibly many factorizations of M, meaning the original factors X and Y are not generally unique. Therefore, for the attacker’s purpose of producing a valid forged signature in Protocol 2, it is sufficient to find any factors that pass the verification process. This non-uniqueness in factorization can be exploited as a basis for some heuristic attacks. Thus, in the proposed attacks that follow, the attacker’s objective is to find $X^{\prime }$ and $Y^{\prime }$ such that $X^{\prime }\otimes Y^{\prime }=M$, with the additional constraints that their degrees sum to $2d$ and their coefficients are from $[0,r]$, so they can pass the verification process. Successfully finding $X^{\prime }$ and $Y^{\prime }$ enables the attacker to impersonate the signer and hence produce a valid signature to any arbitrary message. Specifically, with H being the polynomial formed from an arbitrary hashed message, and choosing U and V with $deg\left(U\right)=deg\left(Y^{\prime }\right)$ and $deg\left(V\right)=deg\left(X^{\prime }\right)$, with coefficients in $[0,r]$, the forged signature $(H,H\otimes X^{\prime }\otimes U,H\otimes Y^{\prime }\otimes V,N=U\otimes V)$ is verified correctly, as all of the above verification steps clearly hold, and it is highly unlikely that the second and third polynomials of this tuple will be shifted versions of the public polynomials $H\otimes M$ or $H\otimes N$, respectively. We now propose two attacks utilizing this approach.

• Kotov–Ushakov-based attack

Note that M essentially represents a convolution of the two sequences X and Y, with max-plus operations. This allows the problem to be formulated as a one-sided linear system using matrices, by treating each product of the secret coefficients as a variable. However, the length of the original sequences is unknown. Consequently, the attack must iterate over possible lengths for $X^{\prime }$ until a suitable solution to the one-sided linear system is found.

Formally, we know that each coefficient $m_k$ of $M=X\otimes Y$ can be represented as

$$m_k=\underset{(i,j):i+j=k}{⨁}{x}_i\otimes y_j,$$

where $m_k$, $x_i$, and $y_j$ denote the coefficients of the polynomials M, X, and Y, respectively. Then, with $x_i$ and $y_j$ being the unknowns, this system can be equivalently written as the linear system $A\otimes z=b$, where A is a binary matrix that indicates which variables are present in the k-th equation, z is the vector of unknowns with each element $z_{ij}=x_i\otimes y_j$, and b is the vector containing the known coefficients of M. The following example shows an illustration of this representation.

Example 1

(One-sided linear system representation of polynomial multiplication). For a polynomial M of degree 4, and polynomials X and Y each of degree 2, the polynomial multiplication $M=X\otimes Y$ can be represented as the following linear system:

$$\left[\begin{array}{ccccccccc}0& -\infty & -\infty & -\infty & -\infty & -\infty & -\infty & -\infty & -\infty \\ -\infty & 0& -\infty & 0& -\infty & -\infty & -\infty & -\infty & -\infty \\ -\infty & -\infty & 0& -\infty & 0& -\infty & 0& -\infty & -\infty \\ -\infty & -\infty & -\infty & -\infty & -\infty & 0& -\infty & 0& -\infty \\ -\infty & -\infty & -\infty & -\infty & -\infty & -\infty & -\infty & -\infty & 0\end{array}\right]\otimes \left[\begin{array}{c}{x}_0\otimes y_0\\ x_0\otimes y_1\\ x_0\otimes y_2\\ x_1\otimes y_0\\ x_1\otimes y_1\\ x_1\otimes y_2\\ x_2\otimes y_0\\ x_2\otimes y_1\\ x_2\otimes y_2\end{array}\right]=\left[\begin{array}{c}{m}_0\\ m_1\\ m_2\\ m_3\\ m_4\end{array}\right]$$

Thus, the attacker’s goal is to find a solution to this linear system. That is, a solution $r_{ij}$ that satisfies $r_{ij}=x_i\otimes y_j$ for all $i\in \{0,1,\dots ,d_x\}$ and $j\in \{0,1,\dots ,d_y\}$, for some $x_i$ and $y_j$. Additional constraints must be imposed on $x_i$ and $y_j$ to ensure that the forged signature is verified correctly. These constraints are $x_i,y_j\in [0,r]$ and $d_x+d_y=2d$, where r and d are public parameters of the protocol. Note that this system is not guaranteed to have a solution unless $d_x$ equals the original degree of the polynomial X, but this degree is secret. Consequently, the attacker must test multiple values of $d_x$ until a solution is found. However, it is possible that a solution can be found even when $d_x$ differs from the original degree of X due to the possible non-unique factorization of M. The attack is formally described below.

Figure 10 presents the performance of this attack when $t=3$, showing the success rate and computational time over 10 trails for multiple values of d. Note that, for all numerical experiments, the degree of X in the protocol instance is chosen as specified by the authors, i.e., randomly selected from the interval $\left[{\displaystyle \frac{3}{4}}d,{\displaystyle \frac{5}{4}}d\right]$. The degree of Y is then determined accordingly, as the sum of the degrees of X and Y must equal $2d$.

Figure 10. Success rate and computational time of Algorithm 9.

Figure 10. Success rate and computational time of Algorithm 9.

Algorithm 9 Kotov–Ushakov-based attack on Protocol 2

Input: Public key polynomial M, signature parameters $t,r$, degree bound $2d$. Output: Recovered factors $X^{\prime },Y^{\prime }$. 1: for  $dx=1$  to  t  do 2:   Set $dy=2d-dx$. 3:   Construct binary matrix A and vector b for the linear system as in Example 1. 4:   Compute the greatest solution $c_{ij}={\min }_i(b_i-A_{ij})$ and the sets $S_{ij}=arg{\min }_i(b_i-A_{ij})$ for all $i\in \{0,\dots ,d_x\}$ and $j\in \{0,\dots ,d_y\}$. 5:   Among all minimal covers of $\{0,1,\dots ,2d\}$ by $S_{ij}$, that is, all minimal subsets $\mathcal{C}\subseteq \{0,1,\dots ,2d\}$ such that $$\bigcup _{(i,j)\in \mathcal{C}}{S}_{ij}=\{0,1,\dots ,2d\},$$ find a cover for which the system $$\begin{array}{l}{x}_i+y_j=c_{ij},\mathrm{if}(i,j)\in \mathcal{C},\\ x_i+y_j⩽c_{ij},\mathrm{otherwise},\\ x_i,y_j\in [0,r].\end{array}$$ is solvable. 6:   If a solution is found, break the loop. If no solution is found, proceed to the next $d_x$ until a solution is found. 7: Construct the polynomials $X^{\prime }$ and $Y^{\prime }$ using the derived $x_i$ and $y_j$, respectively. 8: return $(X^{\prime },Y^{\prime })$.

While the attack achieves a considerable success rate, its efficiency is limited, even for short polynomial lengths, due to the large number of enumerated minimal covers. Therefore, it is impractical for the recommended protocol parameters ($d=150$).

• Mixed-integer linear programming (MILP) attack

The attacker similarly aims to find $X^{\prime }$ and $Y^{\prime }$ that recovers the original M. In this attack, similar to the approach used in the attacks discussed in Section 3.2 and Section 3.3, the attacker transforms the disjunctive constraints in the formula for each $m_k$ into a set of linear constraints by introducing Boolean variables $z_{kij}$. This reformulation allows the problem to be solved as a mixed-integer linear program.

More precisely, since each coefficient $m_k$ of M satisfies

$$m_k=\underset{(i,j):i+j=k}{\max }(x_i+y_j),$$

it can be equivalently expressed through the following subsystem of inequalities:

$$\begin{array}{c}{x}_i+y_j\le m_k,\forall i,j,\\ x_i+y_j+(1-z_{kij})T\ge m_k,\forall i,j,\\ \sum _{i,j}{z}_{kij}=1,z_{kij}\in \{0,1\},\forall i,j.\end{array}$$

Here, T is a sufficiently large constant. This approach can be used to propose the following attack.

Figure 11 shows the performance of this attack with $t=3$, where it achieves a success rate comparable to the previous attack but with significantly greater efficiency, even for the recommended protocol parameters ($d=150$).

In practical terms, this success rate means that the attacker can successfully factor the public key in approximately half of all randomly generated instances. Consequently, if the protocol were deployed, the attacker could potentially impersonate half of the users and sign messages using their signatures. Recall that this success rate is explained by the existence of alternative factors $X^{\prime }$ and $Y^{\prime }$ different from the original pair, which still satisfy the verification process and can be efficiently found via the MILP formulation.

Figure 11. Success rate and computational time of Algorithm 10.

Figure 11. Success rate and computational time of Algorithm 10.

Algorithm 10 MILP-based attack on Protocol 2

Input: Public key coefficients $m_k$, signature parameters $t,r$, degree bound $2d$, big constant T. Output: Recovered factors $X^{\prime },Y^{\prime }$. 1: for  $d_x=1$  to  t  do 2:   Set $d_y=2d-d_x$. 3:   Solve the following system for all $k\in \{0,1,\dots ,2d\}$, and for all $i\in \{0,1,\dots ,d_x\}$ and $j\in \{0,1,\dots ,d_y\}$ such that $i+j=k$ using a MILP solver. $$\left\{\begin{array}{c}{x}_i+y_j\le m_k,\hfill \\ x_i+y_j+(1-z_{kij})T\ge m_k,\hfill \\ \sum _{(i,j):i+j=k}{z}_{kij}=1,z_{kij}\in \{0,1\}.\hfill \end{array}\right.$$ 4: Construct the polynomials $X^{\prime }$ and $Y^{\prime }$ using the derived $x_i$ and $y_j$, respectively. 5: return $(X^{\prime },Y^{\prime })$.

6. Conclusions

In this paper, we proposed three new attacks against the tropical and max–min implementations of Stickel protocol. Our aim was to avoid the problem of minimal covers enumeration and the associated worst case exponential complexity encountered in the Kotov–Ushakov attacks. While we previously proposed an attack against these protocols [5,14] that avoided enumerating all minimal solutions by carefully selecting a single minimal solution, this method, although very successful for the tropical case, occasionally fails. Consequently, it is plausible that Alice and Bob could design the protocol’s public matrices to resist this attack, and this method still shows increasing complexity with the polynomial degree used, though not exponentially. Thus, the goal of the techniques implemented in Algorithms 3–8 was to achieve a success rate above $95\%$ with the lowest possible execution time and reduced dependence on the polynomial degree, which is commonly the variable parameter controlled by Alice and Bob.

The first proposed attack (Algorithms 3 and 4) aims to find a solution x that minimizes an objective function of the shape ${\sum }_i{({(A\otimes x)}_i-b_i)}^2$ instead of finding all minimal solutions of a system $A\otimes x=b$ as in the typical Kotov–Ushakov attack. This attack employs the simulated annealing algorithm, a global optimization technique, to find such solution. It achieved a perfect success rate $100\%$ against the tropical Stickel protocol and a high success rate (above $90\%$) against the max–min Stickel protocol, both with very fast execution times. Additionally, the execution time showed only a minor increase as the polynomial degree increased. However, unlike the Kotov–Ushakov attack, this approach is sensitive to the size of public matrix entries and polynomial coefficients used in the protocol. While it remains usually effective even for large values, we are more likely to encounter some trials that take significantly longer than average to solve. Also, we cannot definitely say that simulated annealing outperforms other attacks in the max–min case since it is not achieving a perfect success rate in our experiments (or rather, we have to “sacrifice” the success rate in order for the attack to be complete within a reasonable timeframe).

The second proposed attack (Algorithms 5 and 6) aims to solve the system $A\otimes x=b$ by transforming it into a mixed-integer linear system and then solving it using MILP solver. Unfortunately, this attack demonstrated slower execution times compared to the typical Kotov–Ushakov attack, and it remains heavily dependent on the polynomial degree used in the targeted protocols. Consequently, similar to the typical Kotov–Ushakov attack, Alice and Bob can resist this attack by increasing the polynomial degree.

The third proposed attack (Algorithms 7 and 8), which we call Shpilrain’s attack, aims to solve equations (11) by formulating them as a mixed-integer linear program. Interestingly, this attack is completely independent from the used polynomial degree in the protocol, which makes it effective even if Alice and Bob use very high polynomial degrees. The attack has also demonstrated remarkably fast execution times, taking roughly 21 s for the tropical case with dimension 10 and polynomial degree 50. A significant limitation of this attack is its high memory requirement due to the need of encoding a large number of equations, namely on the order of $n^4$. Consequently, Alice and Bob could potentially defend against it by employing large matrix dimensions. However, it is worth noting that the typical Kotov–Ushakov attack would likely encounter similar challenges in such scenarios, specifically those related to the high number of minimal covers.

Let us also observe that Shpilrain’s attack also applies to the modifications of Stickel protocol based on Jones matrices and Linde-de la Puente matrices suggested in [15]. Namely, the protocol based on Jones matrices is only replacing the tropical polynomials of A and B with tropical quasi-polynomials of the same matrices, so we can still find X and Y directly from (11) (and its MILP reformulation). As for the Linde-de la Puente matrices, equations $X\otimes A=A\otimes X$ and $Y\otimes B=B\otimes Y$ have to be replaced with linear inequalities and equations that define Linde-de la Puente matrices. We are not including the numerical results here but the situation is similar to what is reported in Figure 7.

Finally, it is notable that the findings presented in this paper likely indicate that the max-min and hence also “digital” implementations of the Stickel protocol overall tend to be more resistant to the attacks described in this paper and [5] than the tropical implementation. This conclusion arises because two of the three proposed attacks in this paper, alongside the single cover heuristic [14], demonstrate much greater effectiveness against the tropical case. Furthermore, the typical Kotov–Ushakov attack is more efficient against the tropical Stickel protocol compared to its analogue against the max–min Stickel protocol. Better implementation of Shpilrain’s attack and alternative ideas which would allow for solving Problems 1 and 2 with higher dimensional matrices are still to be considered. Also, the reasons behind the relatively good performance of simulated annealing in the tropical case and “satisfactory” performance in the max–min case are not clear to us and can be a topic of further research, as well as the conditions under which the simulated annealing based attacks are guaranteed to solve a problem within a reasonable timeframe.