Partial Exposure Attacks on a New RSA Variant

Abstract

In 2022, Cotan and Teşeleanu presented a variant of the RSA cryptosystem where the modulus is of the form $N=pq$, and the private and the public exponents satisfy $ed\equiv 1(\mathrm{mod}{\psi }_n\left(N\right))$ with $n\ge 2$, and ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$. This variant of RSA was recently cryptanalyzed by Nitaj, Adenan, and Ariffin at Africacrypt 2024. In this paper, we push further the cryptanalysis of the scheme of Cotan and Teşeleanu by presenting a method to solve the equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ where c is a constant that is independent of x and y. This enables us to propose more attacks on the scheme, including a partial key exposure attack, an attack when the most significant bits of one of the prime factors are known, and an attack when the least significant bits of one of the prime factors are known.

1. Introduction

Invented in 1978 by Rivest, Shamir, and Adleman [1], the RSA cryptosystem is one of the most used public key cryptosytems regarding its practical applications. Its security is related to the hardness of factoring composite large integers. To use the RSA scheme, one starts by generating two large prime numbers p and q of the same bit size, and it computes $N=pq$ as the RSA modulus. Then, one selects an integer e, called the public exponent, satisfying $\gcd \left(e,p-1\left)\right(q-1)\right)=1$. This enables us to compute the private exponent d as the inverse of e modulo $(p-1)(q-1)$, that is $ed\equiv 1(\mathrm{mod}(p-1\left)\right(q-1\left)\right)$. The encryption process allows transforming a plaintext $m<N$ to a ciphertext $c\equiv m^e(\mathrm{mod}N)$. To recover the plaintext m, one applies the decryption process $m\equiv c^d(\mathrm{mod}N)$. The efficiency of both encryption and decryption is based on the run time of the modular exponentiation. To reduce the run time, specifically in the decryption, it is tempting to use small private exponents. Unfortunately, in 1990, Wiener [2] showed that such a choice is vulnerable when $d\le \frac{1}{3}{N}^{\frac{1}{4}}$. The former bound was improved later by Boneh and Durfee [3] up to $N^{0.292}$.

Based on these obstacles, several variants have been proposed to improve the efficiency as well as the security of RSA. Some of these variants employ a modulus of the form $N=pq$ as in CRT-RSA [4], rebalanced RSA [2], and KMOV [5]. In contrast, other variants utilize different types of moduli, such as Multi-Prime RSA [6] and Prime-Power RSA [7].

In 2018, Murru and Saettone [8] introduced a new variant of the RSA scheme based on the cubic Pell equation $x^3+a{y}^3+a^2{z}^3-3axyz=1$, where a is a cubic non-residue modulo $N=pq$. They used $N=pq$ as a modulus, with the public key being $(N,e)$ and the private key $(N,d)$, where e and d satisfy $ed\equiv 1\left(\mathrm{mod}\frac{(p^3-1)(q^3-1)}{(p-1)(q-1)}\right)$. This variant of RSA has been intensively cryptanalyzed in [9,10,11,12].

In 2022, Cotan and Teşeleanu [13] proposed a generalization of the scheme of Murru and Saettone. They used a modulus $N=pq$, a public exponent e, and a private exponent d such that $ed\equiv 1\left(\mathrm{mod}\frac{(p^n-1)(q^n-1)}{(p-1)(q-1)}\right)$ for $n\ge 2$. The special case $n=3$ is the scheme of Murru and Saettone. The authors also presented an attack based on the continued fraction algorithm whenever $n\le 4$, $d=N^{\delta }$, $e=N^{\alpha }$, $\alpha \le n-\frac{1}{2}$, and $\delta <\frac{1}{4}(2n-2\alpha -1)$.

In 2024, Nitaj et al. [14] developed a novel attack on the Cotan and Teşeleanu scheme using Coppersmith’s method and lattice basis reduction. They demonstrated that one can efficiently factor the modulus $N=pq$ if $e=N^{\alpha }$, $d\le N^{\delta }$, $\frac{n-1}{2}\le \alpha \le 2(n-1)$, and $\delta <n-1-\frac{\sqrt{2}}{2}\sqrt{(n-1)\alpha }$.

In the work of Nitaj et al. [14], the authors started by solving the modular equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ where $H\left(y\right)$ is a monic polynomial of degree r, under certain conditions, namely, $e=N^{\alpha }$, $\left|x\right|<N^{\beta }$, $\left|y\right|<N^{\gamma }$, ${c<\left|x\right|\left|y\right|}^r<e$, and $\beta <\alpha -\sqrt{r\alpha \gamma }$. As a by-product, they presented an attack on the scheme of Cotan and Teşeleanu and showed that $N=pq$ can be factored for any $n\ge 2$ if e and d satisfy $ed\equiv 1\left(\mathrm{mod}\frac{(p^n-1)(q^n-1)}{(p-1)(q-1)}\right)$ and $\delta <n-1-\frac{\sqrt{2}}{2}\sqrt{(n-1)\alpha }$. This significantly improved the bound $\delta <\frac{1}{4}(2n-2\alpha -1)$ of Cotan and Teşeleanu.

In this paper, for a monic univariate polynomial $H\left(y\right)\in \mathbb{Z}\left[y\right]$ of degree r, we propose a new lattice-based method to solve the equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ when $N=pq$, $e=N^{\alpha }$, $\left|x\right|\le N^{\beta }$, $\left|y\right|\le N^{\gamma }$, ${\left|x\right|\left|y\right|}^r<e$, and $\beta <\alpha +\frac{1}{3}r\gamma -\frac{2}{3}\sqrt{3r\alpha \gamma +r^2{\gamma }^2}$. This can be achieved for any value of c; in particular, the condition $\left|c\right|<\left|x{y}^r\right|$ is no more required. This allows us to perform four attacks on the scheme of Cotan and Teşeleanu. The first attack deals with the situation where the least significant bits (LSBs) of the private exponent d are known. The second attack concerns the situation where an approximation of one of the primes is known. The third attack concerns the situation when the primes share their most significant bits (MSBs). The fourth attack concerns the situation where the primes share their least significant bits.

The paper is organized as follows. In Section 2, we present some preliminaries and provide a new expression for ${\psi }_n\left(N\right)$ that is useful in the sequel. In Section 3, we present the new method to find the small solutions of the equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$. In Section 4, we apply the proposed method to perform the first attack on the cryptosystem of Cotan and Teşeleanu, namely, an attack with known LSBs. In Section 5, we present the second attack, which is a partial prime exposure attack. In Section 6, we apply the third attack when the prime factors of the modulus share their MSBs. In Section 7, we present another expression for ${\psi }_n$ which allows performing the fourth attack when the prime factors of the modulus share their LSBs. Finally, we conclude the paper in Section 8.

2. Preliminaries

Let $N=pq$ be an RSA modulus with $q<p<2q$. Then, p and q can be bounded in terms of N as in the following simple lemma.

Lemma 1.

Let $N=pq$ be an RSA modulus with $q<p<2q$. Then,

$${\displaystyle \frac{\sqrt{2}}{2}}\sqrt{N}<q<\sqrt{N}<p<\sqrt{2}\sqrt{N}.$$

The following lemma shows how to find an approximation of q if an approximation of p is given (see [9]).

Lemma 2.

Let $N=pq$ be an RSA modulus with $q<p<2q$. Let $p_0$ be an approximation of p such that $|p - p_0|=N^{\mu }$. Then, $q_0=⌊\frac{N}{p_0}⌋$ is an approximation of q such that

$$|q - q_0|<N^{\mu }and|p+q-p_0-q_0|<2N^{\mu }.$$

The generalized totient function in the system of Cotan and Teşeleanu [13] is defined for $N=pq$ and $n\ge 2$ by

$${\psi }_n\left(N\right)={\displaystyle \frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}}.$$

The following result gives simple upper and lower bounds for ${\psi }_n\left(N\right)$.

Lemma 3.

Let $N=pq$, $n\ge 2$, and ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$. Then

$$N^{n-1}<{\psi }_n\left(N\right)<4N^{n-1}.$$

Proof. 

For the lower bound, we have

$${\psi }_n\left(N\right)=\left(p^{n-1}+p^{n-2}\dots +1\right)\left(q^{n-1}+q^{n-2}+\dots +1\right)>p^{n-1}{q}^{n-1}=N^{n-1}.$$

For the upper bound, using $x^{n-1}+x^{n-2}+\dots +1<2x^{n-1}$ for $x>2$, we obtain

$${\psi }_n\left(N\right)=\left(p^{n-1}+p^{n-2}\dots +1\right)\left(q^{n-1}+q^{n-2}+\dots +1\right)<4p^{n-1}{q}^{n-1}=4N^{n-1}.$$

This terminates the proof. □

The following result shows how to compute ${\psi }_n\left(N\right)$ (see [14]).

Lemma 4.

Let $N=pq$ and $S=p+q$. Then, ${\psi }_1\left(N\right)=1$, ${\psi }_2\left(N\right)=N+1+S$, and for $n\ge 3$,

$${\psi }_n\left(N\right)=N^{n-1}+1+S{\psi }_{n-1}\left(N\right)-N{\psi }_{n-2}\left(N\right).$$

The following result shows that ${\psi }_n\left(N\right)$ can be expressed as a polynomial of $p+q$ (see [14]).

Lemma 5.

Let $N=pq$ and $n\ge 2$. Then, there exist $n-1$ integer coefficients $a_{n-2},\dots ,a_0$ depending only on N and n such that

$${\psi }_n\left(N\right)={(p+q)}^{n-1}+\sum _{j=0}^{n-2}{a}_j{(p+q)}^j.$$

Note that in Lemma 5, the coefficients $a_i$ can be computed only by using N and n. Nevertheless, ${\psi }_n\left(N\right)$ cannot be computed by an adversary who does not know $p+q$.

The former result can be extended in the following form.

Lemma 6.

Let $N=pq$, $n\ge 2$, ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$, and $M\in \mathbb{Z}$. Then, there exist $n-1$ coefficients $a_j^{\left(n\right)}\in \mathbb{Z}$, $j=0,\dots ,n-2$, depending only on N, n, and M such that

$${\psi }_n\left(N\right)={(p+q-M)}^{n-1}+\sum _{j=0}^{n-2}{a}_j^{\left(n\right)}{(p+q-M)}^j.$$

Proof. 

We proceed by recursion. We have ${\psi }_1\left(N\right)=1$, and

$$\begin{array}{cc}\hfill {\psi }_2\left(N\right)& =(p+1)(q+1)\hfill \\ \hfill & =(p+q-M)+N+M+1,\hfill \\ \hfill {\psi }_3\left(N\right)& =(p^2+p+1)(q^2+q+1)\hfill \\ \hfill & ={(p+q-M)}^2+(N+2M+1)(p+q-M)\hfill \\ \hfill & +M(N+M+1)+N^2-N+1.\hfill \end{array}$$

Assume that, for $n\ge 4$, we have

$$\begin{array}{c}\hfill {\psi }_{n-2}\left(N\right)={(p+q-M)}^{n-3}+\sum _{j=0}^{n-4}{a}_j^{(n-2)}{(p+q-M)}^j,a_j^{(n-2)}\in \mathbb{Z},\\ \hfill {\psi }_{n-1}\left(N\right)={(p+q-M)}^{n-2}+\sum _{j=0}^{n-3}{a}_j^{(n-1)}{(p+q-M)}^j,a_j^{(n-1)}\in \mathbb{Z}.\end{array}$$

Using Lemma 4, we obtain

$$\begin{array}{cc}\hfill {\psi }_n\left(N\right)& =(p+q){\psi }_{n-1}\left(N\right)-N{\psi }_{n-2}\left(N\right)+N^{n-1}+1\hfill \\ \hfill & =(p+q-M){\psi }_{n-1}\left(N\right)+M{\psi }_{n-1}\left(N\right)-N{\psi }_{n-2}\left(N\right)+N^{n-1}+1\hfill \\ \hfill & ={(p+q-M)}^{n-1}+\left(a_{n-3}^{(n-1)}+M\right){(p+q-M)}^{n-2}\hfill \\ \hfill & +\left(a_{n-4}^{(n-1)}+M{a}_{n-3}^{(n-1)}-N\right){(p+q-M)}^{n-3}\hfill \\ \hfill & +\sum _{j=1}^{n-4}\left(a_{j-1}^{(n-1)}+M{a}_j^{(n-1)}-N{a}_j^{(n-2)}\right){(p+q-M)}^j\hfill \\ \hfill & +M{a}_0^{(n-1)}-N{a}_0^{(n-2)}+N^{n-1}+1\hfill \\ \hfill & ={(p+q-M)}^{n-1}+\sum _{j=0}^{n-2}{a}_j^{\left(n\right)}{(p+q-M)}^j,\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill & a_{n-2}^{\left(n\right)}=a_{n-3}^{(n-1)}+M,\hfill \\ \hfill & a_{n-3}^{\left(n\right)}=a_{n-4}^{(n-1)}+M{a}_{n-3}^{(n-1)}-N,\hfill \\ \hfill & a_j^{\left(n\right)}=a_{j-1}^{(n-1)}+M{a}_j^{(n-1)}-N{a}_j^{(n-2)},j=1,\dots ,n-4,\hfill \\ \hfill & a_0^{\left(n\right)}=M{a}_0^{(n-1)}-N{a}_0^{(n-2)}+N^{n-1}+1.\hfill \end{array}$$

This shows that all the coefficients $a_j^{\left(n\right)}$, $0\le j\le n-2$ are integers and depend only on N, n, and M. This terminates the proof. □

Using Lemma 4, one can express the first values of ${\psi }_n\left(N\right)$ as a polynomial in $T=p+q-M$. For instance, we have

$$\begin{array}{cc}\hfill {\psi }_1\left(N\right)& =1,\hfill \\ \hfill {\psi }_2\left(N\right)& =T+M+N+1,\hfill \\ \hfill {\psi }_3\left(N\right)& =T^2+(2M+N+1)T+M(M+N+1)+N^2-N+1,\hfill \\ \hfill {\psi }_4\left(N\right)& =T^3+(3M+N+1)T^2+\left(M(3M+2N+2)+N^2-2N+1\right)T\hfill \\ \hfill & +M^2+M^3+M+M(N^2+MN-2N)+N^3-N^2-N+1,\hfill \\ \hfill {\psi }_5\left(N\right)& =T^4+(4M+N+1)T^3+\left(M(6M+3N+3)+N^2-3N+1\right)T^2\hfill \\ \hfill & +\left(4M^3+3M^2+2M+M(3MN+2N^2-6N)+N^3-2N^2-2N+1\right)T\hfill \\ \hfill & +M^4+M^3+M^2+M+M(M^{2}N+M{N}^2-3MN+N^3-2N^2-2N)\hfill \\ \hfill & +N^4-N^3+N^2-N+1.\hfill \end{array}$$

2.1. Lattice Basis Reduction and Coppersmith’s Method

Let $\omega$ and n be positive integers with $\omega \le n$. Let $v_1,v_2,\dots ,v_{\omega }$ be $\omega$ linearly independent vectors of ${\mathbb{R}}^n$. A lattice $\mathcal{L}\subset {\mathbb{R}}^n$ is the set of all integer linear combinations of $v_1,v_2,\dots ,v_{\omega }$, that is,

$$\mathcal{L}=\mathbb{Z}{v}_1+\mathbb{Z}{v}_2+\dots +\mathbb{Z}{v}_{\omega }.$$

The lattice $\mathcal{L}$ can be represented by a matrix B whose rows are the vectors $v_1,v_2,\dots ,v_{\omega }$. The parameter n is the dimension of the lattice $\mathcal{L}$, and $\omega$ is its rank. Its determinant is defined to be $\det \left(\mathcal{L}\right)=\sqrt{\det \left(B^{t}B\right)}$ where $B^t$ is the transpose of B. When $\omega =n$, we say that the lattice $\mathcal{L}$ is full-rank, and then its determinant is simplified to $\det \left(\mathcal{L}\right)=|\det \left(B\right)|$.

It is known that a lattice $\mathcal{L}$ has infinitely many bases, and finding a basis with short vectors is a hard task especially when the dimension of the lattice is large. In 1982, Lenstra, Lenstra and Lovász [15] proposed LLL, which is a polynomial time algorithm to find a short basis. The following result [16] is widely used to estimate the output of the LLL algorithm.

Theorem 1.

Let $\mathcal{L}$ be a lattice spanned by a basis $(v_1,v_2,\dots ,v_{\omega })$. The LLL algorithm produces a reduced basis $(u_1,u_2,\dots ,u_{\omega })$ satisfying

$$\parallel u_1\parallel \le \dots \le \parallel u_i\parallel \le 2^{\frac{\omega (\omega -1)}{4(\omega +1-i)}}\det {\left(\mathcal{L}\right)}^{\frac{1}{\omega +1-i}},\mathrm{for}i=1,\dots ,\omega .$$

2.2. Coppersmith’s Method

In 1996, Coppersmith [17] proposed an efficient way to find small roots of modular polynomial equations of the form $f\left(x\right)\equiv 0(\mathrm{mod}M)$, mainly when the factorization of the modulus M is unknown. Since then, Coppersmith’s method has been generalized to polynomials with more variables, specifically polynomials of the form

$$f(x_1,x_2,\dots ,x_n)=\sum _{i_1,i_2,\dots ,i_n}{a}_{i_1,i_2,\dots ,i_n}{x}_1^{i_1}{x}_2^{i_2}\cdots x_n^{i_n},$$

with $a_{i_1,i_2,\dots ,i_n}\in \mathbb{Z}$. For such polynomials, the Euclidean norm is defined by $\parallel f(x_1,x_2,\dots ,x_n)\parallel =\sqrt{\sum a_{i_1,i_2,\dots ,i_n}^2}$.

In 1997, Howgrave-Graham [18] clarified Coppersmith’s method in the following sense.

Theorem 2

(Howgrave-Graham). Let $f(x_1,x_2,\dots ,x_n)\in \mathbb{Z}[x_1,x_2,\dots ,x_n]$ be a multivariate polynomial with at most ω monomials. Let e and m be positive integers. Suppose that

1. 

$f(y_1,y_2,\dots ,y_n)\equiv 0(\mathrm{mod}{e}^m)$.

2. 

$\parallel f(x_1{X}_1,x_2{X}_2,\dots ,x_n{X}_n)\parallel <\frac{e^m}{\sqrt{\omega }}$, $|y_i| <X_i$, for $i=1,\dots ,n$.

Then, $f(y_1,y_2,\dots ,y_n)=0$ holds over the integers.

When more than two variables are involved, the methods based on Coppersmith’s technique are heuristic. In this paper, we use the following assumption [3,12,19,20]. This is a reasonable assumption that holds true when the parameters are sufficiently smaller than the theoretical bounds.

Assumption 1.

The reduced polynomials $h_1,h_2,\dots ,h_{\omega }$ generated by the LLL algorithm are algebraically independent.

Under the former assumption, the common root $(y_1,y_2,\dots ,y_n)$ of the polynomial equations $h_i(y_1,y_2,\dots ,y_n)=0$, $i=1,\dots ,\omega$ can be extracted by the Gröbner basis method or resultant techniques.

2.3. The Scheme of Cotan and Teşeleanu

Before describing the scheme, we need to define some mathematical objects that are useful in the sequel. Let $(\mathbb{F},+,·)$ be a field. Let n be an integer and $a\in \mathbb{F}$ such that $x^n-a$ is irreducible in $\mathbb{F}\left[x\right]$. Define the quotient field

$${\mathbb{A}}_n=\mathbb{F}\left[x\right]/(x^n-a)=\{a_0+a_{1}x+\dots +a_{n-1}{x}^{n-1}\mid a_0,\dots ,a_{n-1}\in \mathbb{F}\}.$$

The product of two elements $a\left(x\right)={\sum }_{i=0}^{n-1}{a}_i{x}^i$ and $b\left(x\right)={\sum }_{i=0}^{n-1}{b}_i{x}^i$ of ${\mathbb{A}}_n$ can be computed by the rule

$$a\left(x\right)\circ b\left(x\right)=\sum _{i=0}^{n-2}\left(\sum _{j=0}^i{a}_j{b}_{i-j}+a\sum _{j=0}^{i+n}{a}_j{b}_{i-j+n}\right)x^i+\sum _{j=0}^{n-1}{a}_j{b}_{n-1-j}{x}^{n-1}.$$

Consider the quotient group ${\mathbb{B}}_n={\mathbb{A}}_n^{\ast }/{\mathbb{F}}^{\ast }$; then, elements of ${\mathbb{B}}_n$ are equivalence classes of the form

$$[a_0+\dots +a_{n-1}{x}^{n-1}]=\left\{\gamma a_0+\dots +\gamma a_{n-1}{x}^{n-1}\mid \gamma \in {\mathbb{F}}^{\ast },a_0,\dots ,a_{n-1}\in \mathbb{F}\right\}.$$

Note that ${\mathbb{B}}_n={\bigcup }_{k=0}^{n-1}{\mathbb{B}}_k$, where

$${\mathbb{B}}_k=\{a_0+\dots +a_{k-1}{x}^{k-1}+x^k\mid a_0,\dots ,a_{k-1}\in \mathbb{F}\},k=0,\dots ,n-1,$$

and ${\mathbb{B}}_i\cap {\mathbb{B}}_j=\varnothing$ whenever $i\ne j$.

When p is a prime number and $\mathbb{F}={\mathbb{F}}_p$ is the finite field of p elements, ${\mathbb{A}}_n$ becomes the Galois field of order $p^n$. Also, ${\mathbb{B}}_n$ is a cyclic group of order

$$\sum _{k=0}^{n-1}{\left|{\mathbb{F}}_p\right|}^k={\displaystyle \frac{p^n-1}{p-1}}.$$

If m is a positive integer and $y\in {\mathbb{B}}_n$, denote by $y^m$ the product of y in ${\mathbb{B}}_n$, $m-1$ times. Hence, an analogous of Fermat’s little theorem is given by

$${\left[a\left(x\right)\right]}^{|{\mathbb{B}}_n|}\equiv 1(\mathrm{mod} p),\forall \left[a\left(x\right)\right]\in {\mathbb{B}}_n.$$

Observe that if $N=pq$ is the product of two prime numbers, and $\mathbb{F}=\mathbb{Z}/N\mathbb{Z}$, we obtain

$$|{\mathbb{B}}_n| ={\displaystyle \frac{(p^n-1)(q^n-1)}{(p-1)(q-1)}}.$$

Furthermore, for every $\left[a\left(x\right)\right]\in {\mathbb{B}}_n$, we also have

$${\left[a\left(x\right)\right]}^{|{\mathbb{B}}_n|}\equiv 1(\mathrm{mod}N).$$

The scheme of Cotan and Teşeleanu can be summarized as follows.

Key Generation

1.

Select a positive integer $n>1$ and a security size $\lambda >0$.

2.

Generate randomly two distinct large prime numbers of size $\lambda$.

3.

Calculate $N=pq$ and ${\psi }_n\left(N\right)=\frac{(p^n-1)(q^n-1)}{(p-1)(q-1)}$.

4.

Choose an integer a for which $x^n-a$ is irreducible in $\mathbb{Z}/p\mathbb{Z}\left[x\right]$, $\mathbb{Z}/q\mathbb{Z}\left[x\right]$, and $\mathbb{Z}/N\mathbb{Z}\left[x\right]$.

5.

Select an integer e such that $gcd(e,{\psi }_n\left(N\right))=1$ and compute d, the inverse of e modulo ${\psi }_n\left(N\right)$.

6.

The public key is $(N,n,a,e)$ and the private key is $(p,q,d)$.

Encryption

1.

Represent the plaintext as a polynomial

$$m\left(x\right)=m_0+m_{1}x+\dots +m_{n-2}{x}^{n-2}+x^{n-1}\in {\mathbb{B}}_n.$$

2.

Compute $c\left(x\right)\equiv {\left[m\left(x\right)\right]}^e(\mathrm{mod} N)$.

3.

The ciphertext is $c\left(x\right)$.

Decryption

To recover the plaintext $m\left(x\right)$, one needs to compute

$$m\left(x\right)\equiv {\left[c\left(x\right)\right]}^d(\mathrm{mod} N).$$

3. Solving the Equation $\mathit{xH}\left(\mathit{y}\right)+\mathit{c}\equiv \mathbf{0}(\mathbf{mod} \mathit{e})$

In this section, we propose a new technique to find the small solutions of the modular equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ where c is a constant, and $H\left(y\right)\in \mathbb{Z}\left[y\right]$ is a monic polynomial of degree r. The equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ was previously studied by Kunihiro [21] and recently by Nitaj et al. [14]. In both works, the value $x{y}^r$ is replaced by $z-c$, and the assumption ${\left|c\right|<\left|x\right|\left|y\right|}^r$ is used. In this paper, we present a different method where $x{y}^r$ is independent of c. This relaxes the condition ${\left|c\right|<\left|x\right|\left|y\right|}^r$ used in [14,21], and it permits more applications in the cryptanalysis of some variants of RSA.

3.1. The New Method

Theorem 3.

Let $N=pq$ be an RSA modulus with $q<p<2q$. Let $H\left(y\right)\in \mathbb{Z}\left[y\right]$ be a monic polynomial of degree $r\ge 1$. If $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ with $e=N^{\alpha }$, $\left|x\right|\le N^{\beta }$, $\left|y\right|\le N^{\gamma }$, ${\left|x\right|\left|y\right|}^r<e$, and

$$\begin{array}{c}\hfill \beta <\alpha +{\displaystyle \frac{1}{3}}r\gamma -{\displaystyle \frac{2}{3}}\sqrt{3r\alpha \gamma +r^2{\gamma }^2},\end{array}$$

then one can find x and y in polynomial time.

Proof. 

Let $f(x,y)=xH\left(y\right)+c$ with $H\left(y\right)=y^r+a_{r-1}{y}^{r-1}+\dots +a_0\in \mathbb{Z}\left[y\right]$. We use Coppersmith’s technique [17] and the strategy of Jochemsz and May [19] to find the small solutions of the equation $f(x,y)\equiv 0(\mathrm{mod}e)$. Let m be a positive integer and t be a positive value. For $0\le k\le m$, consider the set

$$\begin{array}{cc}\hfill M_k=\bigcup _{0\le j^{\prime }\le \lfloor t\rfloor }\{x^i{y}^{j+j^{\prime }}|& x^i{y}^j\mathrm{is}\mathrm{a}\mathrm{monomial}\mathrm{of}{f}^m(x,y)\hfill \\ & \mathrm{and}{\displaystyle \frac{x^i{y}^j}{{\left(x{y}^r\right)}^k}}\mathrm{is}\mathrm{a}\mathrm{monomial}\mathrm{of}{f}^{m-k}(x,y)\}.\hfill \end{array}$$

A direct computation shows that the monomials $x^i{y}^j$ of $f^m(x,y)$ are composed by the couples $(i,j)$ with

$$i=0,\dots ,m,j=0,\dots ,ri.$$

Also, the monomials $x^i{y}^j$ of $f^{m-k}(x,y)$ are composed by $(i,j)$ with

$$i=0,\dots ,m-k,j=0,\dots ,ri.$$

This implies that the monomials $x^i{y}^j$ of $M_k$ are composed by $(i,j)$ with

$$i-k=0,\dots ,m-k,j-rk=0,\dots ,r(i-k)+\lfloor t\rfloor ,$$

or equivalently

$$i=k,\dots ,m,j=rk,\dots ,ri+\lfloor t\rfloor$$

In the strategy of Jochemsz and May [19], we need to form the set $M_k\setminus M_{k+1}$. Since $M_{k+1}$ is composed by the monomials $x^i{y}^j$ with

$$i=k+1,\dots ,m,j=rk+r,\dots ,ri+\lfloor t\rfloor ,$$

then $M_k\setminus M_{k+1}$ is the set of the monomials $x^i{y}^j$ composed by

$$\begin{array}{cc}\hfill & i=k+1,\dots ,m,j=rk,rk+1,\dots ,rk+r-1,\hfill \\ \hfill & i=k,j=rk,\dots ,rk+\lfloor t\rfloor .\hfill \end{array}$$

As in the strategy of Jochemsz and May, consider the list of polynomials

$$g_{k,i,j}(x,y)={\displaystyle \frac{x^i{y}^j}{{\left(x{y}^r\right)}^k}}f{(x,y)}^k{e}^{m-k},x^i{y}^j\in M_k\setminus M_{k+1}.$$

These polynomials reduce to

$$\begin{array}{cc}\hfill & g_{k,i,j}(x,y)=x^i{y}^{j}f{(x,y)}^k{e}^{m-k},\hfill \\ \hfill & i=1,\dots ,m-k,j=0,\dots r-1,\hfill \\ \hfill & i=0,j=0,\dots ,\lfloor t\rfloor .\hfill \end{array}$$

Using $f(x,y)=xH\left(y\right)+c=x{y}^r+x\left(a_{r-1}{y}^{r-1}+\dots +a_0\right)+c$, we set $x{y}^r=z$, and $F(x,y,z)=z+x\left(a_{r-1}{y}^{r-1}+\dots +a_0\right)+c$. Then, the polynomials $g_{k,i,j}(x,y)$ can be transformed into the following ones,

$$\begin{array}{cc}\hfill & G_{k,i,j}(x,y,z)=x^i{y}^{j}F{(x,y,z)}^k{e}^{m-k},\hfill \\ \hfill & k=0,\dots ,m,i=1,\dots ,m-k,j=0,\dots ,r-1,\hfill \\ \hfill & k=0,\dots ,m,i=0,j=0,\dots ,\lfloor t\rfloor ,\hfill \end{array}$$

where each term $x{y}^r$ is replaced by z.

Let $(x_0,y_0)$ be a solution of the equation $f(x,y)\equiv 0(\mathrm{mod}e)$, and $z_0=x_0{y}_0^r$. Then, $(x_0,y_0,z_0)$ is a solution of the equation $F(x,y,z)\equiv 0(\mathrm{mod}e)$, and the polynomials $G_{k,i,j}(x,y,z)$ satisfy $G_{k,i,j}(x_0,y_0,z_0)\equiv 0(\mathrm{mod}{e}^m)$.

Define the bounds

$$X=N^{\beta },Y=N^{\gamma },Z=N^{\beta +r\gamma },$$

and assume that the solution $(x_0,y_0,z_0)$ satisfies $|x_0| \le X$, $|y_0| \le Y$, $|z_0| \le Z$. Following Coppersmith’s method, we use the coefficient vectors of the polynomials $G_{k,i,j}(Xx,Yy,Zz)$ to form a matrix which is used as the basis matrix of a lattice $\mathcal{L}$. In this matrix, the rows are ordered so that $G_{k,i,j}(Xx,Yy,Zz)\prec G_{k^{\prime },i^{\prime },j^{\prime }}(Xx,Yy,Zz)$ if $k<k^{\prime }$, or if $k=k^{\prime }$ and $i<i^{\prime }$, or if $k=k^{\prime }$, $i=i^{\prime }$, and $j<j^{\prime }$. Similarly, the monomials are ordered so that $z^k{x}^i{y}^j\prec z^{k^{\prime }}{x}^{i^{\prime }}{y}^{j^{\prime }}$ if $k<k^{\prime }$, or if $k=k^{\prime }$ and $i<i^{\prime }$, or if $k=k^{\prime }$, $i=i^{\prime }$, and $j<j^{\prime }$. In

Table 1. The matrix of the lattice for $m=2$, $t=1$ with the polynomial $H\left(y\right)=y^3+a_2{y}^2+a_{1}y+a_0$.

${\mathit{G}}_{\mathit{k},\mathit{i},\mathit{j}}$	1	y	x	$\mathit{xy}$	${\mathit{xy}}^2$	${\mathit{x}}^2$	${\mathit{x}}^2\mathit{y}$	${\mathit{x}}^2{\mathit{y}}^2$	z	$\mathit{yz}$	$\mathit{xz}$	$\mathit{xyz}$	${\mathit{xy}}^2\mathit{z}$	${\mathit{z}}^2$	${\mathit{yz}}^2$
$G_{0,0,0}$	$e^2$	0	0	0	0	0	0	0	0	0	0	0	0	0	0
$G_{0,0,1}$	0	$e^{2}Y$	0	0	0	0	0	0	0	0	0	0	0	0	0
$G_{0,1,0}$	0	0	$e^{2}X$	0	0	0	0	0	0	0	0	0	0	0	0
$G_{0,1,1}$	0	0	0	$e^{2}XY$	0	0	0	0	0	0	0	0	0	0	0
$G_{0,1,2}$	0	0	0	0	$e^{2}X{Y}^2$	0	0	0	0	0	0	0	0	0	0
$G_{0,2,0}$	0	0	0	0	0	$e^2{X}^2$	0	0	0	0	0	0	0	0	0
$G_{0,2,1}$	0	0	0	0	0	0	$e^2{X}^{2}Y$	0	0	0	0	0	0	0	0
$G_{0,2,2}$	0	0	0	0	0	0	0	$e^2{X}^2{Y}^2$	0	0	0	0	0	0	0
$G_{1,0,0}$	★	0	★	★	★	0	0	0	$Ze$	0	0	0	0	0	0
$G_{1,0,1}$	0	★	0	★	★	0	0	0	★	$YZe$	0	0	0	0	0
$G_{1,1,0}$	0	0	★	0	0	★	★	★	0	0	$XZe$	0	0	0	0
$G_{1,1,1}$	0	0	0	★	0	0	★	★	0	0	★	$XYZe$	0	0	0
$G_{1,1,2}$	0	0	0	0	★	0	0	★	0	0	★	★	$X{Y}^{2}Ze$	0	0
$G_{2,0,0}$	★	0	★	★	★	★	★	★	★	0	★	★	★	$Z^2$	0
$G_{2,0,1}$	0	★	0	★	★	0	★	★	★	★	★	★	★	★	$Y{Z}^2$

, we present an example of the matrix of the lattice for $m=2$, $t=1$ where the symbols ★ are non-zero entries.

By construction, the matrix of the lattice is triangular, and its determinant is the product of the diagonal terms

$$\begin{array}{c}\hfill det\left(\mathcal{L}\right)=X^{n_X}{Y}^{n_Y}{Z}^{n_Z}{e}^{n_e}.\end{array}$$

(1)

To compute the former exponents, consider the function

$$S\left(v\right)=\sum _{k=0}^m\sum _{i=1}^{m-k}\sum _{j=0}^{r-1}v+\sum _{k=0}^m\sum _{i=0}^0\sum _{j=0}^{\lfloor t\rfloor }v.$$

Set $t=m\tau$ for $\tau \ge 0$. To ease the computations, we take $\lfloor m\tau \rfloor \approx m\tau$. The dominant parts of the exponents $n_X$, $n_Y$, $n_Z$, $n_e$ as well as of the dimension $\omega$ of the lattice satisfy

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill n_X& =S\left(i\right)=\frac{1}{6}r{m}^3+o\left(m^3\right)\hfill \\ \hfill n_Y& =S\left(j\right)=\frac{1}{2}{\tau }^2{m}^3+o\left(m^3\right)\hfill \\ \hfill n_Z& =S\left(k\right)=\frac{1}{6}(3\tau +r)m^3+o\left(m^3\right)\hfill \\ \hfill n_e& =S(m-k)=\frac{1}{6}(3\tau +2r)m^3+o\left(m^3\right)\hfill \\ \hfill \omega & =S\left(1\right)=\frac{1}{2}(2\tau +r)m^2+o\left(m^2\right).\hfill \end{array}\end{array}$$

(2)

After applying the LLL algorithm to the matrix of the lattice $\mathcal{L}$, we obtain a reduced matrix from which we can extract $\omega$ new polynomials $h_{k,i,j}(x,y,z)$. To combine Theorems 1 and 2 with $i=3$, we set

$$2^{\frac{\omega (\omega -1)}{4(\omega -2)}}det{\left(\mathcal{L}\right)}^{\frac{1}{\omega -2}}<\frac{e^m}{\sqrt{\omega }}.$$

Using (1), this reduces to

$$\begin{array}{c}\hfill e^{n_e-m(\omega -2)}{X}^{n_X}{Y}^{n_Y}{Z}^{n_Z}<{\displaystyle \frac{2^{-\frac{\omega (\omega -1)}{4}}}{{\left(\sqrt{\omega }\right)}^{\omega -2}}}.\end{array}$$

(3)

Using the dominant parts (2) with $X=N^{\beta }$, $Y=N^{\gamma }$, $Z=N^{\beta +r\gamma }$, and $e=N^{\alpha }$, we obtain, after neglecting some small terms

$$\left({\displaystyle \frac{1}{6}}(3\tau +2r)-{\displaystyle \frac{1}{2}}(2\tau +r)\right)\alpha +{\displaystyle \frac{1}{6}}r\beta +{\displaystyle \frac{1}{2}}\gamma {\tau }^2+{\displaystyle \frac{1}{6}}(3\tau +r)(\beta +r\gamma )<0.$$

Rearranging, we obtain

$$\begin{array}{c}\hfill 3\gamma {\tau }^2+3(r\gamma -\alpha +\beta )\tau +r^2\gamma -r\alpha +2r\beta <0,\end{array}$$

(4)

in which the optimal value for $\tau$ is ${\tau }_0=\frac{\alpha -\beta -r\gamma }{2\gamma }.$ Since ${e>\left|x\right|\left|y\right|}^r$, then $\alpha >\beta +r\gamma$, and ${\tau }_0>0$. Then, plugging ${\tau }_0$ in (4), we obtain

$$-3{\beta }^2+(6\alpha +2r\gamma )\beta -3{\alpha }^2+2r\gamma \alpha +r^2{\gamma }^2<0,$$

which leads to

$$\begin{array}{c}\hfill \beta <\alpha +{\displaystyle \frac{1}{3}}r\gamma -{\displaystyle \frac{2}{3}}\sqrt{3r\alpha \gamma +r^2{\gamma }^2}.\end{array}$$

We notice that the former bound is positive since $\alpha >\beta +r\gamma$. Under this bound, using three reduced polynomials $h_1(x,y,z)$, $h_2(x,y,z)$, $h_3(x,y,z)$, we can extract the solution $(x_0,y_0,z_0)$ by the Gröbner basis method or resultant computations. This terminates the proof. □

3.2. A Numerical Example

In this section, we present a small numerical example to show the details of the resolution method of Theorem 3 with $n=4$, and $r=n-1=3$. Consider the following parameters

$$\begin{array}{cc}\hfill N=& 463028995904606051817018641173,\hfill \\ \hfill c=& 895087879645377698399589802186741096954354552299285\setminus \hfill \\ \hfill & 87492654228177046463498977617360027022,\hfill \\ \hfill e=& 172459409963116822030248732348419638390904926885797\setminus \hfill \\ \hfill & 13115090719406582906246851863033916922.\hfill \end{array}$$

Then, $e=N^{\alpha }$ with $\alpha \approx 2.97437$, and $p+q<3\sqrt{N}$, so that $y<3N^{\gamma }$ with $\gamma =\frac{1}{2}$. Set $\beta =\frac{1}{2}$. Then, the conditions of Theorem 3 are satisfied since $\alpha >\beta +r\gamma =2$, and $\beta <\alpha +\frac{1}{3}r\gamma -\frac{2}{3}\sqrt{3r\alpha \gamma +r^2{\gamma }^2}\approx 0.838$. The goal is to find a small solution $(x_0,y_0)$ of the equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ where $H\left(y\right)$ is derived from

$${\psi }_4\left(N\right)={(p+q)}^3+(N+1){(p+q)}^2+\left(N^2-2N+1\right)(p+q)+N^3-N^2-N+1,$$

with $p+q=y$, that is

$$H\left(y\right)=y^3+(N+1)y^2+\left(N^2-2N+1\right)y+N^3-N^2-N+1.$$

Consider the bounds $|x_0| \le X$, $|y_0| \le Y$, and $|x_0{y}_0^3| \le Z$ with

$$\begin{array}{cc}\hfill & X=⌊N^{0.5}⌋=680462339813605,\hfill \\ \hfill & Y=⌊3N^{0.5}⌋=2041387019440815,\hfill \\ \hfill & Z=X{Y}^3=578868797830754738565836771991739782740725532698185\setminus \hfill \\ \hfill & 4011616875.\hfill \end{array}$$

Let $m=4$, $t=2$, and

$$F(x,y,z)=z+x\left((N+1)y^2+\left(N^2-2N+1\right)y+N^3-N^2-N+1\right)+c.$$

The lattice $\mathcal{L}$ is constructed with the coefficients of the polynomials defined by

$$\begin{array}{cc}\hfill & G_{k,i,j}(x,y,z)=x^i{y}^{j}F{(x,y,z)}^k{e}^{m-k},\hfill \\ \hfill & k=0,\dots ,m,i=1,\dots ,m-k,j=0,\dots ,r-1,\hfill \\ \hfill & k=0,\dots ,m,i=0,j=0,\dots ,\lfloor t\rfloor ,\hfill \end{array}$$

where each term $x{y}^r$ is replaced by z. The dimension of the lattice is $\omega =45$. After reducing the lattice with the LLL algorithm, and solving a system formed by three polynomial equations over the integers with the Gröbner basis method, we find the solution

$$\begin{array}{cc}\hfill & x_0=16165734257585,\hfill \\ \hfill & y_0=1360935721901674,\hfill \\ \hfill & z_0=40748185648950035910680304028872647558518309799826755032040.\hfill \end{array}$$

Using $p+q=y_0$ and $pq=N$, we obtain

$$\begin{array}{c}\hfill p=683209007134751,q=677726714766923,\end{array}$$

and the factorization of N is complete. Notice that $\frac{c}{x_0{y}_0^3}>{10}^{30}$, and c is much larger than $\left|x_0{y}_0^3\right|$. This shows that the methods described in [14,21] cannot be applied to solve the equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$.

4. Partial Key Attack on the Scheme of Cotan and Teşeleanu with Known LSBs

In this section, we apply Theorem 3 to attack the scheme of Cotan and Teşeleanu when the attacker knows the s least significant bits (LSBs) of d so that $d=d_{1}M+d_0$ for $M=2^s$, with known $d_0$, and unknown $d_1$.

Theorem 4.

Let $n\ge 2$, and $N=pq$ be the product of two unknown prime factors with $q<p<2q$. Let $e=N^{\alpha }$, and $d\le N^{\delta }$ such that $ed\equiv 1(\mathrm{mod}{\psi }_n\left(N\right))$ with ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$. Let M and $d_0$ be two known integers such that $d=d_{1}M+d_0$ with $M=N^{\mu }$. Then, one can factor N in polynomial time if

$$\delta <\mu +{\displaystyle \frac{7}{6}}(n-1)-{\displaystyle \frac{1}{3}}\sqrt{6(n-1)(\alpha +\mu )+{(n-1)}^2}.$$

Proof. 

In the equation $ed-k{\psi }_n\left(N\right)=1$, assume that $d=d_{1}M+d_0$ where M and $d_0$ are known, and $d_1$ is unknown. We assume the following bounds

$$\begin{array}{c}\hfill e=N^{\alpha },M=N^{\mu },d\le N^{\delta }.\end{array}$$

We rewrite the equation $ed-k{\psi }_n\left(N\right)=1$ as

$$k{\psi }_n\left(N\right)-e{d}_0+1=e{d}_{1}M,$$

where by Lemma 5, ${\psi }_n\left(N\right)={(p+q)}^{n-1}+{\sum }_{j=0}^{n-2}{a}_j{(p+q)}^j$ with known coefficients $a_j$, $j=0,\dots ,n-2$. Let $H\left(y\right)=y^{n-1}+{\sum }_{j=0}^{n-2}{a}_j{y}^j$, and consider the polynomial

$$f(x,y)=xH\left(y\right)-e{d}_0+1.$$

Then, $(x_0,y_0)=(k,p+q)$ satisfies $f(x_0,y_0)\equiv 0(\mathrm{mod}eM)$. By Lemma 1, we have $y_0<3\sqrt{N}$. Also, we have

$$x_0=k={\displaystyle \frac{ed-1}{{\psi }_n\left(N\right)}}<N^{\alpha +\delta -n+1}.$$

We can then apply Theorem 3 where $\alpha$ is replaced by $\alpha +\mu$, $\beta$ is replaced by $\alpha +\delta -n+1$, $\gamma =\frac{1}{2}$, and $r=n-1$. Then, the inequality $\beta <\alpha +\frac{1}{3}r\gamma -\frac{2}{3}\sqrt{3r\alpha \gamma +r^2{\gamma }^2}$ in Theorem 3 leads to

$$\delta <\mu +{\displaystyle \frac{7}{6}}(n-1)-{\displaystyle \frac{1}{3}}\sqrt{6(n-1)(\alpha +\mu )+{(n-1)}^2}.$$

After finding the solutions of the equation $f(x,y)\equiv 0(\mathrm{mod}eM)$, only one satisfies $(x_0,y_0)=(k,p+q)$. Then, combining $y_0=p+q$, and $N=pq$, this leads to the factorization of N and terminates the proof. □

5. Cryptanalysis of the Scheme of Cotan and Teşeleanu with a Known Approximation of One of the Primes

In this section, we consider the scheme of Cotan and Teşeleanu with $N=pq$ when $p<q<2q$, and an approximation $p_0$ of p is known.

Theorem 5.

Let $n\ge 2$, and $N=pq$ be the product of two unknown prime factors with $q<p<2q$. Suppose that $ed-k{\psi }_n\left(N\right)=1$ with ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$, $e=N^{\alpha }$, and $d\le N^{\delta }$. Let $p_0$ be an approximation of p with $|p - p_0| <N^{\mu }$. Then, one can factor N in polynomial time if

$$\delta <\left(1+{\displaystyle \frac{1}{3}}\mu \right)(n-1)-{\displaystyle \frac{2}{3}}\sqrt{3(n-1)\mu \alpha +{(n-1)}^2{\mu }^2}.$$

Proof. 

Suppose that $ed-k{\psi }_n\left(N\right)=1$ with $e=N^{\alpha }$ and $d\le N^{\delta }$. This implies that $k{\psi }_n\left(N\right)+1\equiv 0(\mathrm{mod}e)$. Let $p_0$ be an approximation of p with $|p - p_0| <N^{\gamma }$. Then, by Lemma 2, the integer $q_0=⌊\frac{N}{p_0}⌋$ is an approximation of q such that $|q - q_0| <N^{\mu }$ and $|p+q - p_0-q_0| <2N^{\mu }$. Set $M=p_0+q_0$. By Lemma 6, one has ${\psi }_n\left(N\right)={(p+q-M)}^{n-1}+{\sum }_{j=0}^{n-2}{a}_j^{\left(n\right)}{(p+q-M)}^j$. Then, the equation $k{\psi }_n\left(N\right)+1\equiv 0(\mathrm{mod}e)$ can be rewritten as

$$k\left({(p+q-M)}^{n-1}+\sum _{j=0}^{n-2}{a}_j^{\left(n\right)}{(p+q-M)}^j\right)+1\equiv 0(\mathrm{mod}e).$$

Consider the polynomial $F(x,y)=xH\left(y\right)+1$ with $H\left(y\right)=y^{n-1}+{\sum }_{j=0}^{n-2}{a}_j^{\left(n\right)}{y}^j$. Then, $(x_0,y_0)=(k,p+q-M)$ is a solution of the modular polynomial equation $F(x,y)\equiv 0(\mathrm{mod}e)$. Using $ed-k{\psi }_n\left(N\right)=1$, $e=N^{\alpha }$, $d\le N^{\delta }$, and since ${\psi }_n\left(N\right)>p^{n-1}{q}^{n-1}=N^{n-1}$, we obtain

$$k={\displaystyle \frac{ed-1}{{\psi }_n\left(N\right)}}<{\displaystyle \frac{N^{\alpha +\delta }}{N^{n-1}}}=N^{\alpha +\delta -n+1}.$$

Let $X=N^{\alpha +\delta -n+1}$ and $Y=N^{\gamma }$. Then, using $r=n-1$, $\gamma =\mu$, and $\beta =\alpha +\delta -n+1$ in Theorem 3, we obtain

$$\delta <\left(1+{\displaystyle \frac{1}{3}}\mu \right)(n-1)-{\displaystyle \frac{2}{3}}\sqrt{3(n-1)\mu \alpha +{(n-1)}^2{\mu }^2}.$$

After finding the solutions of the equation $F(x,y)\equiv 0(\mathrm{mod}e)$, only one satisfies $(x_0,y_0)=(k,p+q-M)$. Then, combining $y_0+M=p+q$, and $N=pq$, this leads to the factorization of N and terminates the proof. □

6. Cryptanalysis of the Scheme of Cotan and Teşeleanu with Primes Sharing MSBs

The following result is a direct application of Theorem 5. It concerns the case of a modulus $N=pq$ where the prime difference $|p-q|$ is small.

Corollary 1.

Let $n\ge 2$ and $N=pq$ be the product of two unknown prime factors with $q<p<2q$ and $p-q<N^{\mu }$. Suppose that $ed-k{\psi }_n\left(N\right)=1$ with ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$, $e=N^{\alpha }$, and $d\le N^{\delta }$. Then, one can factor N in polynomial time if

$$\delta <\left(1+{\displaystyle \frac{1}{3}}\mu \right)(n-1)-{\displaystyle \frac{2}{3}}\sqrt{3(n-1)\mu \alpha +{(n-1)}^2{\mu }^2}.$$

Proof. 

Suppose that $p-q<N^{\mu }$. Since, by Lemma 1, we have $q<\sqrt{N}<p$, one obtains

$$0<p-\sqrt{N}<p-q<N^{\mu }.$$

This implies that $p_0=\sqrt{N}$ is an approximation of p such that $|p - p_0| <N^{\mu }$. Then, using Theorem 5, one can factor $N=pq$ if

$$\delta <\left(1+{\displaystyle \frac{1}{3}}\mu \right)(n-1)-{\displaystyle \frac{2}{3}}\sqrt{3(n-1)\mu \alpha +{(n-1)}^2{\mu }^2}.$$

This terminates the proof. □

7. Cryptanalysis of the Scheme of Cotan and Teşeleanu with Primes Sharing LSBs

In this section, we propose an attack on the scheme of Cotan and Teşeleanu when the prime factors share an amount of their least significant bits.

Let $N=pq$ be an RSA modulus with $q<p<2q$. Suppose that p and q share their least significant bits so that $p-q=2^{s}u$ for a known s and an unknown u. Then, the following result shows that one can find the s least significant bits of p and q and the $2s$ least significant bits of $p+q$ (see [22,23]).

Lemma 7.

Let $N=pq$ be an RSA modulus with $q<p<2q$. Suppose that $p-q=2^{s}u$ with a known s and an unknown u. Let $u_0$ be a solution of the equation $z^2\equiv N(\mathrm{mod}{2}^s)$ and

$$v_0\equiv 2u_0+\left(N-u_0^2\right)u_0^{-1}(\mathrm{mod}{2}^{2s}).$$

Then, $p=2^s{p}_1+u_0$, $q=2^s{q}_1+u_0$, and $p+q=2^{2s}v+v_0$ for some integers $p_1$, $q_1$, and v.

For $p+q=2^{2s}v+v_0$, the following Lemma shows that ${\psi }_n\left(N\right)$ can be expressed as a polynomial in v with integer coefficients.

Lemma 8.

Let $N=pq$, $n\ge 2$, ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$, with $p+q=2^{2s}v+v_0$. Then, there exist $n-1$ coefficients $b_j^{\left(n\right)}\in \mathbb{Z}$, $j=0,\dots ,n-2$, depending only on N, n, s, and $v_0$ such that

$${\psi }_n\left(N\right)=2^{2s(n-1)}{v}^{n-1}+\sum _{j=0}^{n-2}{b}_j^{\left(n\right)}{v}^j.$$

Proof. 

Since $p+q=2^{2s}v+v_0$, then $p+q-v_0=2^{2s}v$. Then, by Lemma 6, with $M=v_0$, there exist $n-1$ integers $a_j^{\left(n\right)}$, $j=0,\dots ,n-2$, such that

$${\psi }_n\left(N\right)={(p+q-v_0)}^{n-1}+\sum _{j=0}^{n-2}{a}_j^{\left(n\right)}{(p+q-v_0)}^j.$$

Then,

$$\begin{array}{cc}\hfill {\psi }_n\left(N\right)& ={\left(2^{2s}v\right)}^{n-1}+\sum _{j=0}^{n-2}{a}_j^{\left(n\right)}{\left(2^{2s}v\right)}^j\hfill \\ \hfill & =2^{2s(n-1)}{v}^{n-1}+\sum _{j=0}^{n-2}{2}^{2sj}{a}_j^{\left(n\right)}{v}^j\hfill \\ \hfill & =2^{2s(n-1)}{v}^{n-1}+\sum _{j=0}^{n-2}{b}_j^{\left(n\right)}{v}^j,\hfill \end{array}$$

where $b_j=2^{2sj}{a}_j^{\left(n\right)}$, $j=0,\dots ,n-2$. This terminates the proof. □

The following result concerns the situation where the prime factors p and q share their least significant bits.

Theorem 6.

Let $n\ge 2$ and $N=pq$ be an RSA modulus with $q<p<2q$. Suppose that $e=N^{\alpha }$ is odd and satisfies the equation $ed-k{\psi }_n\left(N\right)=1$ with ${\psi }_n\left(N\right)=\frac{\left(p^n-1\right)\left(q^n-1\right)}{(p-1)(q-1)}$ and $d\le N^{\delta }$. Suppose that p and q share their s least significant bits with $2^s=N^{\mu }$. If

$$\begin{array}{c}\hfill \delta <\left({\displaystyle \frac{7}{6}}-{\displaystyle \frac{2}{3}}\mu \right)(n-1)-{\displaystyle \frac{2}{3}}\sqrt{3(n-1)\alpha \left({\displaystyle \frac{1}{2}}-2\mu \right)+{(n-1)}^2{\left({\displaystyle \frac{1}{2}}-2\mu \right)}^2}.\end{array}$$

then one can factor N in polynomial time.

Proof. 

Assume that p and q share their least significant bits so that $p-q=2^{s}v$. Let $u_0$ be a solution of the equation $z^2\equiv N(\mathrm{mod}{2}^s)$ and,

$$v_0\equiv 2u_0+\left(N-u_0^2\right)u_0^{-1}(\mathrm{mod}{2}^{2s}).$$

Then, by Lemma 7, we have $p=2^s{p}_1+u_0$, $q=2^s{q}_1+u_0$, and $p+q=2^{2s}v+v_0$. The equation $ed-k{\psi }_n\left(N\right)=1$ can be rewritten as $k{\psi }_n\left(N\right)+1\equiv 0(\mathrm{mod}e)$, and by Lemma 8, we have

$${\psi }_n\left(N\right)=2^{2s(n-1)}{v}^{n-1}+\sum _{j=0}^{n-2}{b}_j^{\left(n\right)}{v}^j.$$

Suppose that e is odd. Then, $gcd(2,e)=1$, and the equation $k{\psi }_n\left(N\right)+1\equiv 0(\mathrm{mod}e)$ can be rewritten as

$$k\left(v^{n-1}+\sum _{j=0}^{n-2}{b}_j^{\left(n\right)}{2}^{-2s(n-1)}{v}^j\right)+2^{-2s(n-1)}\equiv 0(\mathrm{mod}e),$$

where $2^{-2s(n-1)}$ is the inverse of $2^{2s(n-1)}$ modulo e. Consider the polynomial $F(x,y)=xH\left(y\right)+c$ where $H\left(y\right)=y^{n-1}+{\sum }_{j=0}^{n-2}{b}_j^{\left(n\right)}{2}^{-2s(n-1)}{y}^j(\mathrm{mod}e)$, and $c\equiv 2^{-2s(n-1)}(\mathrm{mod}e)$. Then, $(x_0,y_0)=(k,v)$ is a solution of the equation $F(x,y)\equiv 0(\mathrm{mod}e)$. Theorem 3 can then be applied to find the small solutions. Assume that $e=N^{\alpha }$, $d\le N^{\delta }$, and $2^s=N^{\mu }$. Then, using $ed-k{\psi }_n\left(N\right)=1$, we obtain

$$k={\displaystyle \frac{ed-1}{{\psi }_n\left(N\right)}}<N^{\alpha +\delta -n+1}.$$

Also, using $p+q=2^{2s}v+v_0<3\sqrt{N}$, we obtain

$$v=\frac{p+q-v_0}{2^{2s}}<3N^{\frac{1}{2}-2\mu }.$$

Observe that $\frac{1}{2}-2\mu >0$. Otherwise, one obtains $v\le 2$, that is $p+q=2^{2s}v+v_0$ with $v\in \{1,2\}$. This leads to the factorization of N.

Let $X=N^{\alpha +\delta -n+1}$, and $Y=3N^{\frac{1}{2}-2\mu }$. Then, applying Theorem 3 with $\beta =\alpha +\delta -n+1$, $\gamma =\frac{1}{2}-2\mu$, and $r=n-1$, we can find the solution $(x_0,y_0)=(k,v)$ if

$$\begin{array}{c}\hfill \delta <\left({\displaystyle \frac{7}{6}}-{\displaystyle \frac{2}{3}}\mu \right)(n-1)-{\displaystyle \frac{2}{3}}\sqrt{3(n-1)\alpha \left({\displaystyle \frac{1}{2}}-2\mu \right)+{(n-1)}^2{\left({\displaystyle \frac{1}{2}}-2\mu \right)}^2}.\end{array}$$

Using $N=pq$ and $v=y_0$, we obtain $p+q=2^{2s}v+v_0$. This leads to the factorization of N. □

8. Conclusions

In this paper, we proposed a new technique to solve the modular equation $xH\left(y\right)+c\equiv 0(\mathrm{mod}e)$ for small unknown integers x, y, and for an arbitrary value of c where $H\left(y\right)\in \mathbb{Z}\left[y\right]$ is a monic polynomial of degree $r\ge 1$. The methodology is based on Coppersmith’s method and lattice basis reduction. It finds the solutions in contrast to the former methods which fail when $\left|c\right|\ge |x{y}^r|$. As an application of our method, we present four attacks on the scheme of Cotan and Teşeleanu, namely a partial key exposure attack with known least significant bits, a partial prime exposure attack, and two attacks when the prime factors share their least or most significant bits.