Why Dilated Convolutional Neural Networks: A Proof of Their Optimality

Abstract

One of the most effective image processing techniques is the use of convolutional neural networks that use convolutional layers. In each such layer, the value of the layer’s output signal at each point is a combination of the layer’s input signals corresponding to several neighboring points. To improve the accuracy, researchers have developed a version of this technique, in which only data from some of the neighboring points is processed. It turns out that the most efficient case—called dilated convolution—is when we select the neighboring points whose differences in both coordinates are divisible by some constant ℓ. In this paper, we explain this empirical efficiency by proving that for all reasonable optimality criteria, dilated convolution is indeed better than possible alternatives.

1. Introduction

1.1. Convolutional Layers: Input and Outpu

At present, one of the most efficient techniques in image processing and in other areas is a convolutional neural network; see, e.g., [1]. Convolutional neural networks include special types of layers that perform linear transformations.

Each such layer is characterized by integer-values parameters $\underline{X}\le \overline{X}$, $\underline{Y}\le \overline{Y}$, $d_{\mathrm{in}}\ge 1$, and $d_{\mathrm{out}}\ge 1$; then:

• the input to this layer consists of the values $F_{d^{\prime }}(x^{\prime },y^{\prime })$, where $d^{\prime }$, $x^{\prime }$, and $y^{\prime }$ are integers for which $\underline{X}\le x^{\prime }\le \overline{X}$, $\underline{Y}\le y^{\prime }\le \overline{Y}$, and $1\le d^{\prime }\le d_{\mathrm{in}}$; and
• the output of this layer consists of the values $G_d(x,y)$, where d, x, and y are integers for which $\underline{X}\le x\le \overline{X}$, $\underline{Y}\le y\le \overline{Y}$, and $1\le d\le d_{\mathrm{out}}$.

1.2. Convolutional Layer: Transformation

A general linear transformation has the form

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{in}}\left(\sum _{x^{\prime }=\underline{X}}^{\overline{X}}\sum _{y^{\prime }=\underline{Y}}^{\overline{Y}}{K}_d(x,x^{\prime },y,y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime },y^{\prime })\right),$$

(1)

for some coefficients $K_d(x,x^{\prime },y,y^{\prime },d^{\prime })$.

Transformations performed by a convolutional layer are a specific case of such generic linear transformations, where the following two restrictions are imposed:

• first, each value $G_d(x,y)$ depends only on the values $F_{d^{\prime }}(x^{\prime },y^{\prime })$, for which both differences $|x-x^{\prime }|$ and $|y-y^{\prime }|$ do not exceed some fixed integer L, and
• the coefficients $K_d(x,x^{\prime },y,y^{\prime },d^{\prime })$ depend only on the differences $x-x^{\prime }$ and $y-y^{\prime }$:

  $$K_d(x,x^{\prime },y,y^{\prime },d^{\prime })=k_d(x-x^{\prime },y-y^{\prime },d^{\prime })$$

  (2)

  for some coefficients $k_d(i,j,d^{\prime })$ defined for all pairs $(i,j)$ for which $\left|i\right|,\left|j\right|\le L$.

The values $k_d(i,j,d^{\prime })$ are known as a filter.

The resulting linear transformation takes the form

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{-L\le i,j\le L}{k}_d(i,j,d^{\prime })·F_{d^{\prime }}(x-i,y-j)\right).$$

(3)

Thus, the output $G_d(x,y)$ of a convolutional layer corresponding to the point $(x,y)$ is determined by the values $F_{d^{\prime }}(x-i,y-j)$ of the input to this layer at points $(x-i,y-j)$ corresponding to $\left|i\right|\le L$ and $\left|j\right|\le L$. This is illustrated by Figure 1, where, for $L=1$ and for a point $(x,y)$ marked by an asterisk, we show all the points $(x^{\prime },y^{\prime })=(x_0-i,y_0-j)$ that determine the values $G_d(x,y)$. For convenience, points $(x^{\prime },y^{\prime })$ that do not affect the values $G_d(x,y)$, are marked by zeros.

Figure 1. Convolution filter: case of $L=1$.

For $L=2$, a similar picture has the following form. This is illustrated by Figure 2.

Figure 2. Convolution filter: case of $L=2$.

1.3. Sparse Filters and Dilated Convolution

Originally, convolutional neural networks used filters in which all the values $k_d(i,j,d^{\prime })$ for $\left|i\right|,\left|j\right|\le L$ can be non-zero. It turned out, however, that we can achieve a better accuracy if we consider sparse filters, i.e., filters in which, for some pairs $(i,j)$ with $\left|i\right|,\left|j\right|\le L$, all the values $k_d(i,j,d^{\prime })$ are fixed at 0; see, e.g., [2,3,4].

In Figure 3, we show an example of such a situation, when $L=2$ and only values $k_d(i,j,d^{\prime })$, for which both i and j are even allowed to be non-zero.

Figure 3. Case when $L=2$ and only values $k_d(i,j,d^{\prime })$ with even i and j can be no-zero.

In general, it turned out that such a restriction works best if we only allow $k_d(i,j,d^{\prime })\ne 0$ for pairs $(i,j)$ which are divisible by some integer ℓ, i.e., if we take

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d=d_{\mathrm{in}}}\left(\sum _{-L\le i,j\le L:i/\ell \in \mathbb{Z},j/\ell \in \mathbb{Z}}{k}_d(i,j,d^{\prime })·F_{d^{\prime }}(x-i,y-j)\right).$$

(4)

In this case, the layer’s output signal $G_d(x,y)$ can be written in the following equivalent form:

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{-\tilde{L}\le \tilde{i},\tilde{j}\le \tilde{L}}{\tilde{k}}_d\left(\tilde{i},\tilde{j},d^{\prime }\right)·F_{d^{\prime }}\left(x-\ell ·\tilde{i},y-\ell ·\tilde{j}\right)\right),$$

(5)

where we denoted $\tilde{L}\stackrel{\mathrm{def}}{=}L/\ell$, $\tilde{i}\stackrel{\mathrm{def}}{=}i/\ell$, $\tilde{j}\stackrel{\mathrm{def}}{=}j/\ell$, and ${\tilde{k}}_d\left(\tilde{i},\tilde{j},d^{\prime }\right)\stackrel{\mathrm{def}}{=}k\left(\ell ·\tilde{i},\ell ·\tilde{j},d^{\prime }\right)$.

The resulting networks are known as dilated convolutional neural networks, since skipping some points $(i,j)$ in the description of the filter is kind of equivalent to extending (dilating) the distance between the remaining points; see, e.g., [2,3,4].

1.4. Empirical Fact That Needs Explanation

In principle, we could select other points $(i,j)$ at which the filter can be non-zero. For example, we could select points for which j is even, but i can be any integer. This is illustrated by Figure 4.

Figure 4. Case when $L=2$ and only values $k_d(i,j,d^{\prime })$ with even j can be non-zero.

Alternatively, for $L=2$, as points $(i,j)$ at which $k_d(i,j,d^{\prime })$ can be non-zero, we could select the points $(0,0)$, $(0,\pm 1)$, $(\pm 1,0)$, and $(\pm 2,\pm 2)$, see Figure 5.

Figure 5. A possible selection of points $(i,j)$ for which $k_d(i,j,d^{\prime })$ can be no-zero.

However, empirical evidence shows that the selection corresponding to dilated convolution—when we select points for which i and j are both divisible by some integer ℓ—works the best [2,3,4].

To the best of our knowledge, there is no theoretical explanation for this empirical result—that dilated convolution leads to better results that select other sets of non-zero-valued points $(i,j)$. The main objective of this paper is to provide such an explanation.

Comment. Let us emphasize that the only objective of this paper is to explain this empirical fact; we are not yet at a stage where we can propose a new method or even any improvements to the known methods.

2. Analysis of The Problem

2.1. Let Us Reformulate This Situation in Geometric Terms: Case of Traditional Convolution

In the original convolution Formula (1), to find the values $G_d(x,y)$, the layer’s output signal at a point $(x,y)$, we need to know the values $F_{d^{\prime }}(x^{\prime },y^{\prime })$, the layer’s input signal at all the points $(x^{\prime },y^{\prime })$ of the type $(x-i,y-j)$ for $\left|i\right|,\left|j\right|\le L$. We can reformulate it by saying that we need to know the values $F_{d^{\prime }}(x^{\prime },y^{\prime })$ at all the points $(x^{\prime },y^{\prime })$ at which the ${\ell }_{\infty }$ distance

$$d_{\infty }((x,y),(x^{\prime },y^{\prime }))\stackrel{\mathrm{def}}{=}max\left(\right|x-x^{\prime }|,|y-y^{\prime }\left|\right),$$

(6)

does not exceed L:

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{(x^{\prime },y^{\prime })\in D:d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L}{k}_d(x-x^{\prime },y-y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime },y^{\prime })\right),$$

(7)

where we denoted

$$D\stackrel{\mathrm{def}}{=}(\mathbb{Z}\cap [\underline{X},\overline{X}])\times (\mathbb{Z}\cap [\underline{Y},\overline{Y}]).$$

(8)

We use, in this formula, the bounded subset D of the “grid” $\mathbb{Z}\times \mathbb{Z}$ and not the whole set $\tilde{S}\stackrel{\mathrm{def}}{=}\mathbb{Z}\times \mathbb{Z}$ only matters at the border of the domain D. So, to simplify our formulas, we can follow the usual tradition (see, e.g., [3]) and simply use the whole set $\tilde{S}=\mathbb{Z}\times \mathbb{Z}$ instead of the bounded set D:

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{(x^{\prime },y^{\prime })\in \tilde{S}:d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L}{k}_d(x-x^{\prime },y-y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime },y^{\prime })\right).$$

(9)

Comment. Note that the set $\tilde{S}$ is potentially infinite. What makes the set of all the points $(x^{\prime },y^{\prime })$—which affects the values $G_d(x,y)$—finite is the restriction $d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L$, whose meaning is that such points $(x^{\prime },y^{\prime })$ should belong to the corresponding neighborhood of the point $(x,y)$.

2.2. Case of Dilated Convolution

The dilated convolution can be described in a similar way. Namely, we can describe the Formula (4) as

$$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{(x^{\prime },y^{\prime })\in S_{\ell }(x,y):d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L}{k}_d(x-x^{\prime },y-y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime },y^{\prime })\right),$$

(10)

where $S_{\ell }(x,y)$ denotes the set of all the points $(x^{\prime },y^{\prime })$ for which both differences $x-x^{\prime }$ and $y-y^{\prime }$ are divisible by ℓ:

$$S_{\ell }(x,y)\stackrel{\mathrm{def}}{=}\{(x^{\prime },y^{\prime }):x^{\prime }\equiv x\mathrm{mod}\ell ,y^{\prime }\equiv y\mathrm{mod}\ell \}.$$

(11)

Note that, in this representation of dilated convolution, while we have several different sets $S_{\ell }(x,y)$ for different points $(x,y)$, there is only one filter $k_d(x-x^{\prime },y-y^{\prime },d^{\prime })$, namely the same filter that was used in the original representation (4). So, in this new representation, we have exactly as many parameters as before.

The main difference between this formula and the Formula (9) is that, in contrast to the usual convolution (9), where the same set $\tilde{S}=\mathbb{Z}\times \mathbb{Z}$ could be used for all the points $(x,y)$, here, in general, we may need different sets $S_{\ell }(x,y)$ for different points $(x,y)$.

For example, if $\ell =2$, then we need four such sets:

• for points $(x,y)$ for which both x and y are even, the Formula (10) holds for

  $$S_2(0,0)=S_2(0,2)=\dots =S_{0,0}^{(\ell =2)}\stackrel{\mathrm{def}}{=}\{(x,y)\in \mathbb{Z}\times \mathbb{Z}:x\mathrm{and}y\mathrm{are}\mathrm{even}\};$$

  (12)
• for points $(x,y)$ for which x is even but y is odd, the Formula (10) holds for

$$S_2(0,1)=S_2(0,3)=\dots =S_{0,1}^{(\ell =2)}\stackrel{\mathrm{def}}{=}\{(x,y)\in \mathbb{Z}\times \mathbb{Z}:x\mathrm{is}\mathrm{even}\mathrm{and}y\mathrm{is}\mathrm{odd}\};$$

(13)

• for points $(x,y)$, for which x is odd but y is even, the Formula (10) holds for

$$S_2(1,0)=S_2(1,2)=\dots =S_{1,0}^{(\ell =2)}\stackrel{\mathrm{def}}{=}\{(x,y)\in \mathbb{Z}\times \mathbb{Z}:x\mathrm{is}\mathrm{odd}\mathrm{and}y\mathrm{is}\mathrm{even}\};$$

(14)

• finally, for points $(x,y)$ for which x and y are both odd, the Formula (10) holds for

  $$S_2(0,1)=S_2(0,3)=\dots =S_{1,1}^{(\ell =2)}\stackrel{\mathrm{def}}{=}\{(x,y)\in \mathbb{Z}\times \mathbb{Z}:x\mathrm{and}y\mathrm{are}\mathrm{odd}\}.$$

  (15)

In this case, instead of the single set $S_1(x,y)=\tilde{S}$ (as in the case of the traditional convolution), we have a set of such sets

$$\mathcal{F}=\left\{S_{0,0}^{(\ell =2)},S_{0,1}^{(\ell =2)},S_{1,0}^{(\ell =2)},S_{1,1}^{(\ell =2)}\right\}.$$

(16)

To avoid confusion, we will call subsets of the original “grid” $\mathbb{Z}\times \mathbb{Z}$ sets, while the set of such sets will be called a family. In these terms, the Formula (10) can be described as follows:

$$\begin{gathered} G_d(x,y)= \\ \sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{(x^{\prime },y^{\prime })\in S_{\mathcal{F}}(x,y):d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L}{k}_d(x-x^{\prime },y-y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime },y^{\prime })\right), \end{gathered}$$

(17)

where $S_{\mathcal{F}}(x,y)$ denotes the set $S\in \mathcal{F}$ from the family $\mathcal{F}$ that contains the point $(x,y)$:

$$(x,y)\in S_{\mathcal{F}}(x,y)\mathrm{and}{S}_{\mathcal{F}}(x,y)\in \mathcal{F}.$$

(18)

In this representation, all four sets S from the family $\mathcal{F}$ are infinite—just like the set $\tilde{S}$ corresponding to the traditional convolution is infinite. Similarly to the traditional convolution, what makes the set of all the points $(x^{\prime },y^{\prime })$—which affects the values $G_r(x,y)$—finite is the restriction $d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L$, whose meaning is that such points $(x^{\prime },y^{\prime })$ should belong to the corresponding neighborhood of the point $(x,y)$.

Figure 6 describes which of the four sets $S\in \mathcal{F}$ corresponds to each point $(x,y)$ from the “grid” $\mathbb{Z}\times \mathbb{Z}$:

Figure 6. Sets $S_{\mathcal{F}}(x,y)$ corresponding to different points $(x,y)$—for filters presented in Figure 3.

For $\ell =3$, we can get a similar reformulation, with the family

$$\mathcal{F}=\left\{S_{0,0}^{(\ell =3)},S_{0,1}^{(\ell =3)},S_{0,2}^{(\ell =3)},S_{1,0}^{(\ell =3)},S_{1,1}^{(\ell =3)},S_{1,2}^{(\ell =3)},S_{2,0}^{(\ell =3)},S_{2,1}^{(\ell =3)},S_{2,2}^{(\ell =3)}\right\},$$

(19)

where $S_{i,j}^{(\ell =3)}$ is the set of all the pairs $(x,y)\in \mathbb{Z}\times \mathbb{Z}$, in which both differences $x-i$ and $y-j$ are divisible by 3.

In general, for an arbitrary point $(x,y)$, we should use the set $S_{\mathcal{F}}=S_{x\mathrm{mod}\ell ,y\mathrm{mod}\ell }^{(\ell =2)}.$

2.3. Other Cases

Such a representation is possible not only for dilated convolution. For example, the above case when we allow arbitrary value i and require the value j to be even can be described in a similar way, with

$$\mathcal{F}=\{S_0,S_1\},$$

(20)

where:

• for points $(x,y)$ for which y is even, we take

  $$S_{\mathcal{F}}(0,0)=S_{\mathcal{F}}(1,0)=\dots =S_0\stackrel{\mathrm{def}}{=}\{(x,y)\in \mathbb{Z}\times \mathbb{Z}:y\mathrm{is}\mathrm{even}\},$$

  (21)
• and for points $(x,y)$ for which y is odd, we take

  $$S_{\mathcal{F}}(0,1)=S_{\mathcal{F}}(1,1)=\dots =S_1\stackrel{\mathrm{def}}{=}\{(x,y)\in \mathbb{Z}\times \mathbb{Z}:y\mathrm{is}\mathrm{odd}\}.$$

  (22)

We can also have families which have an infinite number of sets; an example of such a family will be given below.

We can also, in principle, consider the situations when we do not require that the coefficients $k_d(x,x^{\prime },y,y^{\prime },d^{\prime })$ depend only on the differences $x-x^{\prime }$ and $y-y^{\prime }$. Thus, we arrive at the following general description.

2.4. General Case

In the general case, we get the following situation:

• we have a family $\mathcal{F}$ of subsets of the “grid” $\mathbb{Z}\times \mathbb{Z}$;
• the values $G_d(x,y)$ of the layer’s output signal at a point $(x,y)$ are determined by the formula

  $$G_d(x,y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{(x^{\prime },y^{\prime })\in S_{\mathcal{F}}(x,y):d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L}{K}_d(x,x^{\prime },y,y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime },y^{\prime })\right),$$

  (23)

  for some values $K_d(x,x^{\prime },y,y^{\prime },d^{\prime })$, where $S_{\mathcal{F}}(x,y)$ denotes the set $S\in \mathcal{F}$ from the family $\mathcal{F}$ that contains the point $(x,y)$.

For the Formula (23) to uniquely determine the values $G_d(x,y)$, we need to make sure that the set $S_{\mathcal{F}}(x,y)$ is uniquely determined by the point $(x,y)$, i.e., that for each point $(x,y)$, the family $\mathcal{F}$ contains one, and only one, set S that contains this point. In other words:

• different sets from the family $\mathcal{F}$ must be disjoint, and
• the union of all the sets $S\in \mathcal{F}$ must coincide with the whole “grid” $\mathbb{Z}\times \mathbb{Z}$.

In mathematical terms, the family $\mathcal{F}$ must form a partition of the “grid” $\mathbb{Z}\times \mathbb{Z}$.

Comment. To avoid possible confusion, it is worth mentioning that while different sets S from the family $\mathcal{F}$ are disjoint, this does not preclude the possibility that sets $S_{\mathcal{F}}(x,y)$ and $S_{\mathcal{F}}(x^{\prime },y^{\prime })$ corresponding to different points $(x,y)\ne (x^{\prime },y^{\prime })$ can be identical. For example, in the description of the traditional convolution, the family $\mathcal{F}$ consists of only one set $\mathcal{F}=\left\{\tilde{S}\right\}$. In this case, for all points $(x,y)$ and $(x^{\prime },y^{\prime })$, we have $S_{\mathcal{F}}(x,y)=S_{\mathcal{F}}(x^{\prime },y^{\prime })=\tilde{S}$.

In terms of sets corresponding to different points, disjointness means that if the sets $S_{\mathcal{F}}(x,y)$ and $S_{\mathcal{F}}(x^{\prime },y^{\prime })$ are different, then these sets must be disjoint: $S_{\mathcal{F}}(x,y)\cap S_{\mathcal{F}}(x^{\prime },y^{\prime })=\varnothing$.

2.5. We Do Not a Priori Require Shift-Invariance

Please note that we do not a priori require that the sets $S_{\mathcal{F}}(x,y)$ and $S_{\mathcal{F}}(x_0,y_0)$ corresponding to two different points $(x,y)$ and $(x_0,y_0)$ should be obtained from each other by shift—this property is known as shift invariance and is satisfied both for the usual convolution and for the dilated convolution.

It should be emphasized, however, that we will show that this shift-invariance property holds for the optimal arrangement.

2.6. Let Us Avoid the Degenerate Case

From a purely mathematical viewpoint, we can have a partition of the “grid” $\mathbb{Z}\times \mathbb{Z}$ into one-point sets $\left\{\right(x,y\left)\right\}$. This is an example when the family $\mathcal{F}$ has infinitely many subsets.

In this case, no matter what value L we choose, the Formula (23) implies that the values $G_d(x,y)$ of the layer’s output signal at a point $(x,y)$ are determined only by the values $F_{d^{\prime }}(x,y)$ of the layer’s input at this same point. This is equivalent to using a convolution with $L=0$; such a convolution is known as the 1-by-1 convolution.

While such a convolution is often useful, in this case, for each point $(x,y)$, there is only one point $(x^{\prime },y^{\prime })=(x,y)$, so it is not possible to select only some of the points $(x^{\prime },y^{\prime })$—which is the whole idea of dilation. Since, in this paper, we study dilation, we will therefore avoid this 1-by-1 situation and additionally require that at least one set from the family $\mathcal{F}$ should contain more than one element.

2.7. What We Plan to Do

We will consider all possible families $\mathcal{F}$ that form a partition of the “grid” $\mathbb{Z}\times \mathbb{Z}$, and we will show that for all optimality criteria that satisfy some reasonable conditions, the optimal family is either the family of sets corresponding to the dilated convolution—or a natural modification of this family.

Let us describe what we mean by optimality criteria.

2.8. What Does “Optimal” Mean?

In our case, we select between different families of sets $\mathcal{F}$, ${\mathcal{F}}^{\prime }$, …. In general, we select between alternatives a, b, etc. Out of all possible alternatives, we want to select an optimal one. What does “optimal” mean?

In many cases, “optimal” is easy to describe:

• we have an objective function $f\left(a\right)$ that assigns a numerical value to each alternative a—e.g., the average approximation error of the numerical method a for solving a system of differential equations, and
• optimal means that we select an alternative for which the value of this objective function is the smallest possible (or, for some objective functions, the largest possible).

However, this is not the only possible way to describe optimality.

For example, if we are minimizing the average approximation error, and there are several different numerical methods with the exact same smallest value of average approximation error, then we can use this non-uniqueness to select, e.g., the method with the shortest average computation time. In this case, we have, in effect, a more complex preference relation between alternatives than in the case when decision is made based solely on the value of the objective function. Specifically, in this case, an alternative b is better than the alternative a—we will denote it by $a<b$—if:

• either we have $f\left(b\right)<f\left(a\right)$,
• or we have $f\left(a\right)=f\left(b\right)$ and $g\left(b\right)<g\left(a\right)$.

If this still leaves several alternatives which are equally good, then we can optimize something else, and thus, have an even more complex optimality criterion.

In general, having an optimality criterion means that we are able to compare pairs of alternatives—at least some such pairs—and conclude that:

• for some of these pairs, we have $a<b$,
• for some of these pairs, we have $b<a$, and
• for some others pairs, we conclude that alternatives a and b are, from our viewpoint, of equal value; we will denote this by $a\sim b$.

Of course, these relations must satisfy some reasonable properties. For example, if b is better than a, and c is better than b, then c should be better than a; in mathematical terms, the relation < must be transitive.

We must have some alternative which is better than or equivalent to all others—otherwise, the optimization problem has no solutions. It also makes sense to require that there is only one such optimal alternative—indeed, as we have mentioned, if there are several equally good optimal alternatives, this means that the original optimality criterion is not final, that we can use this non-uniqueness to optimize something else, i.e., in effect, to modify the original criterion into a final (or at least “more final”) one.

2.9. Invariance

There is an additional natural requirement for possible optimality criteria, which is related to the fact that the original “grid” $\mathbb{Z}\times \mathbb{Z}$ has lots of symmetries, i.e., transformations that transform this “grid” into itself.

For example, if we change the starting point of the coordinate system to a new point $(x_0,y_0)$, then a point that originally had coordinates $(x,y)$ now has coordinates $(x-x_0,y-y_0)$. It makes sense to require that the relative quality of two different families $\mathcal{F}$ and ${\mathcal{F}}^{\prime }$ will not change if we simply change the starting point.

Similarly, we can change the direction of the x-axis, then a point $(x,y)$ becomes $(-x,y)$. If we change the direction of the y-axis, we obtain a transformation $(x,y)\to (x,-y)$. Finally, we can rename the coordinates: what was x will become y and vice versa; this corresponds to the transformation $(x,y)\to (y,x)$. Such transformations should also not affect the relative quality of different families.

Please note that we are not requiring that the family $\mathcal{F}$ of sets be shift-invariant, what we require is that the optimality criterion is shift-invariant.

Let us explain why, in our opinion, it makes sense to require that the optimality criterion is shift-invariance—as well as having other invariance properties. Indeed, let us consider any usual optimality criterion such as accuracy of classification, robustness to noise, etc. What each criterion means is, e.g., the overall classification accuracy over the set $\mathcal{S}$ of all possible cat and not-a-cat images $I\in \mathcal{S}$. We want this method to correctly classify images into cats and not-cats, whether these images are centered or somewhat shifted. Thus, to adequately compare different methods, we should test these methods on a set $\mathcal{S}$ of images that includes both original and shifted images.

Here:

• if we shift each image I from the set $\mathcal{S}$ by the same shift $(x_0,y_0)$, i.e., replace each image $I\in S$ by a shifted image $I^{\prime }=T_{x_0,y_0}\left(I\right)$ for which $I^{\prime }(x,y)=I(x-x_0,y-y_0)$,
• then, we should get, in effect, the exact same set of images:

  $$T_{x_0,y_0}\left(\mathcal{S}\right)\stackrel{\mathrm{def}}{=}\{T(x_0,y_0)\left(I\right):I\in \mathcal{S}\}\approx \mathcal{S}.$$

  (24)

The only difference between these two sets of images may be the few images where the cat is right at the image’s boundary; in this paper, we will ignore this difference—just like we ignored the bounded-ness in the previous text. In this ignoring-bounds approximation, we conclude that

$$T_{x_0,y_0}\left(\mathcal{S}\right)=\{T(x_0,y_0)\left(I\right):I\in \mathcal{S}\}=\mathcal{S}.$$

(25)

How does a shift of the original image affect the input signals to the following convolution layers? In between the very first input layer and the following convolution layers, we may have (and usually do have) layers that perform the “compression” of the $(x,y)$ part—i.e., that transform:

• values corresponding to several points $(x,y)$
• into values corresponding to a single new point $(x^{\prime },y^{\prime })$.

In general, the $(x,y)$-shift of the original data corresponds to a shift of the transformed data—but by smaller shift values. For example, if data corresponding to each new $(x,y)$-point come from data from four different “pre-compression” points, then the shift by $(x_0,y_0)$ in the pre-$(x,y)$-compression layer corresponds to a shift of the convolution layer input by $(x_0/2,y_0/2)$.

Since the set of input images should not change if we apply a shift, we can conclude that for each convolution layer, the set of the corresponding inputs to this layer should also not change if we shift all these inputs, i.e., if we replace each input $F_d(x,y)$ with a shifted input

$$F_d^{\prime }(x,y)\stackrel{\mathrm{def}}{=}{F}_d(x-x_0,y-y_0)$$

(26)

for some shift $(x_0,y_0)$.

The set of inputs on which we compare different methods does not change when we apply a shift. So, if one method was better when we processed original inputs, it should still be better if we process shifted inputs—since the resulting set of inputs is the same. In other words, the quality (e.g., accuracy) $Q_{\mathcal{F}}\left(\mathcal{S}\right)$ of a method corresponding to the family $\mathcal{F}$, when gauged by the set of inputs corresponding to original images, should be the same as this method’s quality $Q_{\mathcal{F}}\left(T_{x_0,y_0}\left(\mathcal{S}\right)\right)$ on the set

$$T_{x_0,y_0}\left(\mathcal{S}\right)=\{T_{x_0,y_0}\left(F_d\right):F_d\in \mathcal{S}\}$$

(27)

of all the inputs obtained from the original set $\mathcal{S}$ by this shift—since these two sets of inputs are, in effect, the same set: $T_{x_0,y_0}\left(\mathcal{S}\right)=\mathcal{S}$. Thus, $Q_{\mathcal{F}}\left(T_{x_0,y_0}\left(\mathcal{S}\right)\right)=Q_{\mathcal{F}}\left(\mathcal{S}\right)$.

However, as one can see, shifting all the inputs is equivalent to shifting all the sets from the family $\mathcal{F}$. Indeed, if we apply the Formula (23) to the shifted layer’s input $F_d^{\prime }(x,y)\stackrel{\mathrm{def}}{=}{F}_d(x-x_0,y-y_0)$, we get

$$\begin{gathered} G_d(x,y)= \\ \sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{(x^{\prime },y^{\prime })\in S_{\mathcal{F}}(x,y):d_{\infty }((x,y),(x^{\prime },y^{\prime }))\le L}{K}_d(x,x^{\prime },y,y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime }-x_0,y^{\prime }-y_0)\right), \end{gathered}$$

(28)

i.e., in terms of the shifted coordinates $X\stackrel{\mathrm{def}}{=}x-x_0$ and $Y\stackrel{\mathrm{def}}{=}y-x_0$ for which $x=X+x_0$ and $y=Y+y_0$, we get—taking into account that the distance $d_{\infty }$ does not change with shift—that:

$$G_d(X,Y)=\sum _{d^{\prime }=1}^{d_{\mathrm{in}}}\left(\sum _{C^{\prime }:d_{\infty }((X,Y),(X^{\prime },Y^{\prime }))\le L}{K}_d^{\prime }(X,X^{\prime },Y,Y^{\prime },d^{\prime })·F_{d^{\prime }}(x^{\prime }-x_0,y^{\prime }-y_0)\right),$$

(29)

where we denoted

$$K_d^{\prime }(X,X^{\prime },Y,Y^{\prime },d^{\prime })\stackrel{\mathrm{def}}{=}{K}_d(X+x_0,X^{\prime }+x_0,Y+y_0,Y^{\prime }+y_0),$$

(30)

and where $C^{\prime }$ denotes the condition $(X^{\prime }+x_0,Y^{\prime }+y_0)\in S_{\mathcal{F}}(X+x_0,Y+y_0)$.

In terms of the family $\mathcal{F}$, the main difference between the Formulas (23) and (29) is that instead of the condition $(x^{\prime },y^{\prime })\in S_{\mathcal{F}}(x,y)$, we now have a new condition

$$C^{\prime }\iff (X^{\prime }+x_0,Y^{\prime }+y_0)\in S_{\mathcal{F}}(X+x_0,Y+y_0),$$

(31)

i.e., equivalently, $(X^{\prime },Y^{\prime })\in S_{\mathcal{F}}(X+x_0,Y+y_0)-(x_0,y_0)$. It is easy to check that this new condition is equivalent to $(Y^{\prime },Y^{\prime })\in S_{{\mathcal{F}}^{\prime }}(X,Y)$, where the new family ${\mathcal{F}}^{\prime }$ is obtained by shifting sets from the original family $\mathcal{F}$.

So:

• the relative quality of two families does not change if we shift all the layer’s inputs;
• however, shifting all the layer’s inputs is equivalent to shifting all the sets from the family $\mathcal{F}$.

Thus, the relative quality of two families does not change if we shift both families. In other words, a reasonable optimality criterion—which describes which family is better—should be invariant with respect to shifts.

Similarly, we can argue that a reasonable optimality criterion should not change if we rename x- and y-axes, etc.

Now, we are ready for the precise formulation of the problem.

3. Definitions and the Main Result

Definition 1.

By a family, we mean a family of non-empty subsets of the “grid” $\mathbb{Z}\times \mathbb{Z}$, a family in which:

• all sets from this family are disjoint, and
• at least one set from this family has more than one element.

Terminological comment. To avoid possible misunderstandings, let us emphasize that here, we consider several levels of sets, and to avoid confusion, we use different terms for sets from different levels:

• first, we consider points $(x,y)\in \mathbb{Z}\times \mathbb{Z}$;
• second, we consider sets of points $S\subseteq \mathbb{Z}\times \mathbb{Z}$; we call them simply sets;
• third, we consider sets of sets of points $\mathcal{F}=\{S,S^{\prime },\dots \}$; we call them families;
• finally, we consider the set of all possible families $\{\mathcal{F},{\mathcal{F}}^{\prime },\dots \}$; we call this a class.

Comment about the requirements. In the previous text, we argued that for each family $\mathcal{F}$, the union of all its sets $\cup \{S:S\in \mathcal{F}\}$ should coincide with the whole “grid” $\mathbb{Z}\times \mathbb{Z}$. However, in our definition of an alternative, we did not impose this requirement. We omitted this requirement to make our result stronger—since, as we see from the following Proposition, this requirement actually follows from all the other requirements.

Definition 2.

By an optimality criterion, we mean a pair of relations $(<,\sim )$ on the class of all possible families that satisfy the following conditions:

• if $\mathcal{F}<{\mathcal{F}}^{\prime }$ and ${\mathcal{F}}^{\prime }<{\mathcal{F}}^{\prime }$, then $\mathcal{F}<{\mathcal{F}}^{″}$;
• if $\mathcal{F}<{\mathcal{F}}^{\prime }$ and ${\mathcal{F}}^{\prime }\sim {\mathcal{F}}^{″}$, then $\mathcal{F}<{\mathcal{F}}^{″}$;
• if $\mathcal{F}\sim {\mathcal{F}}^{\prime }$ and ${\mathcal{F}}^{\prime }<{\mathcal{F}}^{″}$, then $\mathcal{F}<{\mathcal{F}}^{″}$;
• if $\mathcal{F}\sim {\mathcal{F}}^{\prime }$ and ${\mathcal{F}}^{\prime }\sim {\mathcal{F}}^{″}$, then ${\mathcal{F}}^{\prime }\sim {\mathcal{F}}^{″}$;
• we have $\mathcal{F}\sim \mathcal{F}$ for all $\mathcal{F}$; and
• if $\mathcal{F}<{\mathcal{F}}^{\prime }$, then we cannot have $\mathcal{F}\sim {\mathcal{F}}^{\prime }$.

Comment. The pair of relations $(<,\sim )$ between families of subsets forms what is called a pre-order or quasi-order. This notion is more general than partial order, since, in contrast to the definition of the partial order, we do not require that if $a\le b$ and $b\le a$, then $a=b$: in principle, we can have $a\sim b$ for some $a\ne b$.

Definition 3.

We say that a family $\mathcal{F}$ is optimal with respect to the optimality criterion $(<,\sim )$, if for every other family ${\mathcal{F}}^{\prime }$, we have either ${\mathcal{F}}^{\prime }<\mathcal{F}$ or ${\mathcal{F}}^{\prime }\sim \mathcal{F}$.

Definition 4.

We say that the optimality criterion is final if there exists exactly one family which is optimal with respect to this criterion.

Definition 5.

By a transformation $T:\mathbb{Z}\times \mathbb{Z}$, we mean one of the following transformations: $T_{x_0,y_0}(x,y)=(x-x_0,y-y_0)$, $T_{-+}(x,y)=(-x,y)$, $T_{+-}(x,y)=(x,-y)$, and $T_{\leftrightarrow }(x,y)=(y,x)$.

Definition 6.

For each family $\mathcal{F}$ and for each transformation T, by the result $T\left(\mathcal{F}\right)$of applying the transformation T to the family $\mathcal{F}$, we mean the family $T\left(\mathcal{F}\right)=\left\{T\right(S):S\in \mathcal{F}\}$, where, for any set S, $T\left(S\right)\stackrel{\mathrm{def}}{=}\{T(x,y):(x,y)\in S\}.$

Definition 7.

We say that the optimality criterion is invariant if for all transformations T, $\mathcal{F}<{\mathcal{F}}^{\prime }$ implies that $T\left(\mathcal{F}\right)<T\left({\mathcal{F}}^{\prime }\right)$, and $\mathcal{F}\sim {\mathcal{F}}^{\prime }$ implies that $T\left(\mathcal{F}\right)\sim T\left({\mathcal{F}}^{\prime }\right)$.

Proposition 1.

For every final invariant optimality criterion, the optimal family is equal, for some integer $\ell \ge 1$, to one of the following two families:

• the family of all the sets $S_{\ell ,x_0,y_0}\stackrel{\mathrm{def}}{=}\{(x_0+\ell ·n_x,y_0+\ell ·n_y):n_x,n_y\in \mathbb{Z}\}$ corresponding to all possible pairs of integers $(x_0,y_0)$ for which $0\le x_0,y_0<\ell$;
• the family of all the sets

  $$S_{\ell ,x_0,y_0}^{\prime }\stackrel{\mathrm{def}}{=}\{(x_0+\ell ·n_x,y_0+\ell ·n_y):n_x,n_y\in \mathbb{Z}and{n}_x+n_{y}iseven\}$$
  corresponding to all possible pairs of integers $(x_0,y_0)$ for which $0\le x_0,y_0<\ell$.

Comments.

• This proposition takes care of all invariant (and final) optimality criteria. Thus, it should work for all usual criteria based on misclassification rate, time of calculation, used memory, or any others used in neural networks: indeed, if one method is better than another for images in general, it should remain to be better if we simply shift all the images or turn all the images upside down. Images can come as they are, they can come upside down, they can come shifted, etc. If, for some averaging criterion, one method works better for all possible images, but another method works better for all upside-down versions of these images—which is, in effect, the same class of possible images—then, from the common sense viewpoint, this would mean that something is not right with this criterion.
• The first possibly optimal case corresponds to dilated convolution. In the second possibly optimal case, the optimal family contains similar but somewhat different sets; an example of such a set is given in Figure 7.
  Thus, this result explains the effectiveness of dilated convolution—and also provides us with a new alternative worth trying.
• The following proof is similar to several proofs presented in [5,6].

Figure 7. A set from the second possibly optimal family.

Proof. 

$1^{\circ }$. Since the optimality criterion is final, there exists exactly one optimal family ${\mathcal{F}}_{\mathrm{opt}}$. Let us first prove that this family is itself invariant, i.e., that $T\left({\mathcal{F}}_{\mathrm{opt}}\right)={\mathcal{F}}_{\mathrm{opt}}$ for all transformations T.

Indeed, the fact that the family ${\mathcal{F}}_{\mathrm{opt}}$ is optimal means that for every family $\mathcal{F}$, we have $\mathcal{F}<{\mathcal{F}}_{\mathrm{opt}}$ or $\mathcal{F}\sim {\mathcal{F}}_{\mathrm{opt}}$. Since this is true for every family $\mathcal{F}$, it is also true for every family $T^{-1}\left(\mathcal{F}\right)$, where $T^{-1}$ denotes inverse transformation (i.e., a transformation for which $T\left(T^{-1}(x,y)\right)=(x,y)$). Thus, for every family $\mathcal{F}$, we have either $T^{-1}\left(\mathcal{F}\right)<{\mathcal{F}}_{\mathrm{opt}}$ or $T^{-1}\left(\mathcal{F}\right)\sim {\mathcal{F}}_{\mathrm{opt}}$. Due to invariance, we have $\mathcal{F}=T\left(T^{-1}\left(\mathcal{F}\right)\right)<T\left({\mathcal{F}}_{\mathrm{opt}}\right)$ or $\mathcal{F}\sim T\left({\mathcal{F}}_{\mathrm{opt}}\right)$. By definition of optimality, this means that the alternative $T\left({\mathcal{F}}_{\mathrm{opt}}\right)$ is also optimal. However, since the optimality criterion is final, there exists exactly one optimal family, so $T\left({\mathcal{F}}_{\mathrm{opt}}\right)={\mathcal{F}}_{\mathrm{opt}}$.

The statement is proven.

$2^{\circ }$. Let us now prove that the optimal family contains a set $S^{\prime }$ that, in its turn, contains the point $(0,0)$ and some point $(x,y)\ne (0,0).$

Indeed, by definition of a family, every family—including the optimal family—contains at least one set S that has at least two points. Let S be any such set from the optimal family, and let $(x_0,y_0)$ be any of its points. Then, due to Part 1 of this proof, the set $S^{\prime }\stackrel{\mathrm{def}}{=}{T}_{x_0,y_0}\left(S\right)$ also belongs to the optimal family, and this set contains the point

$$T_{x_0,y_0}(x_0,y_0)=(x_0-x_0,y_0-y_0)=(0,0).$$

Since the set S had at least two different points, the set $S^{\prime }=T_{x_0,y_0}\left(S\right)$ also contains at least two different points. Thus, the set $S^{\prime }$ must contain a point $(x,y)$ which is different from $(0,0)$.

The statement is proven.

$3^{\circ }$. In the following text, by $S^{\prime }$, we will mean a set from the optimal family ${\mathcal{F}}_{\mathrm{opt}}$ whose existence is proven in Part 2 of this proof: namely, a set that contains the point $(0,0)$ and a point $(x,y)\ne (0,0)$.

$4^{\circ }$. Let us prove that if the set $S^{\prime }$ contains a point $(x,y)$, then this set also contains the points $(x,-y)$, $(-x,y)$, and $(y,x)$.

Indeed, due to Part 1 of this proof, with the set $S^{\prime }$ the optimal family ${\mathcal{F}}_{\mathrm{opt}}$ also contains the set $T_{+-}\left(S^{\prime }\right)$. This set contains the point $T_{+-}(0,0)=(0,0)$. Thus, the sets $S^{\prime }$ and $T_{+-}\left(S^{\prime }\right)$ have a common element $(0,0)$. Since different sets from the optimal family must be disjoint, it follows that the sets $S^{\prime }$ and $T_{+-}\left(S^{\prime }\right)$ must coincide. The set $T_{+-}\left(S^{\prime }\right)$ contains the points $(x,-y)$ for each point $(x,y)\in S$. Since $T_{+-}\left(S^{\prime }\right)=S^{\prime }$, this implies that for each point $(x,y)\in S^{\prime }$, we have $(x,-y)\in T_{+-}\left(S^{\prime }\right)=S^{\prime }$.

Similarly, we can prove that $(-x,y)\in S^{\prime }$ and $(y,x)\in S^{\prime }$. The statement is proven.

$5^{\circ }$. By combining the two conclusions of Part 4—that $(x,-y)\in S^{\prime }$ and that therefore $T_{-+}(x,-y)=(-x,-y)\in S^{\prime }$, we conclude that for every point $(x,y)\in S^{\prime }$, the point

$$-(x,y)\stackrel{\mathrm{def}}{=}(-x,-y)$$

is also contained in the set $S^{\prime }$.

$6^{\circ }$. Let us prove that if the set $S^{\prime }$ contains two points $(x_1,y_1)$ and $(x_2,y_2)$, then it also contains the point

$$(x_1,y_1)+(x_2,y_2)\stackrel{\mathrm{def}}{=}(x_1+x_2,y_1+y_2).$$

Indeed, due to Part 1 of this proof, the set $T_{-x_2,-y_2}\left(S^{\prime }\right)$ also belongs to the optimal family. This set shares an element

$$T_{-x_2,-y_2}(0,0)=(0-(-x_2),0-(-y_2))=(x_2,y_2)$$

with the original set $S^{\prime }$. Thus, the set $T_{-x_2,-y_2}\left(S^{\prime }\right)$ must coincide with the set $S^{\prime }$. Due to the fact that $(x_1,y_1)\in S^{\prime }$, the element

$$T_{-x_2,-y_2}(x_1,y_1)=(x_1-(-x_2),y_1-(-y_2))=(x_1+x_2,y_1+y_2)$$

belongs to the set $T_{x_1,y_1}\left(S^{\prime }\right)=S^{\prime }$. The statement is proven.

$7^{\circ }$. Let us prove that if the set $S^{\prime }$ contains a point $(x,y)$, then, for each integer c, this set also contains the point

$$c·(x,y)=(c·x,c·y).$$

Indeed, if c is positive, this follows from the fact that

$$(c·x,c·y)=(x,y)+\dots +(x,y)\left(c\mathrm{times}\right).$$

When c is negative, then we first use Part 5 and conclude that $(-x,-y)\in S^{\prime }$, and then conclude that the point $\left(\right|c|·(-x),|c|·(-y\left)\right)=(c·x,c·y)$ is in the set $S^{\prime }$.

$8^{\circ }$. Let us prove that if the set $S^{\prime }$ contains points $(x_1,y_1)$, …, $(x_n,y_n)$, then for all integers $c_1,\dots ,c_n$, it also contains their linear combination

$$c_1·(x_1,y_1)+\dots +c_n·(x_n,y_n)=(c_1·x_1+\dots +c_n·x_n,c_1·y_1+\dots +c_n·y_n).$$

Indeed, this follows from Parts 6 and 7.

$9^{\circ }$. The set $S^{\prime }$ contains some points which are different from $(0,0)$, i.e., points for which at least one of the integer coordinates is non-zero. According to Parts 4 and 5, we can change the signs of both x and y coordinates and still get points from $S^{\prime }$. Thus, we can always consider points with non-negative coordinates.

Let d denote the greatest common divisor of all positive values of the coordinates of points from $S^{\prime }$.

If a value x appears as an x-coordinate of some point $(x,y)\in S^{\prime }$, then, due to Part 4, we have $(x,-y)\in S^{\prime }$ and thus, due to Part 5,

$$(x,y)+(x,-y)=(2x,0)\in S^{\prime }.$$

Similarly, if a value y appears as a y-coordinate of some point $(x,y)\in S^{\prime }$, then we get $(0,2y)\in S^{\prime }$ and thus, due to Part 3, $(2y,0)\in S^{\prime }$.

It is known that a common divisor d of the values $v_1,\dots ,v_n$ can be represented as a linear combination of these values:

$$d=c_1·v_1+\dots +c_n·v_n.$$

For each value $v_i$, we have $(2v_i,0)\in S^{\prime }$, thus, for

$$2d=c_1·\left(2v_1\right)+\dots +c_n·\left(2v_n\right),$$

by Part 8, we get $(2d,0)\in S^{\prime }$. Due to Part 4, we thus have $(0,2d)\in S^{\prime }$, and due to Parts 6 and 7, all points $(n_x·\left(2d\right),n_y·\left(2d\right))$ for integers $n_x$ and $n_y$ also belong to the set $S^{\prime }$.

If $S^{\prime }$ has no other points, then for the set containing $(0,0)$, we indeed conclude that this set is the same as what we described for dilated convolution, with $\ell =2d$.

${10}^{\circ }$. What if there are other points in the set $S^{\prime }$? Since d is the greatest common divisor of all the coordinate values, each of these points has the form $(c_x·d,c_y·d)$ for some integers $c_x$ and $c_y$. Since this point is not of the form $(n_x·\left(2d\right),n_y·\left(2d\right))$, this means that either $c_x$, or $c_y$ is an odd number—or both.

Let us first consider the case when exactly one of the values $c_x$ and $c_y$ is odd. Without losing generality, let us assume that $c_x$ is odd, so $c_x=2n_x+1$ and $c_y=2n_y$ for some integers $n_x$ and $n_y$. Due to Part 9, we have $(2n_x·d,2n_y·d)\in S^{\prime }$, so the difference

$$((2n_x+1)·d,2n_y·d)-(2n_x·d,2n_y·d)=(d,0)$$

also belongs to the set $S^{\prime }$. Thus, similarly to Part 9, we can conclude that for every two integers $c_x$ and $c_y$, we have $(c_x·d,c_y·d)\in S^{\prime }$. So, in this case, $S^{\prime }$ coincides, for $\ell =d$, with the set corresponding to dilated convolution.

The only remaining case is when not all points $(c_x·d,c_y·d)$ belong to the set $S^{\prime }$. This means that for some such point, both values $c_x$ and $c_y$ are odd: $c_x=2n_x+1$ and $c_y=2n_y+1$ for some integers $n_x$ and $n_y$. Due to Part 9, we have $(2n_x·d,2n_y·d)\in S^{\prime }$, so the difference

$$((2n_x+1)·d,(2n_y+1)·d)-(2n_x·d,2n_y·d)=(d,d)$$

also belongs to the set $S^{\prime }$.

Since, due to Part 9, we have $(2n_x·d,2n_y·d)\in S^{\prime }$ for all $n_x$ and $n_y$, we conclude, by using Part 5, that $((2n_x+1)·d,(2n_y+1)·d)\in S^{\prime }$. So, all pairs for which both coordinates are odd multiples of d are in $S^{\prime }$. Thus, we get the new case described in the Proposition.

${11}^{\circ }$. The previous results were about the sets containing the point $(0,0)$.

For all other sets S containing some other point $(x_0,y_0)$, we get the same result if we take into account that the optimal family is invariant, and thus, with the set S, the optimal family also contains the set $T_{x_0,y_0}\left(S\right)$ that contains $(0,0)$ and is, thus, equal either to the family corresponding to dilated convolution or to the new similar family.

The proposition is proven. □

4. Conclusions and Future Work

4.1. Conclusions

One of the efficient machine learning ideas is the idea of a convolutional neural network. Such networks use convolutional layers, in which the layer’s output at each point is a combination of the layer’s input corresponding to several neighboring points. A reasonable idea is to restrict ourselves to only some of the neighboring points. It turns out that out of all such restrictions, the best results are obtained when we only use neighboring points, for which the differences in both coordinates are divisible by some constant ℓ. Networks implementing such restrictions are known as dilated convolutional neural networks.

In this paper, we provide a theoretical explanation for this empirical conclusion.

4.2. Future Work

This paper describes a general abstract result: that for any optimality criterion that satisfies some reasonable properties, some dilated convolution is optimal. To be practically useful, it is desirable to analyze which dilated convolutions are optimal for different practical situations and for specific criteria uses in machine learning, such as misclassification rate, time of calculation, used memory, etc. (and the combination of these criteria). It is also desirable to analyze what size neighborhood we should choose for different practical situations and for different criteria.