On the Depth of Decision Trees with Hypotheses

Abstract

In this paper, based on the results of rough set theory, test theory, and exact learning, we investigate decision trees over infinite sets of binary attributes represented as infinite binary information systems. We define the notion of a problem over an information system and study three functions of the Shannon type, which characterize the dependence in the worst case of the minimum depth of a decision tree solving a problem on the number of attributes in the problem description. The considered three functions correspond to (i) decision trees using attributes, (ii) decision trees using hypotheses (an analog of equivalence queries from exact learning), and (iii) decision trees using both attributes and hypotheses. The first function has two possible types of behavior: logarithmic and linear (this result follows from more general results published by the author earlier). The second and the third functions have three possible types of behavior: constant, logarithmic, and linear (these results were published by the author earlier without proofs that are given in the present paper). Based on the obtained results, we divided the set of all infinite binary information systems into four complexity classes. In each class, the type of behavior for each of the considered three functions does not change.

1. Introduction

Decision trees are studied in different areas of computer science, in particular in exact learning [1], rough set theory [2,3,4], and test theory [5]. In some sense, these theories deal with dual objects: for example, membership queries from exact learning correspond to attributes from test theory and rough set theory. In contrast to test theory and rough set theory, in exact learning, besides membership queries, equivalence queries are also considered.

We extend the model considered in test theory and rough set theory by adding the notion of a hypothesis that is an analog of equivalence query. Papers [6,7,8,9,10] are related mainly to the experimental study of decision trees with hypotheses. The present paper contains a theoretical study of the depth of decision trees with hypotheses.

An infinite binary information system is a pair $U=(A,F)$ where A is an infinite set of elements and F is an infinite set of functions (attributes) from A to $\{0,1\}$. A problem over U is given by a finite number of attributes $f_1,\dots ,f_n$ from F: for $a\in A$, we should find the tuple $(f_1\left(a\right),\dots ,f_n\left(a\right))$. To solve this problem, we can use decision trees with two types of queries. We can ask about the value of an attribute $f_i\in \{f_1,\dots ,f_n\}$. As a result, we obtain an answer of the kind $f_i\left(x\right)=\delta$ where $\delta \in \{0,1\}$. We also can ask if a hypothesis $f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n$ is true where ${\delta }_1,\dots ,{\delta }_n\in \{0,1\}$. Either we obtain the confirmation or a counterexample in the form $f_i\left(x\right)=\neg {\delta }_i$.

The depth of decision trees with hypotheses can be essentially less than the depth of decision trees using only attributes. As an example, we consider the problem of the computation of the disjunction $x_1\vee \cdots \vee x_n$. The minimum depth of a decision tree solving this problem using only attributes $x_1,\dots ,x_n$ is equal to n. However, the minimum depth of a decision tree with hypotheses solving this problem is equal to one: it is enough to ask only about the hypothesis $x_1=0,\dots ,x_n=0$. If it is true, then the considered disjunction is equal to zero. Otherwise, it is equal to one.

Based on the results of exact learning, rough set theory, and test theory [1,11,12,13,14,15,16], we study for an arbitrary infinite binary information system three functions of the Shannon type that characterize the growth in the worth case of the minimum depth of a decision tree solving a problem with the growth of the number of attributes in the problem description. The considered three functions correspond to the following three cases:

(i)

Only attributes are used in decision trees;

(ii)

Only hypotheses are used in decision trees;

(iii)

Both attributes and hypotheses are used in decision trees.

We show that the first function has two possible types of behavior: logarithmic and linear. The second and third functions have three possible types of behavior: constant, logarithmic, and linear. Bounds for the case (i) can be derived from more general results obtained in [15,16]. Results related to the cases (ii) and (iii) were presented in the conference paper [17] without proofs. In the present paper, we consider complete proofs for the cases (ii) and (iii). We also investigate the join behavior of these three functions and describe four complexity classes of infinite binary information systems; these results are completely new.

The obtained results allow us to understand the difference of time complexity for conventional decision trees that use only queries based on one attribute each and for decision trees with hypotheses. Moreover, we know now which combinations of types of behavior of the three Shannon-type functions we can take under consideration of an arbitrary infinite binary system, and we know the criteria for each combination.

This paper consists of six sections. In Section 2 and Section 3, we consider the basic notions and main results. Section 4 and Section 5 contain proofs of the main results, and Section 6 gives a short conclusion.

2. Basic Notions

Let A be a set of elements and F be a set of functions from A to $\{0,1\}$. Functions from F are called attributes, and the pair $U=(A,F)$ is called a binary information system (this notion is close to the notion of information systems proposed by Pawlak [18]). If A and F are infinite sets, then the pair $U=(A,F)$ is called an infinite binary information system.

A problem over U is an arbitrary n-tuple $z=(f_1,\dots ,f_n)$ where $n\in \mathbb{N}$, $\mathbb{N}$ is the set of natural numbers $\{1,2,\dots \}$, and $f_1,\dots ,f_n\in F$. The problem z may be interpreted as a problem of searching for the tuple $z\left(a\right)=(f_1\left(a\right),\dots ,f_n\left(a\right))$ for an arbitrary $a\in A$. The number $dimz=n$ is called the dimension of the problem z. Denote $F\left(z\right)=\{f_1,\dots ,f_n\}$. We denote by $P\left(U\right)$ the set of problems over U.

A system of equations over U is an arbitrary equation system of the kind:

$$\{g_1\left(x\right)={\delta }_1,\dots ,g_m\left(x\right)={\delta }_m\}$$

where $m\in \mathbb{N}\cup \left\{0\right\}$, $g_1,\dots ,g_m\in F$, and ${\delta }_1,\dots ,{\delta }_m\in \{0,1\}$ (if $m=0$, then the considered equation system is empty). This equation system is called a system of equations over z if $g_1,\dots ,g_m\in F\left(z\right)$. The considered equation system is called consistent (on A) if its set of solutions on A is nonempty. The set of solutions of the empty equation system coincides with A.

As algorithms for problem z solving, we consider decision trees with two types of queries. We can choose an attribute $f_i\in F\left(z\right)$ and ask about its value. This query has two possible answers: $\{f_i\left(x\right)=0\}$ and $\{f_i\left(x\right)=1\}$. We can formulate a hypothesis over z in the form $H=\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$ where ${\delta }_1,\dots ,{\delta }_n\in \{0,1\}$ and ask about this hypothesis. This query has $n+1$ possible answers: $H,\{f_1\left(x\right)=\neg {\delta }_1\},\dots ,\{f_n\left(x\right)=\neg {\delta }_n\}$ where $\neg 1=0$ and $\neg 0=1$. The first answer means that the hypothesis is true. Other answers are counterexamples.

A decision tree over z is a marked finite directed tree with the root in which:

• Each terminal node is labeled with an n-tuple from the set ${\{0,1\}}^n$;
• Each node, which is not terminal (such nodes are called working), is labeled with an attribute from the set $F\left(z\right)$ or with a hypothesis over z;
• If a working node is labeled with an attribute $f_i$ from $F\left(z\right)$, then there are two edges, which leave this node and are labeled with the systems of equations $\{f_i\left(x\right)=0\}$ and $\{f_i\left(x\right)=1\}$, respectively;
• If a working node is labeled with a hypothesis:

  $$H=\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$$

  over z, then there are $n+1$ edges, which leave this node and are labeled with the system of equations $H,\{f_1\left(x\right)=\neg {\delta }_1\},\dots ,\{f_n\left(x\right)=\neg {\delta }_n\}$, respectively.

Let $\Gamma$ be a decision tree over z. A complete path in $\Gamma$ is an arbitrary directed path from the root to a terminal node in $\Gamma$. We now define an equation system $\mathcal{S}\left(\xi \right)$ over U associated with the complete path $\xi$. If there are no working nodes in $\xi$, then $\mathcal{S}\left(\xi \right)$ is the empty system. Otherwise, $\mathcal{S}\left(\xi \right)$ is the union of equation systems assigned to the edges of the path $\xi$. We denote by $\mathcal{A}\left(\xi \right)$ the set of solutions on A of the system of equations $\mathcal{S}\left(\xi \right)$ (if this system is empty, then its solution set is equal to A).

We say that a decision tree $\Gamma$ over z solves the problem z relative to U if, for each element $a\in A$ and for each complete path $\xi$ in $\Gamma$ such that $a\in \mathcal{A}\left(\xi \right)$, the terminal node of the path $\xi$ is labeled with the tuple $z\left(a\right)$.

We now consider an equivalent definition of a decision tree solving a problem. Denote by ${\Delta }_U\left(z\right)$ the set of tuples $({\delta }_1,\dots ,{\delta }_n)\in {\{0,1\}}^n$ such that the system of equations $\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$ is consistent. The set ${\Delta }_U\left(z\right)$ is the set of all possible solutions to the problem z. Let $\Delta \subseteq {\Delta }_U\left(z\right)$, $f_{i_1},\dots ,f_{i_m}\in \{f_1,\dots ,f_n\}$, and ${\sigma }_1,\dots ,{\sigma }_m\in \{0,1\}$. Denote:

$$\Delta (f_{i_1},{\sigma }_1)\cdots (f_{i_m},{\sigma }_m)$$

the set of all n-tuples $({\delta }_1,\dots ,{\delta }_n)\in \Delta$ for which ${\delta }_{i_1}={\sigma }_1,\dots ,{\delta }_{i_m}={\sigma }_m$.

Let $\Gamma$ be a decision tree over the problem z. We correspond to each complete path $\xi$ in the tree $\Gamma$ a word $\pi \left(\xi \right)$ in the alphabet $\{(f_i,\delta ):f_i\in F\left(z\right),\delta \in \{0,1\}\}$. If the equation system $\mathcal{S}\left(\xi \right)$ is empty, then $\pi \left(\xi \right)$ is the empty word. If $\mathcal{S}\left(\xi \right)=\{f_{i_1}\left(x\right)={\sigma }_1,\dots ,f_{i_m}\left(x\right)={\sigma }_m\}$, then $\pi \left(\xi \right)=(f_{i_1},{\sigma }_1)\cdots (f_{i_m},{\sigma }_m)$. The decision tree $\Gamma$ over z solves the problem z relative to U if, for each complete path $\xi$ in $\Gamma$, the set ${\Delta }_U\left(z\right)\pi \left(\xi \right)$ contains at most one tuple, and if this set contains exactly one tuple, then the considered tuple is assigned to the terminal node of the path $\xi$.

As the time complexity of a decision tree $\Gamma$, we consider its depth $h(\Gamma )$, that is the maximum number of working nodes in a complete path in the tree $\Gamma$.

Let $z\in P\left(U\right)$. We denote by $h_U^{\left(1\right)}\left(z\right)$ the minimum depth of a decision tree over z, which solves z relative to U and uses only attributes from $F\left(z\right)$. We denote by $h_U^{\left(2\right)}\left(z\right)$ the minimum depth of a decision tree over z, which solves z relative to U and uses only hypotheses over z. We denote by $h_U^{\left(3\right)}\left(z\right)$ the minimum depth of a decision tree over z, which solves z relative to U and uses both attributes from $F\left(z\right)$ and hypotheses over z.

For $i=1,2,3$, we define a function of the Shannon type $h_U^{\left(i\right)}\left(n\right)$ that characterizes the dependence of $h_U^{\left(i\right)}\left(z\right)$ on $dimz$ in the worst case. Let $i\in \{1,2,3\}$ and $n\in \mathbb{N}$. Then:

$$h_U^{\left(i\right)}\left(n\right)=max\{h_U^{\left(i\right)}\left(z\right):z\in P\left(U\right),dimz\le n\}.$$

3. Main Results

Let $U=(A,F)$ be an infinite binary information system and $r\in \mathbb{N}$. The information system U is called r-reduced if, for each consistent on A system of equations over U, there exists a subsystem of this system that has the same set of solutions and contains at most r equations. We denote by $\mathcal{R}$ the set of infinite binary information systems each of which is r-reduced for some $r\in \mathbb{N}$.

The next theorem follows from the results obtained in [15], where we considered closed classes of test tables (decision tables). It also follows from the results obtained in [16], where we considered the weighted depth of decision trees.

Theorem 1.

Let U be an infinite binary information system. Then, the following statements hold:

(a) 

If $U\in \mathcal{R}$, then $h_U^{\left(1\right)}\left(n\right)=\Theta (logn)$;

(b) 

If $U\notin \mathcal{R}$, then $h_U^{\left(1\right)}\left(n\right)=n$ for any $n\in \mathbb{N}$.

A subset $\{f_1,\dots ,f_m\}$ of F is called independent if, for any ${\delta }_1,\dots ,{\delta }_m\in \{0,1\}$, the system of equations $\{f_1\left(x\right)={\delta }_1,\dots ,f_m\left(x\right)={\delta }_m\}$ is consistent on the set A. The empty set of attributes is independent by definition. We now define the independence dimension or I-dimension$I\left(U\right)$ of the information system U (this notion is similar to the notion of the independence number of the family of sets considered by Naiman and Wynn in [19]). If, for each $m\in \mathbb{N}$, the set F contains an independent subset of cardinality m, then $I\left(U\right)=\infty$. Otherwise, $I\left(U\right)$ is the maximum cardinality of an independent subset of the set F. We denote by $\mathcal{D}$ the set of infinite binary information systems with a finite independence dimension.

Let $U=(A,F)$ be a binary information system, which is not necessarily infinite, $f\in F$, and $\delta \in \{0,1\}$. Denote:

$$A(f,\delta )=\{a:a\in A,f(a)=\delta \}.$$

We now define inductively the notion of a k-information system, $k\in \mathbb{N}\cup \left\{0\right\}$. The binary information system U is called a 0-information system if all attributes from F are constant on the set A. Let, for some $k\in \mathbb{N}\cup \left\{0\right\}$, the notion of a m-information system be defined for $m=0,\dots ,k$. The binary information system U is called a $(k+1)$-information system if it is not a m-information system for $m=0,\dots ,k$ and, for any $f\in F$, there exist numbers $\delta \in \{0,1\}$ and $m\in \{0,\dots ,k\}$ such that the information system $\left(A\right(f,\delta ),F)$ is a m-information system. It is easy to show by induction on k that if $U=(A,F)$ is a k-information system, then $U^{\prime }=(A^{\prime },F)$, $A^{\prime }\subseteq A$, is a l-information system for some $l\le k$. We denote by $\mathcal{C}$ the set of infinite binary information systems for each of which there exists $k\in \mathbb{N}$ such that the considered system is a k-information system. The following theorem was presented in [17] without proof.

Theorem 2.

Let U be an infinite binary information system. Then, the following statements hold:

(a) 

If $U\in \mathcal{C}$, then $h_U^{\left(2\right)}\left(n\right)=O\left(1\right)$ and $h_U^{\left(3\right)}\left(n\right)=O\left(1\right)$;

(b) 

If $U\in \mathcal{D}\setminus \mathcal{C}$, then $h_U^{\left(2\right)}\left(n\right)=\Theta (logn)$, $h_U^{\left(3\right)}\left(n\right)=\Omega \left(\frac{logn}{loglogn}\right)$, and $h_U^{\left(3\right)}\left(n\right)=O(logn)$;

(c) 

If $U\notin \mathcal{D}$, then $h_U^{\left(2\right)}\left(n\right)=n$ and $h_U^{\left(3\right)}\left(n\right)=n$ for any $n\in \mathbb{N}$.

Let U be an infinite binary information system. We now consider the join behavior of the functions $h_U^{\left(1\right)}\left(n\right)$, $h_U^{\left(2\right)}\left(n\right)$, and $h_U^{\left(3\right)}\left(n\right)$. It depends on the belonging of the information system U to the sets $\mathcal{R}$, $\mathcal{D}$, and $\mathcal{C}$. We correspond to the information system U its indicator vector $ind\left(U\right)=(c_1,c_2,c_3)\in {\{0,1\}}^3$ in which $c_1=1$ if and only if $U\in \mathcal{R}$, $c_2=1$ if and only if $U\in \mathcal{D}$, and $c_3=1$ if and only if $U\in \mathcal{C}$.

Theorem 3.

For any infinite binary information system, its indicator vector coincides with one of the rows of

Table 1. Possible indicator vectors of infinite binary information systems.

	$\mathcal{R}$	$\mathcal{D}$	$\mathcal{C}$
1	0	0	0
2	0	1	0
3	0	1	1
4	1	1	0

. Each row of Table 1 is the indicator vector of some infinite binary information system.

For $i=1,2,3,4$, we denote by $V_i$ the class of all infinite binary information systems, for which the indicator vector coincides with the ith row of Table 1.

Table 2. Summary of Theorems 1–3.

	$\mathcal{R}$	$\mathcal{D}$	$\mathcal{C}$	${\mathit{h}}_{\mathit{U}}^{\left(1\right)}\left(\mathit{n}\right)$	${\mathit{h}}_{\mathit{U}}^{\left(2\right)}\left(\mathit{n}\right)$	${\mathit{h}}_{\mathit{U}}^{\left(3\right)}\left(\mathit{n}\right)$
$V_1$	0	0	0	n	n	n
$V_2$	0	1	0	n	$\Theta (logn)$	$\Omega \left(\frac{logn}{loglogn}\right),O(logn)$
$V_3$	0	1	1	n	$O\left(1\right)$	$O\left(1\right)$
$V_4$	1	1	0	$\Theta (logn)$	$\Theta (logn)$	$\Omega \left(\frac{logn}{loglogn}\right),O(logn)$

summarizes Theorems 1–3. The first column contains the name of complexity class $V_i$. The next three columns describe the indicator vector of information systems from this class. The last three columns $h_U^{\left(1\right)}\left(n\right)$, $h_U^{\left(2\right)}\left(n\right)$, and $h_U^{\left(3\right)}\left(n\right)$ contain information about the behavior of the functions $h_U^{\left(1\right)}\left(n\right)$, $h_U^{\left(2\right)}\left(n\right)$, and $h_U^{\left(3\right)}\left(n\right)$ for information systems from the class $V_i$.

4. Proof of Theorem 2

We precede with the proof of Theorem 2 by two lemmas.

Let $d\in \mathbb{N}$. A d-complete tree over the information system $U=(A,F)$ is a marked finite directed tree with the root in which:

• Each terminal node is not labeled;
• Each nonterminal node is labeled with an attribute $f\in F$. There are two edges leaving this node that are labeled with the systems of equations $\left\{f\right(x)=0\}$ and $\left\{f\right(x)=1\}$, respectively;
• The length of each complete path (the path from the root to a terminal node) is equal to d;
• For each complete path $\xi$, the equation system $\mathcal{S}\left(\xi \right)$, which is the union of equation systems assigned to the edges of the path $\xi$, is consistent.

Let G be a d-complete tree over U and $F\left(G\right)$ be the set of all attributes attached to the nonterminal nodes of the tree G. The number of nonterminal nodes in G is equal to $2^0+2^1+\dots +2^{d-1}=2^d-1$. Therefore, $\left|F\left(G\right)\right|\le 2^d$.

The results mentioned in the following lemma are obtained by methods similar to those used by Littlestone [12], Maass and Turán [13], and Angluin [11].

Lemma 1.

Let $U=(A,F)$ be a binary information system, $d\in \mathbb{N}$, G be a d-complete tree over U, and z be a problem over U such that $F\left(G\right)\subseteq F\left(z\right)$. Then

(a) 

$h_U^{\left(2\right)}\left(z\right)\ge d$;

(b) 

$h_U^{\left(3\right)}\left(z\right)\ge \frac{d}{{log}_2\left(2d\right)}$.

Proof. 

(a) We prove the inequality $h_U^{\left(2\right)}\left(z\right)\ge d$ by induction on d. Let $d=1$. Then, the tree G has the only one nonterminal node, which is labeled with an attribute f that is not constant on A. Therefore, $|{\Delta }_U\left(z\right)|\ge 2$ and $h_U^{\left(2\right)}\left(z\right)\ge 1$. Let, for $t\in \mathbb{N}$ and for any natural d, $1\le d\le t$, the considered statement hold. Assume now that $d=t+1$, G is a d-complete tree over U, $z$ is a problem over U such that $F\left(G\right)\subseteq F\left(z\right)$, and $\Gamma$ is a decision tree over z with the minimum depth, which solves the problem z and uses only hypotheses. Let f be the attribute attached to the root of the tree G and H be the hypothesis attached to the root of the decision tree $\Gamma$. Then, there is an edge that leaves the root of $\Gamma$ and is labeled with the equation system $\left\{f\right(x)=\delta \}$ where the equation $f\left(x\right)=\neg \delta$ belongs to the hypothesis H. This edge enters to the root of the subtree of $\Gamma$, which is denoted by ${\Gamma }_f$. There is an edge that leaves the root of G and is labeled with the equation system $\left\{f\right(x)=\delta \}$. This edge enters the root of the subtree of G, which is denoted by $G_{\delta }$. One can show that the decision tree ${\Gamma }_f$ solves the problem z relative to the information system $U^{\prime }=(A(f,\delta ),F)$ and $G_{\delta }$ is a t-complete tree over $U^{\prime }$. It is clear that $F\left(G_{\delta }\right)\subseteq F\left(z\right)$. Using the inductive hypothesis, we obtain $h\left({\Gamma }_f\right)\ge t$. Therefore, $h(\Gamma )\ge t+1=d$ and $h_U^{\left(2\right)}\left(z\right)\ge d$.

(b) We now prove the inequality $h_U^{\left(3\right)}\left(z\right)\ge \frac{d}{{log}_2\left(2d\right)}$. Let $z=(f_1,\dots ,f_n)$ and $\Gamma$ be a decision tree over z with the minimum depth, which solves the problem z and uses both attributes and hypotheses. The d-complete tree G has $2^d$ complete paths ${\xi }_1,\dots ,{\xi }_{2^d}$. For $i=1,\dots ,2^d$, we denote by $a_i$ a solution of the equation system $\mathcal{S}\left({\xi }_i\right)$. Denote $B=\{a_1,\dots ,a_{2^d}\}$. We now show that the decision tree $\Gamma$ contains a complete path, the length of which is at least $\frac{d}{{log}_2\left(2d\right)}$. We describe the process of this path construction beginning with the root of $\Gamma$.

Let the root of $\Gamma$ be labeled with an attribute $f_{i_0}$. For $\delta \in \{0,1\}$, we denote by $B^{\delta }$ the set of solutions on B of the equation system $\{f_{i_0}\left(x\right)=\delta \}$ and choose $\sigma \in \{0,1\}$ for which $|B^{\sigma }|=max\{|B^0|,|B^1\left|\right\}$. It is clear that $|B^{\sigma }|\ge \frac{\left|B\right|}{2}\ge \frac{\left|B\right|}{2d}$. In the considered case, the beginning of the constructed path in $\Gamma$ is the root of $\Gamma$, the edge that leaves the root and is labeled with the equation system $\{f_{i_0}\left(x\right)=\sigma \}$, and the node to which this edge enters.

Let as assume now that the root of $\Gamma$ is labeled with a hypothesis $H=\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$. We denote by ${\xi }_H$ the complete path in G for which the system of equations $\mathcal{S}\left({\xi }_H\right)$ is a subsystem of H. Let the nonterminal nodes of the complete path ${\xi }_H$ be labeled with the attributes $f_{i_1},\dots ,f_{i_d}$. For $j=1,\dots ,d$, we denote by $B_j$ the set of solutions on B of the equation system $\{f_{i_j}\left(x\right)=\neg {\delta }_{i_j}\}$. It is clear that $|B_1|+\cdots +|B_d|\ge |B|-1$. Therefore, there exists $l\in \{1,\dots ,d\}$ such that $|B_l|\ge \frac{\left|B\right|-1}{d}\ge \frac{\left|B\right|}{2d}$. In the considered case, the beginning of the constructed path in $\Gamma$ is the root of $\Gamma$, the edge that leaves the root and is labeled with the equation system $\{f_{i_l}\left(x\right)=\neg {\delta }_{i_l}\}$, and the node to which this edge enters.

We continue the construction of the complete path in $\Gamma$ in the same way such that after the tth query, we have at least $\frac{\left|B\right|}{{\left(2d\right)}^t}$ elements from B. The process of path construction continues at least until $\frac{\left|B\right|}{{\left(2d\right)}^t}\le 1$, i.e., at least until ${log}_2\left|B\right|\le t{log}_2\left(2d\right)$. Since $\left|B\right|=2^d,$ we have $h(\Gamma )\ge t\ge \frac{d}{{log}_2\left(2d\right)}$ and $h_U^{\left(3\right)}\left(z\right)\ge \frac{d}{{log}_2\left(2d\right)}$. □

Lemma 2.

Let $U=(A,F)$ be a binary information system, $k\in \mathbb{N}\cup \left\{0\right\}$, and U not be an m-information system for $m=0,\dots ,k$. Then, there exists a $(k+1)$-complete tree over U.

Proof. 

We prove the considered statement by induction on k. Let $k=0$. In this case, U is not a 0-information system. Then, there exists an attribute $f\in F$, which is not constant on A. Using this attribute, it is easy to construct a 1-complete tree over U.

Let the considered statement hold for some k, $k\ge 0$. We now show that it also holds for $k+1$. Let $U=(A,F)$ be a binary information system, which is not an m-information system for $m=1,\dots ,k+1$. Then, there exists an attribute $f\in F$ such that, for any $\delta \in \{0,1\}$, the information system $U_{\delta }=(A(f,\delta ),F)$ is not an m-information system for $m=1,\dots ,k$. Using the inductive hypothesis, we conclude that, for any $\delta \in \{0,1\}$, there exists a $(k+1)$-complete tree $G_{\delta }$ over $U_{\delta }$. Denote by G a directed tree with root in which the root is labeled with the attribute f, and for any $\delta \in \{0,1\}$, there is an edge that leaves the root, is labeled with the equation system $\left\{f\right(x)=\delta \}$, and enters the root of the tree $G_{\delta }$. One can show that the tree G is a $(k+2)$-complete tree over U. □

Proof of Theorem 2.

It is clear that $h_U^{\left(3\right)}\left(z\right)\le h_U^{\left(2\right)}\left(z\right)$ for any problem z over U. Therefore, $h_U^{\left(3\right)}\left(n\right)\le h_U^{\left(2\right)}\left(n\right)$ for any $n\in \mathbb{N}$.

(a) Let $k\in \mathbb{N}\cup \left\{0\right\}$. We now show by induction on k that, for each binary k-information system U (not necessarily infinite) for each problem z over U, the inequality $h_U^{\left(2\right)}\left(z\right)\le k$ holds. Let $U=(A,F)$ be a binary 0-information system and z be a problem over U. Since all attributes from $F\left(z\right)$ are constant on A, the set ${\Delta }_U\left(z\right)$ contains only one tuple. Therefore, the decision tree containing only one node labeled with this tuple solves the problem z relative to U, and $h_U^{\left(2\right)}\left(z\right)=0$.

Let $k\in \mathbb{N}\cup \left\{0\right\}$ and, for each m, $0\le m\le k$, the considered statement hold. Let us show that it holds for $k+1$. Let $U=(A,F)$ be a binary $(k+1)$-information system and $z=(f_1,\dots ,f_n)$ be a problem over U. For $i=1,\dots ,n$, choose a number ${\delta }_i\in \{0,1\}$ such that the information system $(A(f_i,\neg {\delta }_i),F)$ is an $m_i$-information system where $1\le m_i\le k$. Using the inductive hypothesis, we conclude that, for $i=1,\dots ,n$, there is a decision tree ${\Gamma }_i$ over z, which uses only hypotheses, solves the problem z over $(A(f_i,\neg {\delta }_i),F)$, and has depth at most $m_i$. We denote by $\Gamma$ a decision tree in which the root is labeled with the hypothesis $H=\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$, the edge leaving the root and labeled with H enters the terminal node labeled with the tuple $({\delta }_1,\dots ,{\delta }_n)$, and for $i=1,\dots ,n$, the edge leaving the root and labeled with $\{f_i\left(x\right)=\neg {\delta }_i\}$ enters the root of the tree ${\Gamma }_i$. One can show that $\Gamma$ solves the problem z relative to U and $h(\Gamma )\le k+1$. Therefore, $h_U^{\left(2\right)}\left(z\right)\le k+1$ for any problem z over U.

Let $U\in \mathcal{C}$. Then, U is a k-information system for some natural k, and for each problem z over U, we have $h_U^{\left(3\right)}\left(z\right)\le h_U^{\left(2\right)}\left(z\right)\le k$. Therefore, $h_U^{\left(2\right)}\left(n\right)=O\left(1\right)$ and $h_U^{\left(3\right)}\left(n\right)=O\left(1\right)$.

(b) Let $U=(A,F)\in \mathcal{D}\setminus \mathcal{C}$. First, we show that $h_U^{\left(2\right)}\left(n\right)=O(logn)$. Let $z=(f_1,\dots ,f_n)$ be an arbitrary problem over U. From Lemma 5.1 [16], it follows that $|{\Delta }_U{\left(z\right)|\le \left(4n\right)}^{I\left(U\right)}$. The proof of this lemma is based on results similar to the ones obtained by Sauer [20] and Shelah [21]. We consider a decision tree $\Gamma$ over z, which solves z relative to U and uses only hypotheses. This tree is constructed by the halving algorithm [1,12]. We describe the work of this tree for an arbitrary element a from A. Set $\Delta =$${\Delta }_U\left(z\right)$. If $|\Delta |=1$, then the only n-tuple from $\Delta$ is the solution $z\left(a\right)$ of the problem z for the element a. Let $|\Delta |\ge 2$. For $i=1,\dots ,m$, we denote by ${\delta }_i$ a number from $\{0,1\}$ such that $|\Delta (f_i,{\delta }_i)|\ge |\Delta (f_i,\neg {\delta }_i)|$. The root of $\Gamma$ is labeled with the hypothesis $H=\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$. After this query, either the problem z is solved (if the answer is H) or we halve the number of objects in the set $\Delta$ (if the answer is a counterexample $\{f_i\left(x\right)=\neg {\delta }_i\}$). In the latter case, set $\Delta =$${\Delta }_U\left(z\right)(f_i,\neg {\delta }_i)$. The decision tree $\Gamma$ continues to work with the element a and the set of n-tuples $\Delta$ in the same way. Let, during the work with the element a, the considered decision tree make q queries. After the $(q-1)$th query, the number of remaining n-tuples in the set $\Delta$ is at least two and at most ${\left(4n\right)}^{I\left(U\right)}/2^{q-1}$. Therefore, $2^q\le {\left(4n\right)}^{I\left(U\right)}$ and $q\le I\left(U\right){log}_2\left(4n\right)$. Therefore, during the processing of the element a, the decision tree $\Gamma$ makes at most $I\left(U\right){log}_2\left(4n\right)$ queries. Since a is an arbitrary element from A, the depth of $\Gamma$ is at most $I\left(U\right){log}_2\left(4n\right)$. Since z is an arbitrary problem over U, we obtain $h_U^{\left(2\right)}\left(n\right)=O(logn)$. Therefore, $h_U^{\left(3\right)}\left(n\right)=O(logn)$.

Using Lemma 2 and the relation $U\notin \mathcal{C}$, we obtain that, for any $d\in \mathbb{N}$, there exists d-complete tree $G_d$ over U. Let $F\left(G_d\right)=\{f_1,\dots ,f_{n_d}\}$. We know that $n_d\le 2^d$. Denote $z_d=(f_1,\dots ,f_{n_d})$. From Lemma 1, it follows that $h_U^{\left(2\right)}\left(z_d\right)\ge d$ and $h_U^{\left(3\right)}\left(z_d\right)\ge \frac{d}{{log}_2\left(2d\right)}$. As a result, we have $h_U^{\left(2\right)}\left(2^d\right)\ge d$ and $h_U^{\left(3\right)}\left(2^d\right)\ge \frac{d}{{log}_2\left(2d\right)}$. Let $n\in \mathbb{N}$ and $n\ge 8$. Then, there exists $d\in \mathbb{N}$ such that $2^d\le n<2^{d+1}$. We have $d>{log}_{2}n-1$, $h_U^{\left(2\right)}\left(n\right)\ge {log}_{2}n-1$, $h_U^{\left(2\right)}\left(n\right)=\Omega (logn)$, and $h_U^{\left(2\right)}\left(n\right)=\Theta (logn)$. It is easy to show that the function $\frac{x}{{log}_2\left(2x\right)}$ is nondecreasing for $x\ge 2$. Therefore, $h_U^{\left(3\right)}\left(n\right)\ge \frac{{log}_{2}n-1}{{log}_2\left(2({log}_{2}n-1)\right)}$ and $h_U^{\left(3\right)}\left(n\right)=\Omega \left(\frac{logn}{loglogn}\right)$.

(c) Let $U=(A,F)\notin \mathcal{D}$. We now consider an arbitrary problem $z=(f_1,\dots ,f_n)$ over U and a decision tree over z, which uses only hypotheses and solves the problem z over U in the following way. For a given element $a\in A$, the first query is about the hypothesis $H_1=\{f_1\left(x\right)=1,\dots ,f_n\left(x\right)=1\}$. If the answer is $H_1$, then the problem z is solved for the element a. If, for some $i\in \{1,\dots ,n\}$, the answer is $\{f_i\left(x\right)=0\}$, then the second query is about the hypothesis $H_2$ obtained from $H_1$ by replacing the equality $f_i\left(x\right)=1$ with the equality $f_i\left(x\right)=0$, etc. It is clear that after at most n queries, the problem z for the element a will be solved. Thus, $h_U^{\left(2\right)}\left(z\right)\le n$ and $h_U^{\left(3\right)}\left(z\right)\le n$. Since z is an arbitrary problem over U, we have $h_U^{\left(2\right)}\left(n\right)\le n$ and $h_U^{\left(3\right)}\left(n\right)\le n$ for any $n\in \mathbb{N}$.

Let $n\in \mathbb{N}$. Since $U\notin \mathcal{D}$, there exist attributes $f_1,\dots ,f_n\in F$ such that, for any $({\delta }_1,\dots ,{\delta }_n)\in {\{0,1\}}^n$, the equation system $\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$ is consistent on A. We now consider the problem $z=(f_1,\dots ,f_n)$ and an arbitrary decision tree $\Gamma$ over z, which solves the problem z over U and uses both attributes and hypotheses. Let us show that $h(\Gamma )\ge n$. If $n=1$, then the considered inequality holds since $|{\Delta }_U\left(z\right)|\ge 2$. Let $n\ge 2$. It is easy to show that an equation system over z is inconsistent if and only if it contains equations $f_i\left(x\right)=0$ and $f_i\left(x\right)=1$ for some $i\in \{1,\dots ,n\}$. For each node v of the decision tree $\Gamma$, we denote by $S_v$ the union of systems of equations attached to edges in the path from the root of $\Gamma$ to v. A node v of $\Gamma$ will be called consistent if the equation system $S_v$ is consistent.

We now construct a complete path $\xi$ in the decision tree $\Gamma$, for which the nodes are consistent. We start from the root that is a consistent node. Let the path reach a consistent node v of $\Gamma$. If v is a terminal node, then the path $\xi$ is constructed. Let v be a working node labeled with an attribute $f_i\in F\left(z\right)$. Then, there exists $\delta \in \{0,1\}$ for which the system of equations $S_v\cup \{f_i\left(x\right)=\delta \}$ is consistent. Then, the path $\xi$ will pass through the edge leaving v and labeled with the system of equations $\{f_i\left(x\right)=\delta \}$. Let v be labeled with a hypothesis $H=\{f_1\left(x\right)={\delta }_1,\dots ,f_n\left(x\right)={\delta }_n\}$. If there exists $i\in \{1,\dots ,n\}$ such that the system of equations $S_v\cup \{f_i\left(x\right)=\neg \delta \}$ is consistent, then the path $\xi$ will pass through the edge leaving v and labeled with the system of equations $\{f_i\left(x\right)=\neg \delta \}$. Otherwise, $S_v=H$, and the path $\xi$ will pass through the edge leaving v and labeled with the system of equations H.

Let all edges in the path $\xi$ be labeled with systems of equations containing one equation each. Since all nodes of $\xi$ are consistent, the equation system $\mathcal{S}\left(\xi \right)$ is consistent. We now show that $\mathcal{S}\left(\xi \right)$ contains at least n equations. Let us assume that this system contains less than n equations. Then, the set ${\Delta }_U\left(z\right)\pi \left(\xi \right)$ contains more than one n-tuple, which is impossible. Therefore, the length of the path $\xi$ is at least n. Let there be edges in $\xi$, which are labeled with hypotheses, and the first edge in $\xi$ labeled with a hypothesis H leaves the node v. Then, $S_v=H$, and the length of $\xi$ is at least n. Therefore, $h(\Gamma )\ge n$, $h_U^{\left(3\right)}\left(z\right)\ge n$, and $h_U^{\left(2\right)}\left(z\right)\ge n$. As a result, we obtain $h_U^{\left(3\right)}\left(n\right)\ge n$ and $h_U^{\left(2\right)}\left(n\right)\ge n$. Thus, $h_U^{\left(2\right)}\left(n\right)=n$ and $h_U^{\left(3\right)}\left(n\right)=n$ for any $n\in \mathbb{N}$. □

5. Proof of Theorem 3

First, we prove several auxiliary statements.

Proposition 1.

$\mathcal{R}\subseteq \mathcal{D}$.

Proof. 

Let $U\in \mathcal{R}$. By Theorem 1, $h_U^{\left(1\right)}\left(n\right)=\Theta (logn)$. Let us assume that $U\notin \mathcal{D}$. Then, for any $n\in \mathbb{N}$, there exists a problem $z=(f_1,\dots ,f_n)$ over U such that $|{\Delta }_U\left(z\right)|=2^n$. Let $\Gamma$ be a decision tree over z, which solves the problem z relative to U and uses only attributes. Then, $\Gamma$ should have at least $2^n$ terminal nodes. One can show that the number of terminal nodes in the tree $\Gamma$ is at most $2^{h(\Gamma )}$. Then, $2^n\le 2^{h(\Gamma )}$, $h(\Gamma )\ge n$, and $h_U\left(z\right)\ge$$n.$ Therefore, $h_U^{\left(1\right)}\left(n\right)\ge n$ for any $n\in \mathbb{N}$, which is impossible. Thus, $\mathcal{R}\subseteq \mathcal{D}$. □

Proposition 2.

$\mathcal{C}\subseteq \mathcal{D}$.

Proof. 

Let $U\in \mathcal{C}$. By Theorem 2, $h_U^{\left(2\right)}\left(n\right)=O\left(1\right)$. Let us assume that $U\notin \mathcal{D}$. Then, by Theorem 2, $h_U^{\left(2\right)}\left(n\right)=n$ for any $n\in \mathbb{N}$, which is impossible. Therefore, $\mathcal{C}\subseteq \mathcal{D}$. □

Proposition 3.

$\mathcal{R}\cap \mathcal{C}=\varnothing$.

Proof. 

Assume the contrary: $\mathcal{R}\cap \mathcal{C}\ne \varnothing$ and $U=(A,F)\in \mathcal{R}\cap \mathcal{C}$. Let $r,k\in \mathbb{N}$, U be an r-reduced information system and U be a k-information system. We now consider an arbitrary problem $z=(f_1,\dots ,f_n)$ over U and describe a decision tree $\Gamma$ over z, which uses only attributes, solves the problem z over U, and has depth at most $kr$.

For $i=1,\dots ,n$, let ${\delta }_i$ be a number from $\{0,1\}$ such that $(A(f_i,\neg {\delta }_i),F)$ is an $m_i$-information system with $0\le m_i<k$. Let t be the maximum number from the set $\{1,\dots ,n\}$ such that the system of equations $S=\{f_1\left(x\right)={\delta }_1,\dots ,f_t\left(x\right)={\delta }_t\}$ is consistent. Then, there exists a subsystem $\{f_{i_1}\left(x\right)={\delta }_{i_1},\dots ,f_{i_p}\left(x\right)={\delta }_{i_p}\}$ of the system S, which has the same set of solutions as S and for which $p\le r$. For a given $a\in A$, the decision tree $\Gamma$ computes sequentially values $f_{i_1}\left(a\right),\dots ,f_{i_p}\left(a\right)$.

If, for some $q\in \{1,\dots ,p\}$, $f_{i_1}\left(a\right)={\delta }_{i_1},\dots ,f_{i_{q-1}}\left(a\right)={\delta }_{i_{q-1}}$, and $f_{i_q}\left(a\right)=\neg {\delta }_{i_q}$, then the decision tree $\Gamma$ continues to work with the problem z and the information system $U^{\prime }=(A^{\prime },F)$ where $A^{\prime }$ is the set of solutions on A of the equation system $\{f_{i_1}\left(x\right)={\delta }_{i_1},\dots ,f_{i_{q-1}}\left(x\right)={\delta }_{i_{q-1}},f_{i_q}\left(x\right)=\neg {\delta }_{i_q}\}$. We have that $U^{\prime }$ is an $l^{\prime }$-information system for some $l^{\prime }\le m_{i_q}<k$.

Let $f_{i_1}\left(a\right)={\delta }_{i_1},\dots ,f_{i_p}\left(a\right)={\delta }_{i_p}$. If $t=n$, then $({\delta }_1,\dots ,{\delta }_n)$ is the solution of the problem z for the considered element a. Let $t<n$. Then, the decision tree $\Gamma$ continues to work with the problem z and the information system $U^{″}=(A^{″},F)$ where $A^{″}$ is the set of solutions on A of the equation system $\{f_{i_1}\left(x\right)={\delta }_{i_1},\dots ,f_{i_p}\left(x\right)={\delta }_{i_p}\}$. We know that the equation system $\{f_1\left(x\right)={\delta }_1,\dots ,f_t\left(x\right)={\delta }_t,f_{t+1}\left(x\right)={\delta }_{t+1}\}$ is inconsistent. Therefore, the system $\{f_{i_1}\left(x\right)={\delta }_{i_1},\dots ,f_{i_p}\left(x\right)={\delta }_{i_p},f_{t+1}\left(x\right)={\delta }_{t+1}\}$ is inconsistent. Hence, $A^{″}\subseteq A(f_{t+1},\neg {\delta }_{t+1})$ and $U^{″}$ is an $l^{″}$-information system for some $l^{″}\le m_{t+1}<k$.

As a result, after the computation of the values of at most r attributes, we either solve the problem z or reduce the consideration of the problem z over the k-information system U to the consideration of the problem z over some l-information system where $l<k$. After the computation of the values of at most $rk$ attributes, we solve the problem z since each problem over the 0-information system has exactly one possible solution. Therefore, $h_U^{\left(1\right)}\left(z\right)\le rk$ and $h_U^{\left(1\right)}\left(n\right)=O\left(1\right)$. By Theorem 1, $h_U^{\left(1\right)}\left(n\right)=\Theta (logn)$. The obtained contradiction shows that $\mathcal{R}\cap \mathcal{C}=\varnothing$. □

Proposition 4.

For any infinite binary information system, its indicator vector coincides with one of the rows of Table 1.

Proof. 

Table 3. All 3-tuples from the set ${\{0,1\}}^3$.

	$\mathcal{R}$	$\mathcal{D}$	$\mathcal{C}$
1	0	0	0
2	0	1	0
3	0	1	1
4	1	1	0
5	1	0	0
6	0	0	1
7	1	0	1
8	1	1	1

contains as rows all three-tuples from the set ${\{0,1\}}^3$. We now show that the rows with the numbers 5–8 cannot be indicator vectors of infinite binary information systems. Assume the contrary: there is $i\in \{5,6,7,8\}$ such that the row with the number i is the indicator vector of an infinite binary information system U. If $i=5$, then $U\in \mathcal{R}$ and $U\notin \mathcal{D}$, but this is impossible, since, by Proposition 1, $\mathcal{R}\subseteq \mathcal{D}$. If $i=6$, then $U\in \mathcal{C}$ and $U\notin \mathcal{D}$, but this is impossible, since, by Proposition 2, $\mathcal{C}\subseteq \mathcal{D}$. If $i=7$, then $U\in \mathcal{R}$ and $U\notin \mathcal{D}$, but this is impossible, since, by Proposition 1, $\mathcal{R}\subseteq \mathcal{D}$. If $i=8$, then $U\in \mathcal{R}$ and $U\in \mathcal{C}$, but this is impossible, since, by Proposition 3, $\mathcal{R}\cap \mathcal{C}=\varnothing$. Therefore, for any infinite binary information system, its indicator vector coincides with one of the rows of Table 3 with Numbers 1–4. Thus, it coincides with one of the rows of Table 1. □

Define an infinite binary information system $U_1=(A_1,F_1)$ as follows: $A_1=\mathbb{N}$ and $F_1$ is the set of all functions from $\mathbb{N}$ to $\{0,1\}$.

Lemma 3.

The information system $U_1$ belongs to the class $V_1$.

Proof. 

It is easy to show that the information system $U_1$ has an infinite I-dimension. Therefore, $U_1\notin \mathcal{D}$. Using Proposition 4, we obtain $ind\left(U\right)=(0,0,0)$, i.e., $U_1\in V_1$. □

For any $i\in \mathbb{N}$, we define two functions $p_i:\mathbb{N}\to \{0,1\}$ and $l_i:\mathbb{N}\to \{0,1\}$. Let $j\in \mathbb{N}$. Then, $p_i\left(j\right)=1$ if and only if $j=i$ and $l_i\left(j\right)=1$ if and only if $j>i$.

Define an infinite binary information system $U_2=(A_2,F_2)$ as follows: $A_2=\mathbb{N}$ and $F_2=\{p_i:i\in \mathbb{N}\}\cup \{l_i:i\in \mathbb{N}\}$.

Lemma 4.

The information system $U_2$ belongs to the class $V_2$.

Proof. 

For $n\in \mathbb{N}$, denote $S_n=\{p_1\left(x\right)=0,\dots ,p_n\left(x\right)=0\}$. One can show that the equation system $S_n$ is consistent and each proper subsystem of $S_n$ has a set of solutions different from the set of solutions of $S_n$. Therefore, $U_2\notin \mathcal{R}$. Using attributes from the set $\{l_i:i\in \mathbb{N}\}$, we can construct a d-complete tree over $U_2$ for each $d\in \mathbb{N}$. By Lemma 1 and Theorem 2, $U_2\notin \mathcal{C}$. One can show that $I\left(U_2\right)=1$. Therefore, $U_2\in \mathcal{D}$. Thus, $ind\left(U_2\right)=(0,1,0)$, i.e., $U_2\in V_2$. □

Define an infinite binary information system $U_3=(A_3,F_3)$ as follows: $A_3=\mathbb{N}$ and $F_3=\{p_i:i\in \mathbb{N}\}$.

Lemma 5.

The information system $U_3$ belongs to the class $V_3$.

Proof. 

It is easy to show that $U_3$ is a 1-information system. Therefore, $U_3\in \mathcal{C}$. Using Proposition 4, we obtain $ind\left(U_3\right)=(0,1,1)$, i.e., $U_3\in V_3$. □

Define an infinite binary information system $U_4=(A_4,F_4)$ as follows: $A_4=\mathbb{N}$ and $F_4=\{l_i:i\in \mathbb{N}\}$.

Lemma 6.

The information system $U_4$ belongs to the class $V_4$.

Proof. 

Let us consider an arbitrary consistent system of equations S over $U_4$. We now show that there is a subsystem of S, which has at most two equations and the same set of solutions as S. Let S contain both equations of the kind $l_i\left(x\right)=1$ and $l_j\left(x\right)=0$. Denote $i_0=max\{i:l_i\left(x\right)=1\in S\}$ and $j_0=min\{j:l_j\left(x\right)=0\in S\}$. One can show that the system of equations $S^{\prime }=\{l_{i_0}\left(x\right)=1,l_{j_0}\left(x\right)=0\}$ has the same set of solutions as S. The case when S contains for some $\delta \in \{0,1\}$ only equations of the kind $l_p\left(x\right)=\delta$ can be considered in a similar way. In this case, the equation system $S^{\prime }$ contains only one equation. Therefore, the information system $U_4$ is 2-reduced and $U_4\in$$\mathcal{R}$. Using Proposition 4, we obtain $ind\left(U_4\right)=(1,1,0)$, i.e., $U_4\in V_4$. □

Proof of Theorem 3.

From Proposition 4, it follows that, for any infinite binary information system, its indicator vector coincides with one of the rows of Table 1. Using Lemmas 3–6, we conclude that each row of Table 1 is the indicator vector of some infinite binary information system. □

6. Conclusions

Based on the results of exact learning, test theory, and rough set theory, for an arbitrary infinite binary information system, we studied three functions of the Shannon type, which characterize the dependence in the worst case of the minimum depth of a decision tree solving a problem on the number of attributes in the problem description. These three functions correspond to (i) decision trees using attributes, (ii) decision trees using hypotheses, and (iii) decision trees using both attributes and hypotheses. We described possible types of behavior for each of these three functions. We also studied the join behavior of these functions and distinguished four corresponding complexity classes of infinite binary information systems. In the future, we plan to translate the obtained results into the language of exact learning.

The problems studied in this paper allow us to confine ourselves to considering only the crisp (conventional) sets that are completely defined by attributes. However, in the future, when we investigate approximately defined problems or approximate decision trees, it will be necessary to work with rough sets given by their lower and upper approximations. This will require a wider range of rough set theory techniques than those used in the present paper.