Determinization and Minimization of Automata for Nested Words Revisited

Abstract

We consider the problem of determinizing and minimizing automata for nested words in practice. For this we compile the nested regular expressions ($\mathit{NREs}$) from the usual XPath benchmark to nested word automata ($\mathit{NWAs}$). The determinization of these $\mathit{NWAs}$, however, fails to produce reasonably small automata. In the best case, huge deterministic $\mathit{NWAs}$ are produced after few hours, even for relatively small $\mathit{NREs}$ of the benchmark. We propose a different approach to the determinization of automata for nested words. For this, we introduce stepwise hedge automata ($\mathit{SHA}$s) that generalize naturally on both (stepwise) tree automata and on finite word automata. We then show how to determinize $\mathit{SHA}$s, yielding reasonably small deterministic automata for the $\mathit{NREs}$ from the XPath benchmark. The size of deterministic $\mathit{SHA}$s automata can be reduced further by a novel minimization algorithm for a subclass of $\mathit{SHA}$s. In order to understand why the new approach to determinization and minimization works so nicely, we investigate the relationship between $\mathit{NWAs}$ and $\mathit{SHA}$s further. Clearly, deterministic $\mathit{SHA}$s can be compiled to deterministic $\mathit{NWAs}$ in linear time, and conversely $\mathit{NWAs}$ can be compiled to nondeterministic $\mathit{SHA}$s in polynomial time. Therefore, we can use $\mathit{SHA}$s as intermediates for determinizing $\mathit{NWAs}$, while avoiding the huge size increase with the usual determinization algorithm for $\mathit{NWAs}$. Notably, the $\mathit{NWAs}$ obtained from the $\mathit{SHA}$s perform bottom-up and left-to-right computations only, but no top-down computations. This $\mathit{NWA}$ behavior can be distinguished syntactically by the (weak) single-entry property, suggesting a close relationship between $\mathit{SHA}$s and single-entry $\mathit{NWAs}$. In particular, it turns out that the usual determinization algorithm for $\mathit{NWAs}$ behaves well for single-entry $\mathit{NWAs}$, while it quickly explodes without the single-entry property. Furthermore, it is known that the class of deterministic multi-module single-entry $\mathit{NWAs}$ enjoys unique minimization. The subclass of deterministic $\mathit{SHA}$s to which our novel minimization algorithm applies is different though, in that we do not impose multiple modules. As further optimizations for reducing the sizes of the constructed $\mathit{SHA}$s, we propose schema-based cleaning and symbolic representations based on apply-else rules that can be maintained by determinization. We implemented the optimizations and report the experimental results for the automata constructed for the XPathMark benchmark.

1. Introduction

Nested words are hierarchical structures that are omnipresent in computer science. They were used to represent sequences of data trees, like XML or JSON documents, and to analyze the call structure of recursive programs. The idea of nested words is to generalize on both words and trees, resulting in sequences of unranked trees that are also known as hedges. Otherwise, nested words can be obtained by enriching Dyck words with internal letters, besides opening and closing parentheses. Furthermore, nested words are the elements of the least set containing internal letters from a given alphabet, triples consisting of an opening parenthesis, a nested word, and a closing parenthesis, and all sequences of nested words. Last but not least, nested words can be seen as words over an alphabet with internal letters, opening parentheses, and closing parentheses, under the conditions that the parenthesis are well nested, so that every opening parenthesis is properly closed and every closing parenthesis properly opened.

From the viewpoint of formal language theory, a natural question is how to lift the notions of finite automata and regular expressions, from words and trees to nested words, while preserving their well-known relationships. Nested word automata ($\mathit{NWAs}$) were heavily studied since the 1980s  [1,2,3,4], under the name input-driven automata. They are the same as visibly pushdown automata [5], pushdown forest automata [6], and streaming tree automata [7]. $\mathit{NWAs}$ can recognize the same languages of unranked trees as hedge automata [8], a generalization of tree automata for ranked trees [9]. $\mathit{NWAs}$ are often defined as pushdown automata with visible stacks, meaning that exactly one symbol is pushed when reading an opening parenthesis, and exactly one symbol is popped when reading a closing parenthesis, while the stack is not used otherwise. Their main advantage is a powerful notion of determinism, generalizing both over bottom-up and top-down determinism of tree automata for ranked trees [2,3]. We note that general pushdown automata do not permit determinization in contrast.

Regular expressions for nested words were proposed more recently by Hosoya and Pierce [10] under the name of regular expression types. In the present article, we will call them nested regular expressions ($\mathit{NREs}$) instead. Independently, more complex notions of nested regular expressions were introduced  [11,12] in order to deal with generalizations of nested words with dangling opening and closing parentheses, which are not of interest to us. It was already claimed in [10], that our simpler notion of $\mathit{NREs}$ has the same expressiveness as hedge automata [8,9], which in turn have the same expressiveness as $\mathit{NWAs}$  [3]. However, the question under which conditions $\mathit{NREs}$ can be compiled to small deterministic $\mathit{NWAs}$ has not been studied. For classes of $\mathit{NREs}$ for which deterministic $\mathit{NWAs}$ can be computed in polynomial time, we can decide language inclusion or equivalence in polynomial time too. For other classes, these problems may not be feasible since language inclusion for nondeterministic $\mathit{NWAs}$ is EXP-complete.

Our concrete interest in the universality of deterministic $\mathit{NWAs}$ is motivated by XML stream processing: we want to compute the certain answers of a navigational XPath query on an XML stream  [13,14], i.e., those elements that are selected in all possible futures of the stream. Whether an answer is certain is computationally hard, even for tiny syntactic fragments of navigational XPath  [14,15], but can be done in polynomial time for queries defined by deterministic $\mathit{NWAs}$  [16]. A natural question is, therefore, whether it is possible to compile navigational XPath queries as in the usual benchmark [17] to deterministic $\mathit{NWAs}$ of reasonable size. Unfortunately, the existing compilers fail to do so [18], as they are based on $\mathit{NWA}$ determinization for dealing with disjunction, negation, and recursive steps. Thereby, they produce huge deterministic automata even for very simple navigational XPath queries from the benchmark, or do not terminate after some hours.

In this article, we consider $\mathit{NREs}$ for defining queries on nested words. For benchmarking with realistic example, we consider the navigational XPath queries in the XPathMark benchmark with only forwards axis, that we compiled to $\mathit{NREs}$ of the same size up to a constant factor. The question is then whether these $\mathit{NREs}$ can be compiled to reasonably small deterministic $\mathit{NWAs}$.

As a first approach, we distinguish a subclass of “deterministic” $\mathit{NREs}$ that can be compiled in polynomial time to deterministic $\mathit{NWAs}$ by generalizing on Glushkov’s construction of deterministic finite-state automata (DFAs) from “deterministic” regular expressions  [19,20]. However, the $\mathit{NREs}$ obtained by compilation from navigational XPath queries are rarely deterministic, so neither are the $\mathit{NWAs}$ compiled from them. Moreover, as we cannot apply $\mathit{NWA}$ determinization to them in practice as argued above, this first approach has a much too low coverage to reach the objective. Therefore, we will report it only at the end in Section 9.

For our second approach, we propose a novel variant of automata for nested words that we call stepwise hedge automata ($\mathit{SHA}$s). Even though motivated by the wish to create deterministic automata for the $\mathit{NREs}$ of our benchmark, they are of general interest: they generalize naturally on both (stepwise) tree automata [21] and on finite word automata. In contrast to stepwise tree automata, $\mathit{SHA}$s can not only recognize unranked trees, but also sequences thereof, i.e., hedges or nested words. Furthermore, $\mathit{SHA}$s can be determinized in a bottom-up and left-to-right manner by combining in a natural manner the determinization procedures for tree and word automata.

By adapting existing compilers for stepwise tree automata  [21], $\mathit{SHA}$s can be compiled to $\mathit{NWAs}$ with the same language in linear time while preserving determinism. Conversely, $\mathit{NWAs}$ can be compiled to $\mathit{SHA}$s in polynomial time, but at the cost of introducing nondeterminism. By compiling $\mathit{NWAs}$ to $\mathit{SHA}$s, determinizing the $\mathit{SHA}$, and compiling the obtained deterministic $\mathit{SHA}$ back to a deterministic $\mathit{NWA}$, we can determinize $\mathit{NWAs}$ by determinizing the corresponding $\mathit{SHA}$s. This alternative determinization algorithm for $\mathit{NWAs}$ is different from the usual determinization algorithm for $\mathit{NWAs}$  [2,3,18]. Indeed, it yields reasonably small deterministic $\mathit{NWAs}$ for the $\mathit{NREs}$ from the XPath benchmark.

Yet another alternative algorithm for determinizing $\mathit{NWAs}$ can be obtained by compiling $\mathit{NWAs}$ to $\mathit{SHA}$s and back, and then determinizing the $\mathit{NWAs}$ obtained in this manner. When applied to back-and-forth converted $\mathit{NWAs}$, the usual $\mathit{NWA}$ determinization algorithm turns out to be well behaved: it produces deterministic $\mathit{NWAs}$ of reasonable size for all our benchmark $\mathit{NREs}$. This might be surprising, given that the same determinization algorithm behaved so poorly for the non-converted $\mathit{NWAs}$ that were obtained from the benchmark $\mathit{NREs}$  directly.

We contribute two further solutions for producing deterministic $\mathit{NWAs}$ for our benchmark $\mathit{NREs}$. These are both based on a direct compiler from $\mathit{NREs}$ to $\mathit{SHA}$s. We can then determinize these $\mathit{SHA}$s, followed by compilation to deterministic $\mathit{NWAs}$. Otherwise, we can first compile the $\mathit{SHA}$s to $\mathit{NWAs}$ and then determinize these NWAs.

The next question is why the new determinization algorithms for $\mathit{NWAs}$ that use $\mathit{SHA}$s as intermediates work so nicely. In order to understand this, we need to investigate the relationship between $\mathit{NWAs}$ and $\mathit{SHA}$s more deeply. Clearly, the $\mathit{NWAs}$ obtained via $\mathit{SHA}$s do all their work in a bottom-up and left-to-right manner, and nothing when moving top-down. We can characterize the subclass of $\mathit{NWAs}$ with this restricted behavior syntactically by the (weak) single-entry property: it requires that all opening rules of the $\mathit{NWA}$ go into into the same target state while popping the sources state onto the stack. Note that our single-entry property is weaker than the (multi-module) single-entry property studied previously [22,23,24], which in addition requires that the automaton can be split into at least 2 modules, one for the top level and one for the nested level. The $\mathit{NWAs}$ obtained by compilation from $\mathit{SHA}$s all have the (weak) single-entry property (but not necessarily multiple modules). Therefore, when compiling $\mathit{NWAs}$ to $\mathit{SHA}$s and back, the resulting $\mathit{NWAs}$ also have the (weak) single-entry property. It seems that the usual determinization algorithm from $\mathit{NWAs}$ is well-behaved when applied to $\mathit{NWAs}$ with the (weak) single-entry property. The relationship between $\mathit{SHA}$s and (weak) single-entry $\mathit{NWAs}$ seems sufficiently close, so that their determinization algorithms seem to operate in somehow similar manners.

It is known that the subclass of deterministic multi-module single-entry $\mathit{NWAs}$ (also called call-driven automata) enjoys unique minimization  [22,23]. The separation of the module for the top level from the module for the nested level can be obtained w.l.o.g., by building a product with the $\mathit{NWA}$ with two hedge states that distinguishes the two levels. In our application, minimization could thus be used to reduce the size of the deterministic $\mathit{NWAs}$ produced by our four algorithms for converting $\mathit{NREs}$, with the hope to eventually obtain a unique outcome after minimization. However, as we will use some symbolic representations for sets of rules, the uniqueness will hold only for the non-symbolic counterpart. In any case, the number of states of the deterministic minimal $\mathit{NWAs}$ obtained for the same $\mathit{NRE}$ could be expected to become unique.

Motivated by our application, we found it more relevant to minimize deterministic $\mathit{SHA}$s rather than deterministic (weak) single-entry $\mathit{NWAs}$ (despite of their close correspondence). As for the class of deterministic (weak) single-entry $\mathit{NWAs}$, a restriction is needed for the class of deterministic $\mathit{SHA}$s to obtain unique minimization. We could have required the existence of multiple modules as for multi-module single-entry $\mathit{NWAs}$. Instead, we restrict ourselves to deterministic $\mathit{SHA}$s for which the initial states for trees and hedges coincide. We then show that minimization for such $\mathit{SHA}$s can be reduced to the minimization of tree automata up to a novel encoding of hedges to binary trees.

We implemented all four algorithms for compiling our benchmark $\mathit{NREs}$ to deterministic $\mathit{NWAs}$ and report the experimental results. We have also implemented the novel minimization algorithm for $\mathit{SHA}$s with equal tree and hedge initial states, and used it in our experiments. We propose two further optimization methods for reducing the sizes of the constructed automata.

First, we introduce schema-based cleaning both for $\mathit{SHA}$s and $\mathit{NWAs}$. In our application, the schema expresses the Xml data model, stating that hedges must encode valid Xml documents. More generally, an automaton A can be cleaned relative to an automaton S for the schema, if the language of interest is the intersection $L(A)\cap L(S)$ rather than $L(A)$ itself. The idea of schema-based cleaning is to keep only those transition rules of A that are used to recognize some hedge of $L(S)$. These transition rules can be computed from the product of A and S. Note that schema-based cleaning may change the language of the automaton. Only the intersection $L(A)\cap L(S)$ is preserved, not necessarily $L(A)$.

Second, we propose a symbolic representations for $\mathit{SHA}$s based on apply-else rules. They help to represent more compactly a large number of apply rules produced by the determinization of $\mathit{SHA}$s. Before compiling $\mathit{SHA}$s to $\mathit{NWAs}$, however, we need to eliminate the apply-else rules. This is because we have not developed analogous symbolic representations for $\mathit{NWAs}$ so far. A second limitation is that we have not implemented any minimization algorithm for $\mathit{NWAs}$ at the time being.

The main improvement of this journal article compared with the conference version [25] is the addition of the minimization algorithm for the subclass of $\mathit{SHA}$s with equal tree and hedge initial states. Furthermore, we added the idea of schema-based cleaning and the symbolic representations for $\mathit{SHA}$s by apply-else rules. The experimental results were enhanced with minimization, symbolic representations of rules, and schema-based cleaning. All the nested regular expressions generated for the XPathMark benchmark queries that we consider, as well as their corresponding automata—when we could produce them—can be found at http://researchers.lille.inria.fr/niehren/complementary-material (accessed on 1 February 2021).

Related Work. In the present article, we restrict ourselves to nested words over signatures with a single opening parenthesis, a single closing parenthesis, and possibly many internal letters $a,b$. This permits us to simplify the presentation of nested regular expressions, the notions of $\mathit{NWAs}$ and $\mathit{SHA}$s, their forth and back compilers, as well as the determinization algorithms. Note that any multi-module $\mathit{NWA}$ for such signatures must have exactly 2 modules. From an application perspective, multiple parentheses can be encoded by using internal letters, that is a named opening parenthesis ${⟨}_a$ by the word $a·⟨$ and a named closing parenthesis ${⟩}_b$ by the word $b·⟩$. When encoding XML documents as nested words, such some encoding is needed anyway in order to deal with the complex information in XML tags, and also to provide symbolic representations with else rules that are able to deal with infinite signatures.   

For the minimization of deterministic $\mathit{NWAs}$, general signatures with multiple parentheses raise additional problems. Chervet and Walukiewicz  [23] solved such problems by reducing the minimization for expanded CDAs to the minimization of CDAs. Gauwin, Muscholl, and Raskin [24] showed that the minimization for deterministic $\mathit{NWAs}$ is NP-hard in the case with general signatures. Their approach is based on a reduction from the problem of minimal immersion for sequences of Dfas, for which they construct $\mathit{NWAs}$ with an unbounded number of opening parenthesis and an unbounded number of entry states. Weak single-entry $\mathit{NWAs}$ in our setting do not permit this. Neither do $\mathit{NWAs}$ over fixed general signatures with a finite number of opening parenthesis.

Navigational XPath queries on XML documents can be formalized in the language CoreXPath [26], or more generally by nested regular path queries [27] on data trees. Nested regular path queries were introduced earlier under the name of the propositional dynamic logic (PDL) in the 1970s  [28], where they are applied to labeled graphs that generalize on data trees.

As certain query answering for XPath was considered difficult, the currently existing approaches to XPath query evaluation on XML streams [13,18] either approximate certain query answers based on nondeterministic machines or restrict the queries so that answer certainty can be decided without latency [15,29]. This also holds for recent streaming algorithms on words without nesting in the context of complex event processing [30].

2. Nested Words

Nested words are words with parentheses that are well nested. They can be identified with hedges, that is, sequences of internal symbols and unranked trees.

Nested words are constructed with opening and closing parentheses, respectively, ⟨ and ⟩. An unranked alphabet $\Sigma$ is a possibly infinite set of so-called “internal” symbols, that does not contain the two parentheses. The set of nested words over $\Sigma$ is denoted ${\mathcal{N}}_{\Sigma }$ and is defined by the following abstract syntax:

$$\begin{array}{ccc}\hfill h,h^{\prime }\in {\mathcal{N}}_{\Sigma }& ::=& \epsilon \left|a\right|⟨h⟩|h·h^{\prime }\mathrm{where}a\in \Sigma \hfill \end{array}$$

The empty nested word is denoted by $\epsilon$ and assumed to be the neutral element of the composition operator $\epsilon ·h=h=h·\epsilon$, which furthermore is assumed to be associative, i.e., $h_1·(h_2·h_3)=(h_1·h_2)·h_3$.

Nested words can be identified with hedges, i.e., words of trees and internal symbols. Seen as a graph, the inner nodes are labeled by the tree constructor $⟨⟩$ and the leaves by symbols in $\Sigma$ or the tree constructor. For instance, $⟨a·⟨b⟩·\epsilon ⟩·c·⟨d·⟨\epsilon ⟩⟩$ corresponds to the hedge in Figure 1. A nested word of type tree has the form $⟨h⟩$.

Figure 1. Nested word $⟨a·⟨b⟩·\epsilon ⟩·c·⟨d·⟨\epsilon ⟩⟩$ seen as a graph.

Variants. Our notion of nested words accepts only well-nested words without dangling opening or closing parentheses in contrast to others  [3,5]. This will lead to simpler notion of regular expressions, avoiding the more complex operators as with visibly rational expressions  [12,31]. A less important difference is that we do not support labeled parentheses.

Labeled unranked trees. Labeled parentheses can be simulated by using internal letters. For instance, the labeled tree $a(b(),c())$ can be represented by the nested word of type tree $⟨a·⟨b⟩·⟨c⟩⟩$. In this way, the labeled tree $a()$ is represented by the nested word $⟨a⟩$, which is of type tree (while the internal letter a alone is not). Unranked sequences of subtrees, often called hedges and sometimes forests, can be composed by using the sequence operator.   

XML Documents. Our notion of nested words is sufficiently powerful to express general XML documents. An example of an XML document is given in Figure 2 and the representing nested word in Figure 3.

Figure 2. An Xml document.

Figure 3. The corresponding nested word.

We use the names of XML elements as labels of the nested word, as well as the letters of UTF8 for the string data values. Further labels such as doc and elem are added to express the types of the XML data model document and element, respectively.

When it comes to querying for nodes in Xml documents, we will be interested in nested words encoding Xml documents, in which a unique node is marked. We will use the label x to mark the selected node and the label $\neg x$ for all others. When marking the date in the XML document of Figure 2, we obtain the nested word in Figure 4.

Figure 4. The nested word of the x-marked Xml document from Figure 2.

3. Nested Regular Expressions

We present nested regular expressions ($\mathit{NREs}$), which were introduced under the name regular expression types in the context of XDuce [10] up to minor details. Note that similar nested regular expressions for ranked trees are folklore in the context of tree automata [32].

3.1. Syntax and Semantics

Let the alphabet $\Sigma$ be a set. An $\mathit{NRE}$ over $\Sigma$ is a term describing a language of nested words. It has the following abstract syntax where $a\in \Sigma$:

$$E,E^{\prime }::=\epsilon \mid a\mid \_\mid \varnothing \mid E·E^{\prime }\mid E+E^{\prime }\mid E\&E^{\prime }\mid E^{*}\mid \overline{E}\mid ⟨E⟩\mid \mu a.E$$

The $\mu a.E$ expressions are the same as in $\mu$-calculus [33], except that we restrict them such that all occurrences of a in E are nested below parentheses. Otherwise, nonregular languages could be defined such as with $\mu a.(b·a·c+\epsilon )$ whose language would be $\{b^n·c^n\mid n\in \mathbb{N}\}$. We also forbid intersections and complements in expression $\mu a.E$ on all paths between the $\mu a$-operator and the occurrences of a in E that are bound by this operator. The expressions $\mu a.E$ allow for vertical recursion, while the expressions with the Kleene star $E^{*}$ support horizontal recursion.

Our syntax allows for conjunctions $E\&E^{\prime }$ and negations $\overline{E}$, which are well known to not add expressiveness if $\Sigma$ is finite. They are still relevant from the viewpoints of modeling, and for the treatment of infinite signatures. This comes at the price of increasing the complexity, as for the well-known case of words [34].

For infinite signatures, we can define for any finite subset ${\Sigma }^{\prime }$ of labels the language of single-letter words $\Sigma \backslash {\Sigma }^{\prime }$ by some $\mathit{NRE}$. This can be seen as follows. If ${\Sigma }^{\prime }=\varnothing$, then the expression _ does the job: it matches exactly the set of all labels in $\Sigma$. Moreover, if ${\Sigma }^{\prime }$ is nonempty then we can use negation. For instance, if ${\Sigma }^{\prime }=\{a,b\}$ then the expression $\overline{a+b}\&\_$ describes the language $\Sigma \backslash {\Sigma }^{\prime }$.

The sets of free and bound letters $\mathit{fn}(E)$ and $\mathit{bn}(E)$ are defined as usual. The only binder $\mu a.E$ binds the symbol a with scope E. Note that $fn(\_)=\varnothing$.

There are three differences with respect to the regular expression types from  [10]. First, our $\mathit{NREs}$ treat labels as internal symbols instead of labels of parentheses. Second, they provide recursion through the $\mu$-operator instead of using recursive equation systems. Third, conjunctions and general negations are not considered there.

Any $\mathit{NRE}$ E describes a language $L(E)$ of nested words that we define by induction on the structure of E as follows: ${\mathcal{N}}_{\Sigma }$ is the set of nested words over $\Sigma$, as defined in Section 2.

$$\begin{array}{cc}L(\epsilon )=\{\epsilon \}L(a)=\{a\}\hfill & L(\_)=\Sigma \hfill \\ L(E·E^{\prime })=L(E)·L(E^{\prime })\hfill & L(\overline{E})={\mathcal{N}}_{\Sigma }\backslash L(E)\hfill \\ L(E+E^{\prime })=L(E)\cup L(E^{\prime })\hfill & L(E\&E^{\prime })=L(E)\cap L(E^{\prime })\hfill \\ L(⟨E⟩)=\{⟨h⟩\mid h\in L(E)\}\hfill & L(\mu a.E)={\cup }_{n\ge 0}L({\mu }^{n}a.E)\hfill \\ L(E^{*})=L{(E)}^{*}\hfill & L(\varnothing )=\varnothing \hfill \end{array}$$

For all expressions—$E,E_1$, and $E_2$, the notation $E[E_1/E_2]$ stands for the expression E where all the occurrences of $E_1$ have been replaced by $E_2$. The semantics of a $\mu$-operator is then defined using the shortcuts ${\mu }^{0}a.E=E[a/\varnothing ]$ and ${\mu }^{n}a.E=E[a/{\mu }^{n-1}a.E]$ for all $n\ge 1$. In particular $L(\mu a.\_)=L(\_)=\Sigma$, so that $a\in L(\mu a.\_)$. The semantics of the complement expression $L(\overline{E})$ is the complement of $L(E)$ in the set of all nested words, that is ${\mathcal{N}}_{\Sigma }\backslash L(E)$.

3.2. XPath  Example

We now show how to express navigational XPath queries by $\mathit{NREs}$ that are restricted to forward axis. The idea is to adapt the spirit of a generate-and-test algorithm for query answering. The generation produces a nested word from Xml documents by guessing a single node and marking it by x. This node is a candidate for a query answer that is to be tested. The test is done by a $\mathit{NRE}$.

For expressing XPath queries with child and descendant-or-self axes we will use the following $\mathit{NREs}$ where $a\notin \mathit{fn}(E)$:

$$\begin{array}{ccc}T\hfill & {=}_{\mathrm{df}}\hfill & \mu a.{(⟨a⟩+\_)}^{*}\hfill \\ ch(E)\hfill & {=}_{\mathrm{df}}\hfill & T·⟨E⟩·T\hfill \\ c{h}^{*}(E)\hfill & {=}_{\mathrm{df}}\hfill & \mu a.(E+ch(a))\hfill \\ c{h}^{+}(E)\hfill & {=}_{\mathrm{df}}\hfill & \mu a.(ch(E)+ch(a))\hfill \end{array}$$

For instance, consider the XPath query A5 from the XPathMark benchmark  [17]:

/site/closed_auctions/closed_auction[descendant::keyword]/date

Applied to the above Xml document, it selects all date children of closed_auctions nodes that contain at least one keyword descendant. Query A5 can be compiled to the following $\mathit{NRE}$, which will accept the nested word in Figure 4 in particular:

	$⟨doc·\_·⟨elem·site·\_·ch(elem·closed\_auctions·\_·ch($
	$elem·closed\_auction·\_·(c{h}^{+}(elem·keyword·\_·T)\&ch(elem·date·x·T))))⟩⟩$

The only label that the expression _ may match on a document that is properly annotated with the variable x will be the letter $\neg x\in \Sigma$. The label x is annotated to the marked node, which is tested for being selected by the query. The label $\neg x$ is annotated to all nodes except a unique x-marked node.

Note also that the $\mu$-operator of the $c{h}^{+}(\dots )$-expression expresses the recursion of the descendant axis. Furthermore, the conjunction permits us to connect the main path of A5 with its only filter.

3.3. XPath Benchmark

For testing $\mathit{NREs}$, we rely on the usual XPathMark benchmark  [17]. We restrict ourselves to navigational path queries with forward axis: child, descendant, and following-sibling. We notice that the following axis is excluded in contrast to following-sibling, as following is not strictly forwards. We can also admit path composition and filters with conjunction, disjunction, and negation.

The XPath queries of the benchmark satisfying these restrictions are the queries A1, …, A8 and B3 given in Figure 5. We developed a more general compiler from navigational forward XPath queries to $\mathit{NREs}$, which yields the $\mathit{NREs}$ in Figure A1 of the Appendix B for the benchmark XPath queries. The $\mathit{NREs}$ for A1–A3 do have neither conjunctions nor negations, while the queries A4–A8 contain filters, which are mapped to conjunctions in $\mathit{NREs}$. The compiler uses the $\mu$-operator to capture the recursion of descendant axis as in A2, A3, and A5. Furthermore, nondeterminism is introduced by disjunctions in filters as in A7 and A8. Conjunction in filters appears in A6 which is mapped to conjunctions in $\mathit{NREs}$ too. A detailed description of this compiler is not in the scope of the present article though.

Figure 5. XPath benchmark queries.

4. Nested Word Automata

Nested word automata ($\mathit{NWAs}$) are pushdown automata reading nested words, whose stacks are visible: they push a single stack symbol when reading an opening parenthesis, pop a single stack symbol when reading a closing parenthesis, and do not alter or inspect the stack otherwise.

Definition 1.

An $\mathit{NWA}$ is a tuple $A=(Q_h,Q_t,\Sigma ,\Gamma ,\Delta ,I,F)$ consisting of a possibly infinite set Σ of internal symbols; finite sets $Q_h$ and $Q_t$ of states of type hedge and tree, respectively; sets of initial and final states $I,F\subseteq Q_h$; a finite set Γ of stack symbols; and a finite set Δ of transition rules of the forms:

$$\begin{array}{ccc}\mathit{hedge}\mathit{rules}\hfill & a^{\Delta },{\_}^{\Delta },{\epsilon }^{\Delta }\subseteq Q_h\times Q_h\hfill & \mathit{where}a\in \Sigma \hfill \\ \mathit{opening}\mathit{rules}\hfill & {⟨}_{\gamma }^{\Delta }\subseteq Q_h\times Q_h\hfill & \mathit{where}\gamma \in \Gamma \hfill \\ \mathit{tree}\mathit{rules}\hfill & \mathtt{T}{}^{\Delta }\subseteq Q_h\times Q_t\hfill & \\ \mathit{closing}\mathit{rules}\hfill & {⟩}_{\gamma }^{\Delta }\subseteq Q_t\times Q_h\hfill & \end{array}$$

Our $\mathit{NWAs}$ are symbolic, in that they come with else rules, i.e., elements of $(q,q^{\prime })\in {\_}^{\Delta }$ that we will denote by $q\stackrel{\_}{\to }{q}^{\prime }$, for dealing with large or infinite alphabets.

An example for an $\mathit{NWA}$ is given in a graphical syntax in Figure 6. Tree states are drawn in circles that are filled in light gray , while hedge states are in unfilled circles $\overline{)q}$. Initial states are drawn as $\to \overline{)q}$ and final states as . Hedge rules that have the form $(q_1,q_2)\in o^{\Delta }$ where $o\in \Sigma \cup \{\_,\epsilon \}$ are denoted by $q_1\stackrel{o}{\to }{q}_2$, while any tree rule $(q_1,q_2)\in {\mathtt{T}}^{\Delta }$ is denoted $q_1-\to q_2$. Opening rules $(q_1,q_2){\in ⟨}_{\gamma }^{\Delta }$ are represented as $q_1\stackrel{\downarrow \gamma }{\mathbf{-}🠆}{q}_2$ and closing rules $(q_1,q_2){\in ⟩}_{\gamma }^{\Delta }$ as $q_1\stackrel{\uparrow \gamma }{\mathbf{-}🠆}{q}_2$.

Figure 6. Nested word automaton $nwa(c{h}^{*}(a+b))$.

Our notion of $\mathit{NWAs}$ supports factorization in the spirit of the work in [35]. It is obtained by distinguishing two types of states, $q\in Q_h$ and $p\in Q_t$, and adding explicit type coercion rules $q-\to p$. Semantically, both kinds of states could be merged when replacing the type coercion rules by the epsilon rule $q\stackrel{\epsilon }{\to }p$, but at the cost of introducing additional nondeterminism. This may lead to quadratically larger deterministic automata, as we will illustrate at the $\mathit{NWA}$ in Figure 20.

The language of nested words between two states $q_1,q_2\in Q_h$ is defined as the least language such that

$$\begin{array}{ccc}\hfill L_{q_1,q_2}(\Delta )& =& \{\epsilon \mid \mathrm{if}{q}_1=q_2\mathrm{or}{q}_1\stackrel{\epsilon }{\to }{q}_2\in \Delta \}\cup {\displaystyle \bigcup _{q_3\in Q_h}}{L}_{q_1,q_3}(\Delta )·L_{q_3,q_2}(\Delta )\hfill \\ & \cup & \{a\mid \mathrm{if}{q}_1\stackrel{a}{\to }{q}_2\in \Delta \mathrm{or}(q_1\stackrel{\_}{\to }{q}_2\in \Delta \mathrm{and}\neg \exists q_2^{\prime }.q_1\stackrel{a}{\to }{q}_2^{\prime }\in \Delta )\}\hfill \\ & \cup & \{⟨h⟩\mid \exists q_1^{\prime },q_2^{\prime }\in Q_h.\exists q_3\in Q_t.\exists \gamma \in \Gamma .q_1\stackrel{\downarrow \gamma }{\stackrel{}{\to }}{q}_1^{\prime },h\in L_{q_1^{\prime },q_2^{\prime }}(\Delta ),\hfill \\ & & q_2^{\prime }-\to q_3\in \Delta \mathrm{and}{q}_3\stackrel{\uparrow \gamma }{\mathbf{-}🠆}{q}_2\in \Delta \}.\hfill \end{array}$$

The language of the $\mathit{NWA}$ is then $L(A)={\displaystyle \bigcup _{q_1\in I,q_2\in F}}{L}_{q_1,q_2}(\Delta )$.

4.1. Determinization of $\mathit{NWAs}$

Determinization for $\mathit{NWAs}$ was first studied by von Braunmühl and Verbeek [2] in the 1980s, where $\mathit{NWAs}$ are named input-driven pushdown automata. We notice that the determinization algorithm was published only in the journal version of this paper, but not in the conference version. Later on, the same algorithm was rediscovered in the context of visibly pushdown automata and republished for nested word automata.

Definition 2.

An $\mathit{NWA}$ A is called deterministic or equivalently a $\mathit{dNWA}$ if

• I contains at most one element;
• there is no epsilon rule, i.e., ${\epsilon }^{\Delta }=\varnothing$,
• $a^{\Delta }$ and ${\_}^{\Delta }$ are partial functions from $Q_h$ to $Q_h$ for all $a\in \Sigma$, and ${\mathtt{T}}^{\Delta }$ is a partial function from $Q_h$ to $Q_t$;
• for all $q\in Q_h$ and $\gamma \in \Gamma$ there exists a most one $q^{\prime }\in Q_h$ such that $q^{\prime }{\in ⟨}_{\gamma }^{\Delta }$; and
• ${⟩}_{\gamma }^{\Delta }$ is a partial function from $Q_t$ to $Q_h$ for all $\gamma \in \Gamma$.

Proposition 1

(von Braunmühl and Verbeek [2]). A $\mathit{NWA}$ with n states can be determinized in time $O(2^{n^2})$.

Many of our results are based on the determinization algorithm going back to von Braunmühl and Verbeek. For self-containedness, we recall the version of this algorithm that we will use in the Appendix A. For illustrations, the determinization of the $\mathit{NWA}$ in Figure 6 is also presented here too, see Figure A1. It has size 271 while the nondeterministic $\mathit{NWA}$ has size 39 (12 states + 2 letters + 3 stack symbols + 22 rules). The blow-up is even worse in general as our experimental results will show and as noticed earlier by [18].

4.2. Multi-Module $\mathit{NWAs}$

Multi-module $\mathit{NWAs}$ will play a prominent role for our $\mathit{NWA}$ constructions and are relevant for minimization [23]. For signatures with a single opening parenthesis, each multi-module $\mathit{NWAs}$ has exactly two modules, one for the top level and one for the nested level.

We can define multi-module $\mathit{NWAs}$ based on the natural notion of homomorphisms for $\mathit{NWAs}$. A homomorphism from an $\mathit{NWA}$ A to an $\mathit{NWA}$ $A^{\prime }$ with the same signature is a triple of functions $({\alpha }_h:Q_h\to Q_h^{\prime },{\alpha }_t:Q_t\to Q_t^{\prime },\beta :\Gamma \to {\Gamma }^{\prime })$ that maps all concepts of A to the corresponding concepts of $A^{\prime }$. These concepts are hedge initial states, final states, opening, closing, internal, and tree transitions. We do not enforce the preservation of epsilon rules by homomorphisms.

Definition 3.

A multi-module $\mathit{NWA}$ A is an $\mathit{NWA}$ for which there exists a homomorphism from A to the $\mathit{NWA}$ $\mathcal{T}$ in Figure 7.

Figure 7. The $\mathit{NWA}$ $\mathcal{T}$ maps top level positions to state 0 and nested positions to 1 or $1^{\prime }$.

The $\mathit{NWA}$ $\mathcal{T}$ evaluates all top level positions of a nested word to state 0: all those positions that are not between parentheses. All nested positions are evaluated to state 1. The homomorphism of a multi-module $\mathit{NWA}$ A to $\mathcal{T}$ thus partitions the states of A between those that can be assigned to top level positions, and the others that can be assigned to nested positions.

4.3. Compilation of $\mathit{NREs}$ to $\mathit{NWAs}$

We next discuss a compiler from $\mathit{NREs}$ E to $\mathit{NWAs}$ $\mathit{nwa}(E)$. This compiler extends on the McNaughton–Yamada–Thompson algorithm [36] for regular expressions, which introduces epsilon edges for constructing the automata of composition $E·E^{\prime }$.

Theorem 1.

For any $\mathit{NRE}$ E, we can construct an $\mathit{NWA}$ A such that $L(A)=L(E)$. If E contains neither conjunctions nor negations, then the construction is in time $O(|E|)$.

Proof sketch.

Conjunctions $E\&E^{\prime }$ are compiled to products of automata, so repeated conjunctions may lead to an exponential blow up. Negations $\overline{E}$ are computed by complementing automata based on determinization. Each complementation may lead to an exponential blow-up, so when this is repeated, the construction may become non-elementary.

For expressions without conjunction and negation, no such blow-up may arise. As stated by the theorem, we have to show that expressions can be compiled in linear time.

Case $E=E^{\prime }·E^{″}$:

We use the McNaughton–Yamada–Thompson algorithm for composing the $\mathit{NWAs}$ of $\mathit{nwa}(E^{\prime })$ and $\mathit{nwa}(E^{″})$.

Case $E=⟨E^{\prime }⟩$:

Let $Q_h^{\prime }$, $Q_t^{\prime }$, and ${\Gamma }^{\prime }$ be, respectively, the set of hedge states, tree states, and stack symbols of $\mathit{nwa}(E^{\prime })$. We consider new hedge states $q_i$ and $q_f$ that are not in $Q_h^{\prime }$, a new tree state p not in $Q_t^{\prime }$ and a new stack symbol $\gamma$ not in ${\Gamma }^{\prime }$. Then, $\mathit{nwa}(E)$ is constructed by adding to $\mathit{nwa}(E^{\prime })$ opening rules $q_i\stackrel{\downarrow \gamma }{\mathbf{-}🠆}q$ for all the initial states q of $\mathit{nwa}(E^{\prime })$, tree rules $q^{\prime }-\to p$ for all the final states $q^{\prime }$ of $\mathit{nwa}(E^{\prime })$ and a closing rule $p\stackrel{\uparrow \gamma }{\mathbf{-}🠆}{q}_f$. Furthermore, we set $q_i$ as the only initial state of $\mathit{nwa}(E)$, and $q_f$ as its sole final state.

Case $E=\mu a.E^{\prime }$:

Special care has to be given to repeat expression $\mu a.E$. First of all, the naive compilation approach for these expression turns out to be wrong. Second, fixing the problem in the simplest possible manner does not lead to a linear time algorithm.

Note that we can assume w.l.o.g. that a occurs at most once in E by using the golden lemma of the $\mu$ calculus [37], stating for all names $a_1,\dots ,a_n$ and expressions $E^{″}$ in which $a_1,\dots ,a_n$ can appear free that $\mu a_1.\dots .\mu a_n.E^{″}\equiv \mu a.E^{″}[a_1/a,\dots ,a_n/a]$. Our construction guarantees that all transitions of the form $q\stackrel{a}{\to }{q}^{\prime }$ in $\mathit{nwa}(E)$ will start with the same state q. The wrong naive construction would remove the transitions $q\stackrel{a}{\to }{q}^{\prime }$ from $\mathit{nwa}(E)$ and add $\epsilon$-rules from q to all the initial states of $\mathit{nwa}(E)$, and from all final states of $\mathit{nwa}(E)$ to $q^{\prime }$. Unfortunately, the construction is not correct. For illustration, we consider the $\mathit{NRE}$ $E=\mu a.⟨a^{*}⟩$. The reader should be warned that constructing an $\mathit{NWA}$ for E is less trivial than it might seem at first sight. One has to start from the $\mathit{NWA}$ for $⟨a^{*}⟩$ which is given in Figure 8. Simply adding epsilon edges to capture the operator $\mu a$ will not work though. It will lead to the wrong automaton in Figure 9. This automaton will wrongly accept the hedge $⟨⟩⟨⟩$, as this hedge does not belong to $L(E)$.

Figure 8. Automaton for the $⟨a^{*}⟩$ expression.

Figure 9. Wrong naive construction for $\mu a.⟨a^{*}⟩$.

If the $\mathit{NWA}$ for E is multi-module, then the naive construction of compiling $\mu a.E$ can be made correct. Therefore, the simplest fix is to make the $\mathit{NWA}$ multi-moduled, before applying the naive construction. This can be achieved by typing the states of the automaton, by states of the $\mathit{NWA}$ $\mathcal{T}$ in Figure 7. The added types yield the homomorphism of the constructed automaton to $\mathcal{T}$.

The naive algorithm is then adapted as follows. Let $\mathcal{P}$ be the multi-module $\mathit{NWA}$ obtained from the product of $\mathit{nwa}(E)$ and $\mathcal{T}$. Note that we keep only the accessible top level states (type 0), but all nested states (type 1). In our example, this yields the $\mathit{NWA}$ in Figure 10. We then remove transition $(q,1)\stackrel{a}{\to }(q^{\prime },1)$ and add $\epsilon$-rules from state $(q,1)$ to all states in $I\times \{1\}$, and from all states in $F\times \{1\}$ to $(q^{\prime },2)$, where I and F are, respectively, the set of initial and final states of $\mathit{nwa}(E)$. Then, $\mathcal{P}$ recognizes $L(\mu a.E)$. The result obtained in the example is shown in Figure 11.

Figure 10. The multi-module $\mathit{NWA}$ for $⟨a^{*}⟩$.

Figure 11. The correctly adapted naive construction for $\mu a.⟨a^{*}⟩$.

The algorithm described so far makes the $\mathit{NWA}$ multi-moduled before compiling a $\mu$-operator. For this, two copies of all states are introduced. This, however, could lead to an exponential construction if multiple $\mu$-operators are nested. This problem can be avoided by preserving multi-moduledness as an invariant. Whenever a new state is created, it is created twice: once for the top level and once for the nested level. This information is maintained by typing the states, so that no further copies of the same state are produced later on.

We omit the correctness proof of this construction. □

4.4. Experimental Results Starting with the $\mathit{NWA}$  Compiler

In the first two column of Figure 12, we report the sizes of the $\mathit{NWAs}$ obtained from $\mathit{NREs}$ by our compiler, and the size of the deterministic $\mathit{NWAs}$ produced thereof. For each automaton, we give its total size and in parentheses the number of states.

Figure 12. The size (#states) of the nested word automata ($\mathit{NWAs}$) for the benchmark nested regular expressions ($\mathit{NREs}$) and the automata derived thereof.

The sizes of the nondeterministic $\mathit{NWAs}$ produced by the compiler for the $\mathit{NREs}$ for A1–A8 and B3 are given in column $nwa(.)$ of Figure 12. Note that the $\mathit{NWAs}$ are cleaned so that only accessible and co-accessible states remain. The sizes of the nondeterministic $\mathit{NWAs}$ are acceptable for all $\mathit{NREs}$, except for A8, for which the $\mathit{NWA}$ has more than 3000 states and an overall size greater than $10,000$. This can be partially explained by the fact that the $\mathit{NRE}$ for A8 contains three conjunctions (one for the filter and two for the conjunctions in the filter). Still, the number of states remains surprising.

The determinized $\mathit{NWAs}$ are given in column $det(nwa(.))$. It turns out that only A2 and A3 could be determinized successfully with some few hours of computation time on a standard laptop. However, even in the successful cases, the resulting deterministic $\mathit{NWAs}$ are simply huge. This confirms similar problems first noticed in [18] and not solved since then.

The remaining columns of Figure 12 based on the back-and-forth compiler from $\mathit{SHA}$s to $\mathit{NWAs}$ from the following Section 6. They show that better determinization algorithms can indeed be obtained, yielding $\mathit{NWAs}$ of acceptable size for all benchmark queries, with the exception of A8. The idea of $det(nwa(step(nwa(.)))$ is to compile the $\mathit{NWAs}$ obtained from the $\mathit{NREs}$ to stepwise hedge automata and back before applying the above algorithm for $\mathit{NWAs}$. This might be surprising, as this determinization algorithm failed for the original $\mathit{NWAs}$, while it now proves successful on the forth-and-back transformed $\mathit{NWAs}$.

5. Stepwise Hedge Automata

We propose $\mathit{SHA}$s as an extension of stepwise tree automata [21] that allows to recognize not only unranked trees but also hedges. We avoid more classical hedge automata from [9] that were already introduced in 1967 by Thatcher [8], as their notion of determinism is problematic. For instance, it makes unique minimization fail [38] and universality hard.

Our notion of $\mathit{SHA}$s will be symbolic in using else rules, and factorized as in [35]: there are two types of states for hedges and trees and an operator for explicit type coercion. We also propose a novel treatment of internal letters inspired by nested word automata, so that $\mathit{SHA}$s generalize both on stepwise tree automata and on Nfas.

Definition 4.

A $\mathit{SHA}$ is a tuple $A=(Q_h,Q_t,\Sigma ,\Delta ,I,F)$ such that $Q_t$ and $Q_h$ are finite sets of states of two types: t for tree and h for hedge, respectively; Σ an alphabet of internal letters (that may be infinite); $I,F\subseteq Q_h$ are subsets of hedge initial and final states, respectively; and Δ is a finite set of transition rules such that for all $q\in Q_t$ and $a\in \Sigma$:

$$\begin{array}{cc}\mathrm{hedge}\mathrm{rules}\hfill & q^{\Delta },a^{\Delta },{\_}^{\Delta },{\epsilon }^{\Delta }\subseteq Q_h\times Q_h\hfill \\ \mathrm{tree}\mathrm{final}\mathrm{rules}\hfill & {\mathtt{T}}^{\Delta }\subseteq Q_h\times Q_t\hfill \\ \mathrm{tree}\mathrm{initial}\mathrm{states}\hfill & {⟨⟩}^{\Delta }\subseteq Q_h\hfill \end{array}$$

An example for a $\mathit{SHA}$ is given in graphical syntax in Figure 13. It recognizes all hedges that are either just a or b or contain some tree node that contains either just a or b. In the graphical syntax, the states of type tree $q\in Q_t$ are drawn in circles filled in light gray , while the states of type hedge $q^{\prime }\in Q_h$ are drawn in unfilled circles $\overline{)q^{\prime }}$. The right part of the graph is an Nfa which uses tree states as additional edge labels, while the left part is a stepwise tree automaton that defines the tree languages of these tree states.

Figure 13. Stepwise hedge automaton $\mathrm{step}(c{h}^{*}(a+b))$: the part with the stepwise tree automaton is on the left and middle, and the Nfa part on the right.

Let ${\Delta }_h$ be the restriction of $\Delta$ to the hedge rules. Then, $(Q_h,\Sigma \uplus Q_t,{\Delta }_h,I,F)$ is a standard Nfa with $\epsilon$-rules, which is symbolic [39] in providing else rules for dealing with large or infinite alphabets in addition. Therefore, we denote the hedge initial states $q\in I$ by $\stackrel{h}{\to }\overline{)q}$ and the final states $q\in F$ by . A rule with an internal letter $(q_1,q_2)\in a^{\Delta }$ is denoted by $q_1\stackrel{a}{\to }{q}_2\in \Delta$ stating that a hedge in state $q_1$ can be extended by the internal letter a leading to a hedge in state $q_2$. Similarly, an epsilon rule $(q_1,q_2)\in {\epsilon }^{\Delta }$ is denoted by $q_1\stackrel{\epsilon }{\to }{q}_2$, and an else rule $(q_1,q_2)\in {\_}^{\Delta }$ is denoted by $q_1\stackrel{\_}{\to }{q}_2$. In the same spirit, a hedge rule $(q_1,q_2)\in q^{\Delta }$—also called apply rule—is denoted by $q_1\stackrel{q}{\mathbf{-}🠆}{q}_2\in \Delta$, stating that a hedge in state $q_1$ can be extended by a tree in state q leading to a hedge in state $q_2$.

A tree initial state $q\in {⟨⟩}^{\Delta }$ is graphically denoted by $\stackrel{t}{\to }\overline{)q}$ and a tree final rule $(q_1,q_2)\in {\mathtt{T}}^{\Delta }$ by $q_1-\to q_2$. Intuitively, a tree $⟨h⟩$ can be evaluated to state q if h can be evaluated starting with some tree initial state $q_1\in {⟨⟩}^{\Delta }$ to some state $q_2$ such that $q_2-\to q\in \Delta$. More formally, the hedge languages $L_{q_1,q_2}(A)$ between any two hedge states $q_1,q_2\in Q_h$ are defined as

$$\begin{array}{ccc}\hfill L_{q_1,q_2}(A)& =& \{\epsilon \mid \mathrm{if}{q}_1=q_2\mathrm{or}{q}_1\stackrel{\epsilon }{\to }{q}_2\in \Delta \}\cup {\displaystyle \bigcup _{q_3\in Q_h}}{L}_{q_1,q_3}(A)·L_{q_3,q_2}(A)\hfill \\ \hfill & \cup & \{a\mid \mathrm{if}{q}_1\stackrel{a}{\to }{q}_2\in \Delta \mathrm{or}(q_1\stackrel{\_}{\to }{q}_2\in \Delta \mathrm{and}\neg \exists q_2^{\prime }.q_1\stackrel{a}{\to }{q}_2^{\prime }\in \Delta )\}\hfill \\ \hfill & \cup & {\displaystyle \bigcup _{q_1\stackrel{q}{\mathbf{-}🠆}{q}_2\in \Delta }}{L}_q(A)\hfill \end{array}$$

This definition is mutually recursive with the definition of the tree languages $L_q(A)$ of all tree states $q\in Q_t$:

$$\begin{array}{ccc}\hfill L_q(A)& =& \{⟨h⟩\mid \stackrel{t}{\to }\overline{)q_1}\in \Delta ,h\in L_{q_1,q_2}(A),q_2-\to q\in \Delta \}\hfill \end{array}$$

The hedge language $L(A)$ that is recognized by the automaton is ${\displaystyle \bigcup _{q_1\in I,q_2\in F}}{L}_{q_1,q_2}(S)$. The rules of standard bottom-up tree automata have the form $a(q_1,\dots ,q_n)\to q$ where a is a symbol of arity n. With $\mathit{SHA}$s, this rule can be encoded by the sequence $\stackrel{t}{\to }\overline{)p_0}\stackrel{a}{\to }\overline{)p_1}\stackrel{q_1}{\mathbf{-}🠆}\dots \stackrel{q_n}{\mathbf{-}🠆}\overline{)p_n}-\to q$ where the states $q_1,\dots ,q_n,q$ are all tree states, and $p_0,\dots ,p_n$ new hedge states.

5.1. Determinization of $\mathit{SHA}$s

We formalize the notion of determinism for stepwise hedge automata and show how determinization works.

Definition 5.

A $\mathit{SHA}$ $(Q_h,Q_t,\Sigma ,\Delta ,I,F)$ is deterministic or equivalently a $\mathit{dSHA}$, if it satisfies the following conditions:

• I contains at most one element;
• ${⟨⟩}^{\Delta }$ contains at most one element;
• there is no epsilon transition, i.e., ${\epsilon }^{\Delta }=\varnothing$;
• $a^{\Delta },q^{\Delta }$, ${\_}^{\Delta }$ are partial functions from $Q_h$ to $Q_h$ for all $a\in \Sigma$ and $q\in Q_t$; and
• ${\mathtt{T}}^{\Delta }$ is a partial function from $Q_h$ to $Q_t$.

Proposition 2.

A $\mathit{SHA}$ of size n can be made deterministic in time $O(2^n)$ while preserving the hedge language.

Proof. 

The determinization procedure for $\mathit{SHA}$s combines the determinization algorithms of word and tree automata in the natural manner, while eliminating epsilon transitions. Let ${\epsilon }^{{\Delta }^{*}}$ be the reflexive and transitive closure of ${\epsilon }^{\Delta }$, and for any subset $Q\subseteq Q_h\cup Q_t$ let ${\epsilon }^{{\Delta }^{*}}(Q)={\displaystyle \bigcup _{q\in Q}}{\epsilon }^{{\Delta }^{*}}(q)$. Given a $\mathit{SHA}$ $A=(Q_h,Q_t,\Sigma ,\Delta ,I,F)$, we define an equivalent deterministic $\mathit{SHA}$ $det(A)=(Q_h^{\mathit{det}},Q_t^{\mathit{det}},\Sigma ,$ ${\Delta }^{\mathit{det}},I^{\mathit{det}},F^{\mathit{det}})$ such that $Q_h^{\mathit{det}}=2^{Q_h}$, $Q_t^{\mathit{det}}=2^{Q_t}$, $I^{\mathit{det}}=\{{\epsilon }^{{\Delta }^{*}}(I)\}$ and $F^{\mathit{det}}=\{Q^{\prime }\subseteq Q_h\mid Q^{\prime }\cap F\ne \varnothing \}$. There is a unique tree initial state in ${⟨⟩}^{{\Delta }^{\mathit{det}}}=\{{\epsilon }^{{\Delta }^{*}}({⟨⟩}^{\Delta })\}$ and no $\epsilon$-rule, that is, ${\epsilon }^{{\Delta }^{\mathit{det}}}=\varnothing$. The inference rules in Figure 14 define the missing part of ${\Delta }^{\mathit{det}}$.

Figure 14. Determinization of $\mathit{SHA}$s.

We can show for all $Q_1,Q_2\subseteq Q_h$ and $P\subseteq Q_t$ that

$$L_{Q_1,Q_2}(det(S))={\displaystyle \bigcup _{q_1\in Q_1,q_2\in Q_2}}{L}_{q_1,q_2}(S)$$

so that $L_P(det(S))={\displaystyle \bigcup _{q^{\prime }\in Q^{\prime }}}{L}_{q^{\prime }}(S)$. Therefore, $L(det(S))={\displaystyle \bigcup _{Q^{\prime }\in F^{\mathit{det}}}}{L}_{I,Q^{\prime }}(det(S))$ and thus $L(det(S))={\displaystyle \bigcup _{q_1\in I,q_2\in F}}{L}_{q_1,q_2}(S)=L(S)$. □

For illustration, the deterministic $\mathit{SHA}$ in Figure 15 is obtained by determinization of the $\mathit{SHA}$ in Figure 13.

Figure 15. The determinized $\mathit{SHA}$ $det(\mathrm{step}(c{h}^{*}(a+b)))$.

5.2. Compilation of $\mathit{NREs}$ to $\mathit{SHA}$s

As for $\mathit{NWAs}$, we introduce the notion of multi-module $\mathit{SHA}$s for which the sets of hedge states are partitioned between those that can evaluate top level positions and those to which nested positions are assigned. Therefore, multi-module $\mathit{SHA}$s will have exactly two modules too.

Definition 6.

A $\mathit{SHA}$ $A=(Q_h,Q_t,\Sigma ,\Delta ,I,F)$ is a multi-module $\mathit{SHA}$ if there is a subset of states $Q_h^0\subseteq Q_h$, that we call top level states, such that

• $I\subseteq Q_h^0$ and
• the states in $Q_h^0$ can reach only other states in $Q_h^0$ via Δ.

For instance, consider the multi-module $\mathit{SHA}$ in Figure 13. The states of module for the top level are $Q_0=\{1,2,3,4,5,6,12\}$. The others belong to the module for the nested level.

Any $\mathit{NRE}$ E can be compiled to a multi-module $\mathit{SHA}$ $\mathrm{step}(E)=(Q_h,Q_t,\Sigma ,\Delta ,I,F)$ such that $Q_t=\{E^{\prime }\mid E^{\prime }=⟨E^{″}⟩\mathrm{subexpression}\mathrm{of}E\}$ and $L_t(E^{\prime })=L(E^{\prime })$ for all tree states $E^{\prime }\in Q_t$. The $\mathit{SHA}$ $\mathrm{step}(E)$ can be partitioned into disjoint $\mathit{SHA}$s $\mathrm{step}(E)=A^{top}\cup {\displaystyle \bigcup _{E^{\prime }\in Q_t}}{A}^{E^{\prime }}$ such that $A^{top}=(Q_h^{top},Q_t,\Sigma ,{\Delta }^{top},I,F)$ and $A^{E^{\prime }}=(Q_h^{E^{\prime }},Q_t,\Sigma ,{\Delta }^{E^{\prime }},\varnothing ,\varnothing )$ for all $E^{\prime }\in Q_t$ and ${⟨⟩}^{{\Delta }^{top}}=\varnothing$. Note that the transitions relation $\Delta$ is decomposed thereby into independent connected components. The automaton $A^{top}$ can be identified with an Nfa with signature $\Sigma \cup Q_t$ given that it has no tree initial states. The automata $A^{E^{\prime }}$ are stepwise tree automata that recognize the tree language $L(E^{\prime })$ when taking $E^{\prime }$ as final state. For this, they may have tree initial states, but will not have any initial nor final states.

Theorem 2.

For any $\mathit{NRE}$ E, we can construct a $\mathit{SHA}$ A such that $L(A)=L(E)$. If E contains neither conjunctions nor negations, then the construction is in time $O(|E|)$.

Proof sketch.

For the case of expressions with conjunctions or negations, the construction is analogous to the way it is done for $\mathit{NWAs}$. We next sketch the construction of $\mathit{SHA}$s for expressions without conjunction and negation.

Case $E=E^{\prime }·E^{″}$:

We use McNaughton–Yamada–Thompson algorithm [36] for composing the multi-module Nfas of $\mathrm{step}(E^{\prime })$ and $\mathrm{step}(E^{″})$. The stepwise tree automata $A^{⟨E^{‴}⟩}$ of the subexpressions of type tree are preserved. For succinctness, if some subexpression $⟨E^{‴}⟩$ occurs more than once, then only a single copy of $A^{⟨E^{‴}⟩}$ is kept. References to states of the removed copy should be renamed to their equivalent counterparts.

Case $E=⟨E^{\prime }⟩$:

We construct $\mathrm{step}(E)$ from $\mathrm{step}(E^{\prime })$. The initial states of $\mathrm{step}(E^{\prime })$ are turned into tree initial states. We then add a new tree state $⟨E^{\prime }⟩$ and connect it to all final states of $\mathrm{step}(E^{\prime })$ by a tree final rule $q\to ⟨E^{\prime }⟩$. Furthermore, the previously final state q becomes non final. Finally we add a new initial state $q_i$, a new final state $q_f$ and a transition rule $\stackrel{h}{\to }\overline{)q_i}\stackrel{⟨E^{\prime }⟩}{\mathbf{-}🠆}\overline{)q_f}$.

Case $E=\mu a.E^{\prime }$:

The main idea of the construction is similar to the case of $\mathit{NWAs}$. The correctness argument relies on the invariant that only multi-module $\mathit{SHA}$s are built.

Again by the golden lemma of the $\mu$-calculus, we can assume w.l.o.g. that a occurs at most once in $E^{\prime }$. By using $\epsilon$-rules, we can preserve the invariant that there will be at most one pair $(q,q^{\prime })$ such that $q\stackrel{a}{\to }{q}^{\prime }$ in $\mathrm{step}(E^{\prime })$. Furthermore, these transitions cannot be on top level, given that the occurrence of a in $E^{\prime }$ must be nested below parentheses. The automaton $\mathrm{step}(E)$ is obtained from $\mathrm{step}(E^{\prime })$ by first copying the top level Nfa of $\mathrm{step}(E^{\prime })$, as in Figure 16. We thus obtain two versions for each state of the top level Nfa of $\mathrm{step}(E^{\prime })$: one referred to as the top level copy—$q_{0,0}$ and $q_{3,0}$ in Figure 16, and another one as the nested level – $q_{0,1}$ and $q_{3,1}$ in Figure 16. Only top level states may be initial or final. Then, we add $\epsilon$-rules from q to the nested states that correspond to the initial states of $\mathrm{step}(E^{\prime })$, and from the nested states corresponding to the final states of $\mathrm{step}(E^{\prime })$ to $q^{\prime }$. Finally we remove the rule $q\stackrel{a}{\to }{q}^{\prime }$. The resulting automaton is shown in Figure 17.

Figure 16. $\mathit{SHA}$ for $⟨a^{*}⟩$.

Figure 17. $\mathit{SHA}$ for $\mu a.⟨a^{*}⟩$.

Note that every transition added for a state–top level or nested—in a subsequent step of the construction—except the $\epsilon$-rules added for $\mu$-expressions—must also be added for its copy.

The construction is correct as the $\mu$-bound name a is nested below parenthesis in $E^{\prime }$. Therefore, it can be shown that the $\epsilon$-edges introduced cannot be used to produce unwanted order in successful runs. Maintaining this invariant in polynomial time requires an additional argument. Instead of copying the top level parts of subexpressions, each state is introduced twice during the construction: one version for nesting, and another one for being part of top level parts. This way the size of the automaton is not doubled at each step, but only once.

We omit the correctness proof of this construction. □

Unlike $\mathit{NWAs}$, one cannot preserve the determinism of the expressions of $\mathit{nregexp}(ch,T)$ in $\mathit{SHA}$s, even with Glushkov-like constructions. For instance, for the deterministic $\mathit{NRE}$ $⟨a_1·⟨a_2·\dots ·⟨a_n⟩\dots ⟩⟩$, one would have an $\mathit{SHA}$ having a tree initial state for each of the $⟨a_i\dots ⟩$ subtree, implying nondeterminism.

5.3. Experimental Results Starting with the $\mathit{SHA}$  Compiler

In the first two columns of Figure 18, we report the size of the $\mathit{SHA}$s obtained from $\mathit{NREs}$ by our compiler, and the size of the deterministic $\mathit{SHA}$s produced thereof.

Figure 18. The $\mathit{SHA}$s for the benchmark $\mathit{NREs}$ and the automata derived thereof.

The $\mathit{SHA}$ compiler yields automata of acceptable size from the $\mathit{NREs}$ of all benchmark queries. These sizes are given in the first column $step(.)$ of Figure 18. This even holds for A8, in contrast to the case where the produced $\mathit{SHA}$ has overall size 1106 and 267 states.

The determinization of the $\mathit{SHA}$s in the second column $det(step(.))$ even yields smaller automata in all cases. For A8, we obtain a deterministic stepwise automaton of overall size 749 and with 123 states. This might be surprising, in that the determinization algorithm may lead to an exponential blow-up in the worst case. However, it may also clean the automaton, keeping only accessible sets of states. This is what seems to happen systematically on the benchmark with the exception of A2, where the size goes up by a factor of four and A5 where the size doubles. For A2 the number of states grows by one third, while for A5 it decreases by one third.

Based on the back-and-forth compiler from $\mathit{SHA}$s to $\mathit{NWAs}$ from following Section 6, we can obtain deterministic $\mathit{NWAs}$ of acceptable size for all benchmark queries. The method $nwa(det(step(.)))$ yield for A a $\mathit{dNWA}$ of size 2831 and with 124 states. The alternative method $det(nwa(step(.)))$ yields a $\mathit{dNWA}$ of size 2520, which is even smaller, and the same number of states.

6. NWAs versus SHAs

We next show how to compile $\mathit{SHA}$s to $\mathit{NWAs}$ such that determinism is preserved, and back while introducing nondeterminism. Thereby, we can obtain small $\mathit{NWAs}$ for $\mathit{NREs}$ such as $E=c{h}^{*}(a+b)$ for which $det(nwa(E))$ blew up in size in a surprising manner (see Figure 12).

6.1. SHAs to NWAs

As a first step, we introduce a transformation on $\mathit{SHA}$s, so that for any $\mathit{SHA}$ A:

• if A is deterministic, the transformation returns A, and
• if A is nondeterministic with set of hedge states $Q_h$ and transition relation $\Delta$, the transformation returns a new $\mathit{SHA}$ $A^{\prime }$ with set of hedge states $Q_h^{\prime }=Q_h\uplus \{q_{t-\mathit{init}}\}$ where $q_{t-\mathit{init}}$ is a new hedge state, and set of transitions ${\Delta }^{\prime }$ which equals $\Delta$ except that ${⟨⟩}^{{\Delta }^{\prime }}=\{q_{t-\mathit{init}}\}$ and ${\epsilon }^{{\Delta }^{\prime }}={\epsilon }^{\Delta }\cup \{(q_{t-\mathit{init}},q)\mid q\in {⟨⟩}^{\Delta }\}$.

Then, we compile any $\mathit{SHA}$ $A=(Q_h,Q_t,\Sigma ,\Delta ,I,F)$ obtained after the above transformation to an $\mathit{NWA}$ $nwa(A)=(Q_h,Q_t,\Sigma ,\Gamma ,{\Delta }^{\prime },I,$ $F)$ such that $L(A)=L(nwa(A))$. We set $\Gamma =Q_h$, ${\_}^{{\Delta }^{\prime }}={\_}^{\Delta }$, $a^{{\Delta }^{\prime }}=a^{\Delta }$ for all $a\in \Sigma$, ${\epsilon }^{{\Delta }^{\prime }}={\epsilon }^{\Delta }$, ${\mathtt{T}}^{{\Delta }^{\prime }}={\mathtt{T}}^{\Delta }$:

$$\begin{array}{c}{q}_1\stackrel{\downarrow q_1}{\mathbf{-}🠆}\in \Delta p\in {⟨⟩}^{\Delta }\\ q_1\stackrel{\downarrow q_2}{\mathbf{-}🠆}p\in {\Delta }^{\prime }\mathrm{and}q\stackrel{\uparrow q_1}{\mathbf{-}🠆}{q}_2\in {\Delta }^{\prime }\end{array}$$

Clearly, if S is deterministic then so is $nwa(S)$, as p is unique in this case in particular. One might be tempted to skip the first-step transformation and restrict the above construction rule to states p such that $L_q(S[{⟨⟩}^{\Delta }/\{p\}])\ne \varnothing$. However, this would lead to a huge blow-up when determinizing these $\mathit{NWAs}$, basically as this change spoils the single-entry property discussed in Definition 7.

The conversion of $\mathrm{step}(c{h}^{*}(a+b))$ in Figure 13 yields the $\mathit{NWA}$ in Figure 19. Note that the opening rules are deterministic (but not the whole $\mathit{NWA}$), as for all tree states q there is at most one hedge state p with $⟨⟩\to p$ such that q is accessible from p. The NWA has size 64, while its determinization has size 159 (see Figure A3 of the Appendix C). The size increase raised by determinization is thus $95=159-64$ for this $\mathit{NWA}$.

Figure 19. The single-entry $\mathit{NWA}$ $nwa(\mathrm{step}(c{h}^{*}(a+b)))$ obtained from the $\mathit{SHA}$.

The size increase for determinization is considerably smaller for the $\mathit{NWA}$ obtained from the regular expressions by indirection via a $\mathit{SHA}$ than for $\mathit{NWAs}$ obtained by direct compilation. Indeed, the determinization of $\mathit{nwa}(c{h}^{*}(a+b))$ blows the size from 39 to 271. The size increase for the determinization of $\mathit{nwa}(c{h}^{*}(a+b))$ is thus $242=271-39$ while for $\mathit{nwa}(\mathrm{step}(c{h}^{*}(a+b)))$ it is only $95=159-64$.

The experiments will show that this is not an exception but the general rule. Intuitively, the reason is that $\mathit{NWAs}$ obtained from $\mathit{SHA}$s do all the work bottom-up, where $\mathit{NWAs}$ obtained directly from the regular expression do a considerable amount of work top-down. In terms of the work in [22], this restriction can be characterized syntactically by the single-entry property:

Definition 7.

A (weak) single-entry $\mathit{NWA}$ $A=(Q_h,Q_t,\Sigma ,\Gamma ,\Delta ,I,F)$ is a $\mathit{NWA}$ for which there exists a single state $q_{entry}\in Q_h$ such that all opening rules in Δ have the form $q\stackrel{\downarrow q}{\mathbf{-}🠆}{q}_{entry}$.

Note that call-driven automata (CDAs) discussed in [23] coincide with multi-module single-entry $\mathit{dNWAs}$ and also with (multi-module) single-entry visibly pushdown automata [22,24].

It can be shown that $\mathit{nwa}(S)$ has the (weak) single-entry property for all $\mathit{SHA}$s S for which the p’s are unique in the above construction rule, i.e., such that ${⟨⟩}^{\Delta }=\{p\}$. Note that this was not the case for $\mathrm{step}(c{h}^{*}(a+b))$ in Figure 13 but could have been imposed w.l.o.g. leading to a slightly different $\mathit{NWA}$ than in Figure 19.

The conversion of the determinization $det(\mathrm{step}(c{h}^{*}(a+b)))$ in Figure 15 yields the deterministic $\mathit{NWA}$ in Figure 20. The size goes up slightly from 53 to 73. It should be noticed that factorization avoids a quadratic blow up in this case. This can be observed at state 14, which has 3 incoming tree edges and 10 outgoing closing edges. Without factorization, the 3 tree edges could be replaced by 3 $\epsilon$-edges whose elimination would produce 30 closing edges. This would increase the number $3+10$ edges to $3\ast 10$ edges.

Figure 20. Deterministic $\mathit{NWA}$: $nwa(det(\mathrm{step}(c{h}^{*}(a+b))))$.

6.2. NWAs to SHAs

Conversely, $\mathit{NWAs}$ can be compiled to stepwise hedge automata, but at the cost of introducing nondeterminism, as an $\mathit{NWA}$ may traverse the branches of a tree top-down, while a stepwise must traverse them bottom-up. For this, the stepwise guesses the state in which the $\mathit{NWA}$ will arrive from above and then evaluates the subtree starting with this state, while verifying the correctness of the guess later on. Let $A=(Q_h,Q_t,\Sigma ,{\Delta }^{\prime },I,F)$ be an $\mathit{NWA}$. We build a $\mathit{SHA}$ $\mathrm{step}(A)=(Q_h^s,Q_t^s,\Sigma ,{\Delta }^s,I^s,F^s)$ where $Q_h^s=Q_h\times Q_h$, $Q_t^s=Q_h\times Q_t$, $I^s=\{(q,q)\mid q\in I\}$, $F^s=I\times F$ and ${\Delta }^s$ is the smallest satisfying the following rules:

The construction is such that $L(A)=L(\mathrm{step}(A))$.

For the $\mathit{NWA}$ $\mathit{nwa}(c{h}^{*}(a+b))$ in Figure 6, we obtain the stepwise in Figure A2 up-to removing useless states and separating the top level. Determinization yields $det(\mathrm{step}(nwa(c{h}^{*}(a+b))=det(\mathrm{step}(c{h}^{*}(a+b)))$ in Figure 15.

7. Optimizations

We will use three optimization methods for constructing smaller $\mathit{dSHA}$s and thus smaller $\mathit{dNWAs}$: minimization, symbolic representations of sets of transition rules, and schema-based cleaning.

7.1. Minimization

Our next objective is to reduce the size of deterministic $\mathit{SHA}$s by developing a minimization algorithm for a subclass of $\mathit{dSHA}$s. Even though our implementation can deal with them, we consider $\mathit{SHA}$s without symbolic rules $q\stackrel{-}{\to }{q}^{\prime }$ for simplicity in this section.

We start with an example that motivates the choice of our subclass. In Figure 21 and Figure 22, two $\mathit{dSHA}$s are given that both recognize the language of all hedges with signature $\Sigma =\{x,y\}$ containing exactly one occurrence of the letter x. The $\mathit{dSHA}$ in Figure 22 is the $\mathit{dSHA}$ recognizing this language which has the minimal number of states. The $\mathit{dSHA}$ in Figure 21 is the minimal multi-module $\mathit{dSHA}$ for this language. The question is how a minimization algorithm for $\mathit{dSHA}$s could convert the $\mathit{dSHA}$ in Figure 21 to this minimal one in Figure 22. In particular, why would it merge the tree initial state and the hedge initial state? We do not see how this could be done based on some Myhill–Nerode-like equivalence relation. This motivates an a priori restriction to $\mathit{dSHA}$s imposing that the tree initial state and hedge initial state must be equal.

Figure 21. A $\mathit{dSHA}$ for hedges over $\Sigma =\{x,y\}$ with single occurrence of x. It is minimal in the class of multi-module $\mathit{dSHA}$s.

Figure 22. An equivalent $\mathit{dSHA}$ to that in Figure 21 that is minimal in the class of $\mathit{dSHA}$s with equal tree and hedge initial states.

Note that any $\mathit{SHA}$ can be converted into a $\mathit{dSHA}$ with equal tree and hedge initial states. For this, it is sufficient to “fuse” these states and then to determinize the $\mathit{SHA}$ obtained. When doing so for the $\mathit{dSHA}$ in Figure 21, we indeed obtain the minimal $\mathit{dSHA}$ from Figure 23, so no further minimization is needed in this case.

Figure 23. A single x-marked position.

Given the close relationship between $\mathit{SHA}$s and weak single-entry $\mathit{NWAs}$, it is instructive to consider the existing results on minimization for $\mathit{dNWAs}$. It is known that the class of general $\mathit{dNWAs}$ does not allow for unique minimization [22] and that the minimization becomes NP-hard when admitting general signatures with multiple parenthesis [24].

On the positive side, the best existing minimization algorithm is due to Chervet and Walukiewicz [23]. It applies to the subclass of multi-module single-entry $\mathit{dNWAs}$, called there call-driven automata (CDA) (Chervet and Walukiewicz [23] permit signatures with multiple opening parenthesis. In the case of a single opening parenthesis, the class of CDAs is equal to their subclass of expanded CDAs for which they develop their minimization algorithm in the first place.). They showed that the subclass of multi-module single-entry $\mathit{dNWAs}$ enjoys unique minimization in polynomial time.

In the case of $\mathit{dSHA}$s, we believe that unique minimization holds for the following two subclasses, and will show it for the second:

• the subclass of multi-module $\mathit{dSHA}$s, and
• the subclass of $\mathit{dSHA}$s where the hedge and tree initial state are the same, i.e., ${⟨⟩}^{\Delta }=I$.

The first subclass of multi-module $\mathit{dSHA}$s is motivated by the subclass of multi-module single-entry $\mathit{dNWAs}$. Note, however, that the $\mathit{SHA}$s that are obtained by compilation from single-entry $\mathit{dNWAs}$ need not be deterministic, so the analogy between both automata classes is not perfect. The $\mathit{dSHA}$ in Figure 21 is minimal for the class of multi-module $\mathit{dSHA}$s.

The second subclass of $\mathit{dSHA}$s corresponds to the subclass of single-entry $\mathit{dNWAs}$ in which the single-entry state is equal to the initial state. The $\mathit{dSHA}$ in Figure 22 is minimal for the second subclass. In the remainder of this section, we present a minimization algorithm for the second subclass. For this, we identify $\mathit{dSHA}$s in which tree and hedge initial state coincide with two-sorted deterministic tree automata, so that we can use a minimization algorithm for the latter. Our automaton translation is based on a novel encoding of hedges into ranked well-sorted trees with monadic and binary function symbols, which is inspired by the previous binary encoding of unranked trees known from stepwise tree automata [21]. For any unranked signature $\Sigma$, as for the construction of hedges, we consider two sorts: $\mathtt{h}$ for hedges and $\mathtt{t}$ for trees. We then consider the following ranked signature with these two sorts:

$${\Sigma }_{@}=\{a^{(\mathtt{h})}\mid a\in \Sigma \}\cup \{{@}^{(\mathtt{h}\times \mathtt{t}\to \mathtt{h})},{\epsilon }^{(\mathtt{h})},T^{(\mathtt{h}\to \mathtt{t})}\}$$

The well-sorted trees over ${\Sigma }_{@}$ of both sorts then have the following abstract syntax:

$$\begin{array}{cccc}\text{well-sorted}\mathrm{trees}\mathrm{of}\mathrm{sort}\mathtt{h}\text{:}\hfill & \tau \hfill & ::=\hfill & a(\tau )\mid @(\tau ,{\tau }^{\prime })\mid \epsilon \\ \text{well-sorted}\mathrm{trees}\mathrm{of}\mathrm{sort}\mathtt{t}\text{:}\hfill & {\tau }^{\prime }\hfill & ::=\hfill & T(\tau )\end{array}$$

Any hedge over $\Sigma$ can be encoded into a ranked well-sorted tree of sort $\mathtt{h}$ with signature ${\Sigma }_{@}$. For instance, the hedge

$$h=⟨a·⟨b·⟨c·d·e⟩⟩·f⟩$$

is encoded into the following ranked well-sorted tree of sort $\mathtt{h}$ over ${\Sigma }_{@}$:

$$[[h]]=(f(a(\epsilon )@{\tau }^{\prime }))\mathrm{where}{\tau }^{\prime }=T(b(\epsilon )@{\tau }^{″})\mathrm{and}{\tau }^{″}=T(e(d(c(\epsilon ))))$$

Any $\mathit{SHA}$ $A=(Q_h,Q_t,\Sigma ,\Delta ,I,F)$ with equal tree and hedge initial states, that is, ${⟨⟩}^{\Delta }=I$, can then be encoded into a two sorted tree automaton $[[A]]=(Q_h,Q_t,{\Sigma }_{@},{\Delta }^{\prime },I,F)$ by mapping the transition rules in $\Delta$ to those in ${\Delta }^{\prime }$ as follows:

$$\begin{array}{cc}\Delta & {\Delta }^{\prime }\\ q\stackrel{a}{\to }{q}^{\prime }& a(q)\to q^{\prime }\\ q-\to p& T(q)\to p\\ q\stackrel{p}{\mathbf{-}🠆}{q}^{\prime }& q@p\to q^{\prime }\\ q_i\in {⟨⟩}^{\Delta }=I& \epsilon \to q_i\end{array}$$

We can first note that $[[L(A)]]=L([[A]])$. Second, the translation function is a bijection between $\mathit{SHA}$s over $\Sigma$ and two-sorted tree automata over ${\Sigma }_{@}$. Furthermore, this translation preserves determinism. It follows that if A is a $\mathit{dSHA}$ with a minimal number of states recognizing $L(A)$ then $[[A]]$ is a deterministic two-sorted tree automaton with a minimal number of states recognizing $[[L(A)]]$. Furthermore, the unique minimization of deterministic two-sorted tree automata implies the unique minimization of the class of $\mathit{dSHA}$s with equal tree and hedge initial states.

Using this translation back and forth, we can thus lift the minimization algorithm of deterministic two-sorted tree automata to a minimization algorithm for the subclass of $\mathit{dSHA}$s with equal tree and hedge initial states. This is the minimization algorithm for $\mathit{dSHA}$s that we have implemented. We then used it in our constructions to reduce the size of the $\mathit{dSHA}$s obtained by determinization.

7.2. Symbolic SHAs with Apply-Else Rules

The sizes of the $\mathit{dSHA}$s constructed so far are dominated by the number of transitions. We now propose a class of symbolic $\mathit{dSHA}$s by adding apply-else rules, in order to represent large numbers of apply rules in a more compact and symbolic manner.

An apply-else rule has the form $q\stackrel{\_}{\mathbf{-}🠆}{q}^{\prime }$ where $q,q^{\prime }\in Q_h$. It represents the set of apply rules $q\stackrel{p}{\mathbf{-}🠆}{q}^{\prime }$, where $p\in Q_t$ can be chosen arbitrarily from a subset of tree states distinguished by the automaton.

We have also adapted our determinization for $\mathit{dSHA}$s so that it preserves apply-else rules. What is missing so far is a concept for $\mathit{NWAs}$ that corresponds to the apply-else rules of $\mathit{dSHA}$s. Therefore, we have to eliminate apply-else rules before translating $\mathit{SHA}$s to $\mathit{NWAs}$.

7.3. Schema-Based Cleaning

Automata for XPath queries recognize nested words that can be obtained by encoding Xml documents with a single x-marked position. The class of such nested words is characterized by a schema that we can define as the intersection of the two $\mathit{dSHA}$s in Figure 23 and Figure 24. The first $\mathit{SHA}$ tests whether there is exactly one occurrence of the internal letter x, and the second one tests that the Xml data model is satisfied, and the node annotations with x and $\neg x$ are put at the right positions.

Figure 24. Nested words of x-marked XML documents.

The automata constructed for the XPath queries may accept some trees that do not satisfy the schema (but will never be evaluated on such trees when answering the query). The idea of schema-based cleaning is to remove all transition rules and states that are not used for recognizing any nested word satisfying the schema. Schema-based cleaning of an automaton can be performed by constructing the product of the automaton with the schema, which is in our case an intersection of two $\mathit{dSHA}$s. We then only keep those states of the original $\mathit{SHA}$ that are used in accessible and co-accessible states of the product with the schema.

Note that schema-based cleaning typically changes the language of the automaton. Different languages may be obtained when cleaning different automata for the same query with respect to the schema. If one is interested in a unique language, then one can choose the intersection of the automaton with the schema. This intersection, however, is usually larger than the automaton obtained by schema-based cleaning.

7.4. Experimental Results for Optimizations

The sizes of optimized automata for the benchmark queries are reported in Figure 25.

Figure 25. Optimized automata for derived stepwise automata compiled from $\mathit{NREs}$.

The function $ste{p}_{@}\_$ used in the first column compiles $\mathit{NREs}$ to $\mathit{SHA}$s with apply-else rules. This does not change the number of states, but reduces the number of automata transitions. In the case of A8, the size of the stepwise automaton is reduced from 1106 to 894.

An optimized determinizer is applied by the function $D(.)=det(ste{p}_{@}\_(.))$ in the second column. It preserves apply-else rules in particular. For A8, the size is reduced from 749 to 639 while the number of states is preserved.

Schema-based cleaning is applied by the function $S(.)=schema-clean(D(.))$ in the third column. For A8, the number of rules is reduced further from 639 to 527.

Minimization is applied by the function $M(.)=mini(S(.))$ in the fourth column. In the case of A8, it reduces the number of states from 117 to 101 and the size from 527 to 487.

In order to come back to $\mathit{dNWAs}$, we have to eliminate the apply-else rules in column six. For A8 this increases the number of rules back from 527 to 1257.

In the final column, we apply the compiler from $\mathit{SHA}$s to $\mathit{NWAs}$ which preserves determinism. For $A8$, this results in a $\mathit{dNWA}$ of size 1413 and 102 states. This is better than the previous results, in particular with respect to the number of states.

8. Summary of Experimental Results

We now plug the different compilers and optimization methods all together and compare the sizes of deterministic $\mathit{NWAs}$ that we can obtain thereby.

The overall sizes (#states) of the resulting $\mathit{dNWAs}$ are given in Figure 26. We see that the two methods starting with $\mathit{SHA}$s $nwa(det(step(.)))$ and $det(nwa(step(.)))$ yield reasonably small deterministic $\mathit{NWAs}$ for the $\mathit{NREs}$ of all benchmark queries. The methods starting with $\mathit{NWAs}$ $nwa(det(step(nwa(.))))$ and $det(nwa(step(nwa(.))))$ provide reasonably small deterministic $\mathit{NWAs}$ for queries except for A8.

Figure 26. Deterministic $\mathit{NWAs}$ computed with optimizations for the XPath benchmark queries. Note that different $\mathit{dNWAs}$ for the same query may recognize different languages, due to schema-based cleaning with respect to the Xml data model. Furthermore, our implementation of the minimization algorithm for the subclass of $\mathit{dSHA}$s worked successfully only for $\mathit{dSHA}$s with at most 200 states.

We also tested our algorithms on collections of XPath queries with a scalable parameter, such as the queries $c{h}^n(a)$ for increasing n. This series is known to require automaton with a number of states exponential in n for deterministic bottom-up evaluation. The best methods to produce deterministic $\mathit{NWAs}$ in this case is $\mathit{nwa}(\mathit{det}(\mathrm{step}))$. It works until $n=9$, leading to an $\mathit{dNWA}$ of size $134,929$ with 772 states. The number of states close to doubles when increasing n by 1. The second best method for producing $\mathit{dNWAs}$ for the series $c{h}^n(a)$ works only until $n=6$.

For explaining the different size of the $\mathit{dNWAs}$ for the series $c{h}^n(a)$, we first note that no schema-based cleaning was applied in this experiment. As a consequence, unique minimal single-entry $\mathit{dNWAs}$ in which the single-entry state is the initial state should exist. The reason for the larger number of states with the three other methods is that we have not implemented any minimization algorithm for this subclass of single-entry $\mathit{dNWAs}$. Furthermore, our implementation of the minimization algorithm for our subclass of $\mathit{dSHA}$s failed for too big $\mathit{dSHA}$s. In this case, the number of states reported in Figure 27 could not be reduced to the minimum. In addition, the number of rules seems to be increased further by the lack of any symbolic representation for rules of $\mathit{NWAs}$ that could mimic the apply-else rules for $\mathit{SHA}$s.

Figure 27. Deterministic $\mathit{NWAs}$ for the queries $c{h}^n(a)$ where $n=0,\dots ,9$: size (#states). There is no schema-based cleaning. Our implementation of the minimization algorithm was applied to all $\mathit{dSHA}$ with at most 200 states, as it failed for larger $\mathit{dSHA}$s. No minimization algorithm for subclasses of single-entry $\mathit{dNWAs}$ was implemented.

9. Deterministic Nested Regular Expressions

We finally show how to distinguish $\mathit{NREs}$ that can be evaluated deterministically in polynomial time, for instance, by compilation to deterministic $\mathit{NWAs}$. For this, we consider the language of $\mathit{NREs}$ $\mathit{nregexp}(ch,T)$ that extends the abstract syntax of $\mathit{NREs}$ by a new constant T and a new unary operator $ch$.

Definition 8.

An expression of $\mathit{nregexp}(ch,T)$ is deterministic if it does not contain a subexpression of any of the forms: $E_1+E_2$, $E^{*}$, $T·E$, $\mu a.E$.

Note in particular that $ch(a)$ is a deterministic expression of $\mathit{nregexp}(ch,T)$, as the child operator is added as a primitive there. In contrast, the semantically equivalent expression $T.⟨a⟩.T$ is not deterministic. Similarly, T is deterministic since it is a primitive expression of $\mathit{nregexp}(ch,T)$, while the equivalent expression $\mu x.{(⟨x⟩+\_)}^{*}$ is nondeterministic for three different reasons: the $\mu$-operator, star *, and disjunction +. The recursive expression $c{h}^{*}(E)$ is nondeterministic: it is not primitive in $\mathit{nregexp}(ch,T)$, and its definition is based on the $\mu$-operator and disjunction.

The only query of the benchmark for which we can provide a deterministic $\mathit{NRE}$ is the query A1. The $\mathit{NRE}$ for query A1 in Figure A1 is nondeterministic nevertheless, as we replaced $ch(E)$ with $T·⟨E⟩·T$. This is not problematic, given that we can use a decent method for determinization of $\mathit{NWAs}$. For this reason, it does no more seem worth the effort to maintain specialized compilation methods for deterministic $\mathit{NREs}$. For the same reason, we will not present any experimental results for our specialized compiler from deterministic $\mathit{NREs}$ to deterministic $\mathit{NWAs}$. Instead we use the more general compiler for general nondeterministic $\mathit{NREs}$.

The compiler from Theorem 1 introduces epsilon rules, and thus it does not preserve determinism: some deterministic $\mathit{NREs}$ will be compiled to nondeterministic $\mathit{NWAs}$. This introduction of nondeterminism can be avoided by eliminating epsilon rules on the fly, that is by using Glushkov’s approach rather than that of Thompson.

Theorem 3.

For any deterministic regular expression E of $\mathit{nregexp}(ch,T)$ without conjunction and negation, we can construct in time $O(|E{|}^2)$ a $\mathit{dNWA}$ recognizing the same language.

Proof sketch.

Theorem 3 uses Glushkov’s construction and thus eliminates $\epsilon$-edges on the fly compared to the McNaughton–Yamada–Thompson algorithm. The Glushkov construction is well-known to preserve determinism when compiling regular expressions without nesting to finite state automata [20]. For the additional deterministic expressions $ch(E)$, we adapt the deterministic compilation from the work in [18]. This quadratic time result generalizes a previous result for the Glushkov construction [19] from regular expressions without conjunctions and negations to $\mathit{NREs}$ without conjunctions and negations. □

Small deterministic $\mathit{NREs}$ without conjunction and negation can thus be compiled to small $\mathit{dNWAs}$. On the benchmark, however, this construction can be applied to the query $A1$ only, so only a few queries can be covered in this manner.

10. Conclusions and Future Work

We presented $\mathit{SHA}$s and showed how they can be used to compile $\mathit{NREs}$ to deterministic $\mathit{NWAs}$. When applied to $\mathit{NREs}$ for navigational XPath queries in the usual XPathMark benchmark, we obtained reasonably small deterministic $\mathit{NWAs}$, in contrast to all previous approaches.

The $\mathit{dNWAs}$ that we obtain by compilation from $\mathit{SHA}$s all have the weak single-entry property. This property means that the computation of the $\mathit{NWA}$ is done in a purely bottom-up and left-to-right manner, so in the same way as by an $\mathit{SHA}$. Our experiments show that the usual determinization algorithm for $\mathit{NWAs}$ is well-behaved when applied to weak single-entry $\mathit{NWAs}$, while it quickly fails without the weak single-entry property.

We have also stated a unique minimization algorithm for $\mathit{dSHA}$s with the same tree and hedge initial state. It is open whether unique minimization holds for general $\mathit{dSHA}$s. Neither do we know whether $\mathit{dSHA}$ minimization is NP-hard. The analogous questions remain open for the class of weak single-entry $\mathit{dNWAs}$.

In future work, one needs to tackle the open questions on the minimization of $\mathit{dSHA}$s, weak single-entry $\mathit{dNWAs}$, and $\mathit{dNWAs}$ with fixed general signatures. One has to understand, whether and why unique minimization holds or not, and whether and why minimization is hard or not. Independently, it is interesting to use $\mathit{SHA}$s in various questions in theory and practice. In particular, we want to develop new algorithms for earliest query answering for $\mathit{dSHA}$s that are more efficient than the existing algorithms for $\mathit{dNWAs}$ [16] and to see how they behave in practice.