Weighted Consensus Segmentations

Abstract

The problem of segmenting linearly ordered data is frequently encountered in time-series analysis, computational biology, and natural language processing. Segmentations obtained independently from replicate data sets or from the same data with different methods or parameter settings pose the problem of computing an aggregate or consensus segmentation. This Segmentation Aggregation problem amounts to finding a segmentation that minimizes the sum of distances to the input segmentations. It is again a segmentation problem and can be solved by dynamic programming. The aim of this contribution is (1) to gain a better mathematical understanding of the Segmentation Aggregation problem and its solutions and (2) to demonstrate that consensus segmentations have useful applications. Extending previously known results we show that for a large class of distance functions only breakpoints present in at least one input segmentation appear in the consensus segmentation. Furthermore, we derive a bound on the size of consensus segments. As show-case applications, we investigate a yeast transcriptome and show that consensus segments provide a robust means of identifying transcriptomic units. This approach is particularly suited for dense transcriptomes with polycistronic transcripts, operons, or a lack of separation between transcripts. As a second application, we demonstrate that consensus segmentations can be used to robustly identify growth regimes from sets of replicate growth curves.

1. Introduction

One-dimensional segmentation problems naturally appear in time series analysis across diverse application areas (often referred to as change point or jump point detection in this context). In computational biology, 1D-segmentation problems arise in the analysis of micro-array and high-throughput DNA or RNA sequencing data. Copy number variations in genomes, for example, can be detected by segmenting array-CGH or DNA coverage data into piece-wise constant segments; see Reference [1] for a review. In epigenomics, recurrent patterns of histone modifications define genomic intervals that can be associated with functional units including promoters, enhancers, or gene bodies, see, e.g., Reference [2] and the references therein. The identification of transcriptional units is also a segmentation problem, consisting in the distinction of expressed and non-expressed loci [3,4] or operons [5] or, more generally, in the distinction of adjacent or even overlapping transcripts without the benefit of non-expressed spacers between them. The latter task is particularly relevant in organisms with “compact” genomes, such as bacteria [5] or yeast [6,7], where transcribed loci are rarely separated non-expressed regions. Boundaries between transcriptional units are detectable by differences in RNA levels [6], see, e.g., the SRG1 ncRNA in Figure 1. A plethora of segmentation algorithms for genomic features, as well as time series data, have become available, reviewed and benchmarked, e.g., in References [8,9,10].

The boundaries between segments typically are not clearly visible in individual data tracks. Such limitations can often be alleviated by aggregating multiple experiments or measurements. In the yeast RNA-seq data shown in Figure 1, for instance, transcriptome samples at different time points of the respiratory cycle are aggregated to produce a more informative signal than just the RNA expression level at a single time point. Still, any particular choice of parameters (here the choice of the similarity measure for the temporal coverage profiles at adjacent nucleotides), produce both false positive and false negative segment boundaries.

Figure 1 suggests that an improvement could be achieved by aggregating the different segmentations to a single consensus. In a similar vein, Reference [4] employed a simple heuristic segmentation to detect candidate loci as intervals that are scored by a statistical model for each RNA-seq experiment, followed by a problem-specific greedy heuristic to determine consensus interval boundaries for expressed ncRNA loci. The segmentation method for bacterial RNA-seq data in Reference [5] computes optimal segmentations with different numbers K of segments and uses a voting procedure to obtain a consensus over different values of K. These examples beg the question whether there is a more principled way to aggregated segmentations in a single consensus segmentation.

This question also appears in a much more general context. Modern *omics studies often report their results in the form of genome browser tracks, i.e., as segmentation of the reference genome into intervals. The comparison and consolidation of such data naturally asks for a consensus or reference. This is particularly the case for annotation based on epigenomic or transcriptomic data. Here, principled, efficient methods, to compare annotations beyond quantifying overlaps would be highly desirable to avoid a complete reanalysis of the underlying raw data. In contrast to genome browser tracks, raw data typically require extensive processing and are by no means straightforward to access in all cases. Despite the obvious potential usefulness of consensus segmentations, the literature on systematic comparisons of segmentations is surprisingly sparse. Two natural ways to approach the problem have been considered:

(i)

Focusing on the breakpoints between segments, one can treat them as signal. Significant (consensus) breakpoints are then detectable as unexpected accumulations across multiple data sets by the C-KS algorithm [12]. Somewhat more generally, this can be seen as clustering problem for breakpoints [13].

(ii)

Segmentations of linearly ordered data form partitions of an interval. (Dis)similarity measures for partitions, such as the Rand [14], Fowlkes-Mallows [15], Jaccard [16], and Hubert-Arabie [17] indices, or the Mirkin [18], and Van Dongen [19] metrics, are also applicable to the special case of segmentations. The Median Partition problem, also known as Consensus Clustering Problem, consists in finding a partition that is as close as possible to a given collection w.r.t. to one of these (dis)similarity measures [20,21]. Median Partition is NP-complete [22,23]. It is this second approach that we pursue further in this contribution.

The Segmentation Aggregation problem [24] is the specialization to partitions of a linearly ordered set, such as a time series or genomic sequence: Given a set of m segmentations ${\mathbf{S}}_1$, ${\mathbf{S}}_2$, ..., ${\mathbf{S}}_m$ on an interval, and a distance function D between segmentations, the task is to find a segmentation $\mathbf{C}$ that minimizes the distance sum

$$f\left(\mathbf{C}\right):=\sum _{q=1}^{m}D(\mathbf{C},{\mathbf{S}}_q).$$

(1)

We assume that $D(.,.)$ is a dissimilarity, i.e., that (i) $D(\mathbf{S},{\mathbf{S}}^{\prime })\ge 0$, and (ii) $D(\mathbf{S},{\mathbf{S}}^{\prime })=0$ if $\mathbf{S}={\mathbf{S}}^{\prime }$. In most cases, $D(.,.)$ will be a metric. However, neither symmetry nor the triangle inequality are necessary. It is shown in Reference [24] that Segmentation Aggregation, in contrast to Median Partition in the general case, can be solved exactly by dynamic programming for several interesting distance measures, including the disagreement or Mirkin metric and the information distance. Mailă [25] showed, using an axiomatic approach based on certain additivity conditions in the lattice of partitions, that the “variation of information” [26], i.e., the information distance [24] serves as the essentially unique natural distance between partitions. Nevertheless we consider here a much broader class of dissimilarity measures. Despite its appealing features, namely the almost complete absence of model assumptions and the fact that no detailed knowledge on the provenance of the input segmentations is required, Segmentation Aggregation has rarely been used in practical data analysis. Here, we demonstrate that it can be a useful and efficient approach.

A given set of input data is frequently subject to biases, such as an uneven phylogenetic distribution of taxa in comparative genomics or unbalanced distributions of samples between treatment groups. For the purpose of consensus formation, it is usually desirable to retain all data. As a remedy for sampling biases, a plethora of weighting schemes have been proposed to correct for biases by giving larger weights to underrepresented and smaller weights to over-represented data; see Reference [27] for a comparison of different approaches. In the context of segmentations of genomic features or time series, the confidence in individual segmentations ${\mathbf{S}}_q$ may be different, e.g., due to different noise levels in individual data tracks. It may also be desirable to treat biological replicates different from technical replicates. Naturally, such differences can be expressed by introducing segmentation-specific weights $w_q$. We shall see below that such weights can be introduced in Segmentation Aggregation in a straightforward manner.

In this contribution, we investigate the weighted version of the Segmentation Aggregation problem with the aim of getting insights into the properties of consensus segmentations. In particular, we generalize previous results of the positioning of consensus break points and we derive an upper bound on length of consensus segments for a large class of distance functions. We then consider two very different applications of consensus segmentations: the identification of transcriptional units, using yeast transcriptomes as a show-case example, and the segmentation of microbial growth curves. We close with a brief discussion of several open problems both regarding the theory behind consensus segmentations and their practical applications.

2. Theory

2.1. Dynamic Programming Algorithm

The Weighted Segmentation Aggregation problem is a moderate generalization of the unweighted version considered in Reference [24]. Given a set $\left\{{\mathbf{S}}_q\right|1\le q\le m\}$ of input segmentations and corresponding weights $w_q>0$ that quantify the relative importance of the contributing segmentations ${\mathsf{S}}_q$, the task is to minimize the objective function

$$f\left(\mathbf{C}\right):=\sum _{q=1}^m{w}_{q}D(\mathbf{C},{\mathbf{S}}_q),$$

(2)

i.e., the weighted total dissimilarity of the unknown consensus segmentation $\mathbf{C}$. Without loosing generality we may assume that ${\sum }_{q=1}^m{w}_q=1$.

Following Reference [24], we consider a distance measure D that can be expressed in terms of the common refinement ${\mathbf{S}}^{\prime }\wedge {\mathbf{S}}^{″}:=\{A\cap B|A\in {\mathbf{S}}^{\prime },B\in {\mathbf{S}}^{″}\}$ of two segmentations. ${\mathbf{S}}^{\prime }\wedge {\mathbf{S}}^{″}$ consists of all intersections of the segments of ${\mathbf{S}}^{\prime }$ and ${\mathbf{S}}^{″}$. The common refinement is also known as the union segmentation since its set of segment boundaries is exactly the union of the boundaries of ${\mathbf{S}}^{\prime }$ and ${\mathbf{S}}^{″}$. In particular, therefore, $\mathbf{S}\wedge \mathbf{S}=\mathbf{S}$. Now, define the potential of a segmentation $\mathbf{S}$ as

$$E\left(\mathbf{S}\right)=\sum _{A\in \mathbf{S}}\mathfrak{e}\left(A\right),$$

(3)

where $\mathfrak{e}$ is a potential function evaluating the individual segments. This gives rise to a class of distances between segmentations defined by

$$D({\mathbf{S}}^{\prime },{\mathbf{S}}^{″})=E\left({\mathbf{S}}^{\prime }\right)+E\left({\mathbf{S}}^{″}\right)-2E({\mathbf{S}}^{\prime }\wedge {\mathbf{S}}^{″}).$$

(4)

Substituting for D in Equation (2) yields

$$f\left(\mathbf{C}\right)=\sum _{q=1}^m{w}_{q}E\left(\mathbf{C}\right)+\sum _{q=1}^m{w}_{q}E\left({\mathbf{S}}_{\mathbf{q}}\right)-2\sum _{q=1}^m{w}_{q}E(\mathbf{C}\wedge {\mathbf{S}}_{\mathbf{q}}),$$

(5)

where the middle term depends only on the input. It is, therefore, a constant that can be dropped for the purpose of optimization. Making use of the fact that the weights are normalized, we obtain the objective function

$$\tilde{f}\left(\mathbf{C}\right):=f\left(\mathbf{C}\right)-\sum _{q=1}^m{w}_{q}E\left({\mathbf{S}}_{\mathbf{q}}\right)=E\left(\mathbf{C}\right)-2\sum _{q=1}^m{w}_{q}E(\mathbf{C}\wedge {\mathbf{S}}_{\mathbf{q}}).$$

(6)

Now, we explicitly consider $\mathbf{C}$ as a sequence of intervals A. Using the additivity of the potential E, we obtain

$$\tilde{f}\left(\mathbf{C}\right)=\sum _{A\in \mathbf{C}}\left(\mathfrak{e}\left(A\right)-2\sum _{q=1}^m{w}_q\sum _{\genfrac{}{}{0pt}{}{B\in {\mathbf{S}}_q}{B\cap A\ne \varnothing }}\mathfrak{e}(A\cap B)\right)=:\sum _{A\in \mathbf{C}}{\Delta }\left(A\right).$$

(7)

The additive form of Equation (7) as a sum of the contributions ${\Delta }\left(A\right)$ for the consensus segments $\mathbf{C}$ makes it possible to minimize $\tilde{f}\left(\mathbf{C}\right)$ by dynamic programming [24]. To this end, consider the subset ${\mathcal{C}}_{|k}$ of segmentations that have a segment boundary at k, i.e. , position k is the endpoint of a segment. For a given segmentation $\mathbf{C}\in {\mathcal{C}}_{|k}$, denote by $\tilde{f}\left(\mathbf{C}\right|k)$ the sum of the contributions ${\Delta }\left(A\right)$ with $maxA\le k$. Write $F_k:={min}_{\mathbf{C}\in {\mathcal{C}}_{|k}}\tilde{f}\left(\mathbf{C}\right|k)$ for the minimal value of $\tilde{f}\left(\mathbf{C}\right|k)$. Since k is a segment boundary, the last segment A before k is necessarily of the form $[j+1,k]$, where $j<k$ denotes the segment boundary immediately preceding k. Using this notation, we can compute

$$\begin{array}{ccc}\hfill F_k& =& \underset{\mathbf{C}\in {\mathcal{C}}_{|k}}{min}\tilde{f}\left(\mathbf{C}\right|k)=\underset{j<k}{min}\underset{\mathbf{C}\in {\mathcal{C}}_{|j}}{min}\left({\Delta }\left([j+1,k]\right)+\tilde{f}\left(\mathbf{C}\right|j)\right)\hfill \\ & =& \underset{j<k}{min}\left({\Delta }\left([j+1,k]\right)+\underset{\mathbf{C}\in {\mathcal{C}}_{|j}}{min}\tilde{f}\left(\mathbf{C}\right|j)\right)=\underset{j<k}{min}\left({\Delta }\left([j+1,k]\right)+F_j\right).\hfill \end{array}$$

(8)

Thus, we obtain a simple dynamic programming recursion that has the same form for the weighted and unweighted consensus segmentation; also see [24]. The weights appear only in the scoring function ${\Delta }$. We note, furthermore, that the recursion (8) is the same as for segmentation problems in general [28]. It appears, e.g., in Reference [29] for financial time series, in Reference [30] in context of text segmentation, in Reference [31] for the analysis of array CGH data, and in References [5,7,32] for the identification of transcripts in tiling array and RNA-seq data. It is discussed in the setting of very general similarity measures in Reference [11]. As we shall see below, the effort to compute $F_k$ is dominated by the effort to compute the score ${\Delta }[i,j]$.

Before we proceed, we briefly consider a general condition on the form of the potential function $\mathfrak{e}(.)$. Denote by $\mathbf{D}$ the discrete segmentation in which every interval is a single point and by $\mathbf{J}$ the indiscrete segmentation consisting of a single interval. A function $\mathfrak{e}$ is subadditive if $\mathfrak{e}\left(A\right)\le \mathfrak{e}\left(A_1\right)+\mathfrak{e}\left(A_2\right)$ for every A and every subdivision $A_1\dot{\cup }{A}_2=A$ of A. This inequality is strict for at least one interval if and only if $\mathfrak{e}\left([1,n]\right)<{\sum }_{i=1}^n\mathfrak{e}\left([i,i]\right)$. Comparing $\mathbf{D}$ and $\mathbf{J}$, we observe that, in this case, $D(\mathbf{D},\mathbf{J})=\mathfrak{e}\left([1,n]\right)-{\sum }_{i=1}^n[i,i]<0$, violating that D is a proper distance function. For the limiting case of an additive potential, $\mathfrak{e}\left(A\right)=\mathfrak{e}\left(A_1\right)+\mathfrak{e}\left(A_2\right)$ for all intervals and their subdivisions, we obtain $D(\mathbf{S},{\mathbf{S}}^{\prime })=0$ for any two segmentations $\mathbf{S}$ and ${\mathbf{S}}^{\prime }$. Thus, only potentials that satisfy $\mathfrak{e}\left(A\right)>\mathfrak{e}\left(A_1\right)+\mathfrak{e}\left(A_2\right)$ for at least some $A_1\dot{\cup }{A}_2=A$ are of interest. A function is superadditive if $\mathfrak{e}\left(A\right)\ge \mathfrak{e}\left(A_1\right)+\mathfrak{e}\left(A_2\right)$ for all $A_1\dot{\cup }{A}_2=A$. One easily checks that D is a metric whenever $\mathfrak{e}$ is superadditive. This condition is not necessary, however. For example, the negentropy defined in Equation (17) below, is not superadditive.

2.2. Efficient Computation of the Segment Scores ${\Delta }[i,j]$

The direct evaluation of ${\Delta }\left(\right[i,k\left]\right)$ according to its definition, Equation (8), for given i and k, requires $O\left(nm\right)$ operations because this entails the summation over $O\left(n\right)$ segments for each of the m input segmentations. This results in an impractical total effort of $O\left(n^{3}m\right)$ compared to the quadratic cost of the dynamic programming recursion itself. It is of considerable practical interest, therefore, to find a more efficient way of computing the scoring function. The key idea is to consider, for a given position i, two slightly different partial sums:    

$${\delta }_{<}\left(i\right)\sum _{q=1}^m{w}_q\sum _{\genfrac{}{}{0pt}{}{B\in {\mathbf{S}}_q}{maxB\le i}}\mathfrak{e}\left(B\right)\mathrm{and}{\delta }_{\le }\left(i\right)\sum _{q=1}^m{w}_q\sum _{\genfrac{}{}{0pt}{}{B\in {\mathbf{S}}_q}{minB\le i}}\mathfrak{e}\left(B\right).$$

(9)

For a given boundary i in $\mathbf{C}$, the first term sums all intervals in ${\mathbf{S}}_q$ that do not extend beyond i, while the second sum also includes those that begin before or at i and extend beyond i. Thus, ${\delta }_{<}\left(k\right)-{\delta }_{\le }\left(j\right)$ captures all segments of the ${\mathbf{S}}_q$ that are contained within $[j+1,k]$ with one important exception: Segments, such as B in Figure 2, that contain $[j+1,k]$ contribute to ${\delta }_{\le }\left(j\right)$ but not to ${\delta }_{<}\left(k\right)$. Such overlapping segments will be taken care of in a correction term discussed below. Using the notation $B_{\le i}:=\{b\in B|b\le i\}$ and $B_{\ge i}:=\{b\in B|b\ge i\}$ we, furthermore, define terms

$${\delta }_{<}^{\cap }\left(i\right):=\sum _{q=1}^m{w}_q\sum _{\genfrac{}{}{0pt}{}{B\in {\mathbf{S}}_q}{i\in B,i\ne maxB}}\mathfrak{e}\left(B_{\le i}\right)\mathrm{and}{\delta }_{>}^{\cap }\left(i\right):=\sum _{q=1}^m{w}_q\sum _{\genfrac{}{}{0pt}{}{B\in {\mathbf{S}}_q}{i\in B,i\ne minB}}\mathfrak{e}\left(B_{\ge i}\right),$$

(10)

where each sum contains at most a single term, namely the interval $B\in {\mathbf{S}}_q$ that extends across i. Note that intervals that begin or end in position i do not contribute to ${\delta }_{>}^{\cap }\left(i\right)$ or ${\delta }_{<}^{\cap }\left(i\right)$, respectively. The correction terms correspond to the parts of segments that are non-trivially intersected by $[j+1,k]$, shown in magenta and cyan, resp., in Figure 2. Thus ${\delta }_{<}\left(k\right)-{\delta }_{\le }\left(j\right)+{\delta }_{<}^{\cap }\left(k\right)+{\delta }_{>}^{\cap }(j+1)$ covers exactly all the intervals contributing to $[j+1,k]$ – with the exception of segments $B\in {\mathcal{S}}_q$ that begin at $minB<j+1$ and end at $maxB>k$ as mentioned above. For such segments, instead of the contributions for $B_{\le k}$ and $B_{\ge j+1}$, a single contribution for the interval $[j+1,k]\cap B=[j+1,k]$ has to be used. In addition, the contribution for B that is erroneously subtracted with ${\delta }_{\le }\left(j\right)$, needs to be restored. Collecting these contributions, we obtain the following correction term for intervals that span across the interval $[i^{\prime },i^{″}]$ of interest:

$${\delta }^{*}(i^{\prime },i^{″})\sum _{q=1}^m{w}_q\sum _{\genfrac{}{}{0pt}{}{\genfrac{}{}{0pt}{}{B\in {\mathbf{S}}_q}{i^{\prime },i^{″}\in B}}{minB<i^{\prime }\le i^{″}<maxB}}\left(\mathfrak{e}\left(B\right)+\mathfrak{e}\left([i^{\prime },i^{″}]\right)-\mathfrak{e}\left(B_{\le i^{″}}\right)-\mathfrak{e}\left(B_{\ge i^{\prime }}\right)\right).$$

(11)

For the interval $[j+1,k]$, the correction term ${\delta }^{*}(j+1,k-1)$ defined in Equation (11) can be understood as follows: the first term accounts for the correct contribution of $B\cap [j+1,k]=[j+1,k]$, the second term compensates for the error introduced by ${\delta }_{\le }\left(j\right)$, and the remaining two terms remove the superfluous contributions introduced by ${\delta }_{<}^{\cap }\left(k\right)$ and ${\delta }_{>}^{\cap }(j+1)$. We summarize this derivation in the following form:

Theorem 1.

The potential-dependent segment scores defined in Equation (7) can be expressed as

$${\Delta }\left([j+1,k]\right)=\mathfrak{e}\left([j+1,k]\right)-2\left({\delta }_{<}\left(k\right)-{\delta }_{\le }\left(j\right)+{\delta }_{<}^{\cap }\left(k\right)+{\delta }_{>}^{\cap }(j+1)+{\delta }^{*}(j+1,k)\right).$$

(12)

The only term that depends on both $j+1$ and k is the correction of long input intervals ${\delta }^{*}(j+1,k)$. The restricted sum over the $B\in {\mathbf{S}}_q$ in Equation (11) contains at most one segment for each input segmentation and, thus, can be evaluated in $O\left(m\right)$ time for a given interval. Furthermore, the sum is certainly empty whenever $[j+1,k]$ is larger than the largest segment in any of the ${\mathbf{S}}_q$; this can be used to speed up the evaluation from $O\left(m\right)$ to $O\left(1\right)$ if the segment lengths in the input are bounded by a constant, except for the short intervals.

Lemma 1.

The arrays of correction terms ${\delta }_{<}$, ${\delta }_{\le }$, ${\delta }_{<}^{\cap }$, and ${\delta }_{>}^{\cap }$ can be computed in $O\left(nm\right)$ total time.

Proof. 

The values of ${\delta }_{<}\left(i\right)$ and ${\delta }_{\le }\left(i\right)$ can be computed iteratively: we obtain ${\delta }_{<}\left(i\right)$ by adding the contribution $w_q\mathfrak{e}\left(B_q\right)$ to ${\delta }_{<}(i-1)$ whenever $i=max{B}_q$ for the segmentation ${\mathbf{S}}_{\mathbf{q}}$. Similarly, ${\delta }_{\le }\left(i\right)$ is obtained by adding $w_q\mathfrak{e}\left(B_q\right)$ to ${\delta }_{\le }(i-1)$ if $i=min{B}_q$. For each i, therefore, we require $O\left(m\right)$ operations. The sums in Equation (10) comprise at most one segment of ${\mathbf{S}}_q$ for every q. All terms can be computed in constant time using auxiliary arrays that return, for each i and q, the values of $minB$ and $maxB$ for $i\in B$ and $B\in {\mathbf{S}}_q$. These auxiliary arrays in turn can obviously be constructed in $O\left(nm\right)$ time for the breakpoint list of the input segmentations.    □

It, therefore, makes sense to precompute the arrays ${\delta }_{<}$, ${\delta }_{\le }$, ${\delta }_{<}^{\cap }$, and ${\delta }_{<}^{\cap }\left(k\right)$.

Corollary 1.

The score ${\Delta }[j+1,k]$ can be computed in $O\left(m\right)$ time with $O\left(nm\right)$ preprocessing cost to compute the arrays ${\delta }_{<}$, ${\delta }_{\le }$, ${\delta }_{<}^{\cap }$, and ${\delta }_{>}^{\cap }$.

It is worth noting, finally, that there is nothing to be gained by storing the score values ${\Delta }[j+1,k]$ since each entry is used only once in the recursion.

2.3. Boundaries of Consensus Segments

For a function g on $\mathbb{Z}$, we define the local curvature at x as ${\partial }_x^{2}g\left(x\right):=g(x+1)+g(x-1)-2g\left(x\right)$. A function g is (strictly) convex at x if ${\partial }_x^{2}g\left(x\right)>0$. This condition immediately implies that x is not a local maximum of g since at least one of $g(x+1)$ or $g(x-1)$ is larger than $g\left(x\right)$. Correspondingly, g is (strictly) concave in x if ${\partial }_x^{2}g\left(x\right)<0$, whence x is not a local minimum.

Definition 1.

The potential $\mathfrak{e}$ is boundedly convex if satisfied for all intervals $p^{\prime }\le p\le x\le q\le q^{\prime }$

$${\partial }_x^2\mathfrak{e}\left([x,q]\right)\ge {\partial }_x^2\mathfrak{e}\left([x,q^{\prime }]\right)>0\mathrm{and}{\partial }_x^2\mathfrak{e}\left([p,x]\right)\ge {\partial }_x^2\mathfrak{e}\left([p^{\prime },x]\right)>0.$$

(13)

For boundedly convex $\mathfrak{e}$, the curvature in non-increasing as the intervals become larger. In particular, suppose $\mathfrak{e}\left(\right[p,q\left]\right)$ depends only on the length $z:=q-p+1$ of the interval and is a smooth function, then ${\mathfrak{e}}^{″}\left(z\right)>0$ and ${\mathfrak{e}}^{″\prime }\left(z\right)\le 0$ for all $z>0$ implies $\mathfrak{e}$ is boundedly convex.

Theorem 2.

Let $\{{\mathbf{S}}_1,{\mathbf{S}}_1,\cdots ,{\mathbf{S}}_m\}$ be a set of segmentations with union segmentation $\widehat{\mathbf{S}}$ and suppose $\mathfrak{e}$ is boundedly convex. Then, the consensus $\mathbf{C}$ is refined by the union segmentation $\widehat{\mathbf{S}}$.

Proof. 

Following References [24,33], we assume, for contradiction, that the optimal consensus $\mathbf{C}$ has a segment boundary $\widehat{\mathit{j}}$ that is not contained in the union segmentation $\widehat{\mathbf{S}}$. We aim to show that moving $\widehat{\mathit{j}}$ to some close-by position x will reduce the cost $f\left(\mathbf{C}\right)$. We focus on a fixed input segmentation ${\mathbf{S}}_q$ and denote by ${\widehat{\mathbf{S}}}_{\mathbf{q}}:={\mathbf{S}}_{\mathbf{q}}\wedge \mathbf{C}$. Denote by $p-1$ and q the first boundaries to the left and to the right of $\widehat{\mathit{j}}$ in $\mathbf{C}$. Analogously, $\widehat{p}$ and $\widehat{q}$ are the first boundary to the left and to the right of $\widehat{\mathit{j}}$ in ${\widehat{\mathbf{S}}}_{\mathbf{q}}$, respectively. Thus, $\mathbf{C}$ contains the two segments $[p,\widehat{j}]$ and $[\widehat{\mathit{j}}+1,q]$. Since every segment of ${\widehat{\mathbf{S}}}_{\mathbf{q}}\wedge \mathbf{C}$ is a subset of a unique segment of $\mathbf{C}$, we have $[\widehat{p},\widehat{q}]\subseteq [p,q]$.

We proceed by evaluating how $D({\mathbf{S}}_{\mathbf{q}},\mathbf{C})=E\left({\mathbf{S}}_{\mathbf{q}}\right)+E\left(\mathbf{C}\right)-2E({\mathbf{S}}_{\mathbf{q}}\wedge \mathbf{C})$ varies when the boundary $\widehat{\mathit{j}}$ is perturbed. Let x be the perturbed boundary position. Since only $E\left(\mathbf{C}\right)$ and $2E({\mathbf{S}}_{\mathbf{q}}\wedge \mathbf{C})$ depends on x and all boundaries except $\widehat{\mathit{j}}$ are fixed, it suffices to focus on the intervals $[p,q]$ and $[\widehat{p},\widehat{q}]$, respectively. Collecting all constant terms in $D_0$, we obtain

$$D\left(x\right)=D_0+\mathfrak{e}\left([p,x]\right)+\mathfrak{e}\left([x+1,q]\right)-2\mathfrak{e}\left([\widehat{p},x]\right)-2\mathfrak{e}\left([x+1,\widehat{q}]\right).$$

(14)

Since $\mathfrak{e}$ is boundedly convex, we have $0<{\partial }_x^2\mathfrak{e}\left([p,x]\right)\le {\partial }_x^2\mathfrak{e}\left([\widehat{p},x]\right)$ and $0<{\partial }_x^2\mathfrak{e}\left([x+1,q]\right)\le {\partial }_x^2\mathfrak{e}\left([x+1,\widehat{q}]\right)$, whence ${\partial }_x^{2}D\left(x\right)<-{\partial }_x^2\mathfrak{e}\left([\widehat{p},x]\right)-{\partial }_x^2\mathfrak{e}\left([x+1,\widehat{q}]\right)<0$. Thus, $D\left(x\right)$ is concave at x for every ${\mathbf{S}}_{\mathbf{q}}$ and, thus, also for any non-negative contribution to the linear combination of input segmentations. Thus, $f\left(\mathbf{C}\right)$ as a function of the moving boundary x cannot have a minimum in the interior of the interval $[\widehat{p},\widehat{q}]$, contradicting the assumption that $\widehat{\mathit{j}}$ is a boundary in the optimal consensus $\mathbf{C}$.    □

Theorem 2 establishes a very useful property: All segment boundaries of the consensus are contained in the union segmentation. This property was observed for disagreement distance and information distance (see below) in the unweighted setting [24,33]. Here, we show it holds for a broader class of distance functions and arbitrary weighting schemes. The techniques used in the proof of Theorem 2 do not seem to generalize to potentials with increasing curvature. Numerical data, however, indicate that the union segmentation refines the consensus for a much larger class of potential functions.

From an algorithmic point of view, it implies that it suffices to compute the $F_k$ for those values of k where segment boundaries are in the union segmentation of the inputs $\widehat{\mathbf{S}}$. Correspondingly, we need to store the auxiliary variables only for the intervals of the union segmentation, instead of each i. That is, the recursion (8) reduced to

$$F_{j_k}=\underset{\genfrac{}{}{0pt}{}{i<k}{j_i\in \partial \widehat{\mathbf{S}}}}{min}\left({\Delta }\left([j_i+1,j_k]\right)+F_{j_i}\right),$$

(15)

where $j_i\in \partial \widehat{\mathbf{S}}$ denotes the i-th segment boundary in the union segmentation $\widehat{\mathbf{S}}$.

Recursion (15) also speeds up the computation of the scoring function ${\Delta }$, which now is also needed only for the segment boundaries. First, note that we still obtain ${\delta }_{<}\left(i_k\right)$ from ${\delta }_{<}\left(i_{k-1}\right)$ by adding the contributions $\mathfrak{e}\left(B\right)$ for the intervals ending at the boundary $i_k$ to ${\delta }_{<}\left(i_{k-1}\right)$ since by definition $i_{k-1}$ and $i_k$ are consecutive breakpoints. Analogously, ${\delta }_{>}\left(i_k\right)$ is obtained by adding $\mathfrak{e}\left(B\right)$ for all blocks beginning at $i_{k-1}$ to ${\delta }_{>}\left(i_{k-1}\right)$. The terms ${\delta }_{>}^{\cap }\left(i_k\right)$ and ${\delta }_{<}^{\cap }\left(i_k\right)$ remain the same. The correction term ${\delta }^{*}$ could be stored for all pairs of the boundaries in $\widehat{\mathbf{S}}$. Alternatively, it suffices to store the m boundaries at which the intervals crossing $i_k$ start and to keep track of the correct correction term directly in recursion (15). Equation (15), thus, can be evaluated in $O\left(s^2\right)$, where s is the number of breakpoints in the union segmentation.

2.4. Length Bounds on Consensus Segments

It is reasonable to expect that a consensus segmentation cannot be a lot coarser than the individual input segmentations. To see that this is indeed the case, we start with a technical observation.

Lemma 2.

Consider intervals $A=[i,k]$, $A^{\prime }=[i,x]$ and $A^{″}=[x+1,k]$. Then ${\Delta }\left(A\right)>{\Delta }\left(A^{\prime }\right)+{\Delta }\left(A^{″}\right)$ if for every ${\mathbf{S}}_{\mathbf{q}}$ there is $B\in {\mathbf{S}}_{\mathbf{q}}$ with $x\in B$ such that

$$\mathfrak{e}\left(A\right)-\left(\mathfrak{e}\left(A^{\prime }\right)+\mathfrak{e}\left(A^{″}\right)\right)>2\left[\mathfrak{e}\left(B\right)-\left(\mathfrak{e}(B\cap A^{\prime })+\mathfrak{e}(B\cap A^{″})\right)\right]$$

(16)

Proof. 

Equation (7) implicitly defined ${\Delta }\left(A\right)$ as the term in parentheses, which in turn is the $w_q$-weighted sum of contributions for each ${\mathbf{S}}_{\mathbf{q}}$. Consider $B\in {\mathbf{S}}_{\mathbf{q}}$ with $B\subset A$. The contribution $d_q\left(A\right)$ of ${\mathbf{S}}_{\mathbf{q}}$ to ${\Delta }\left(A\right)$ is

$$\begin{array}{ccc}\hfill d_q\left(A\right)& =& \mathfrak{e}\left(A\right)-2\sum _{B^{\prime }\in {\mathbf{S}}_{\mathbf{q}}}\mathfrak{e}(A\cap B^{\prime })\hfill \\ & =& \mathfrak{e}\left(A\right)-2\left(\sum _{\genfrac{}{}{0pt}{}{B^{\prime }\in {\mathbf{S}}_{\mathbf{q}}:}{max{B}^{\prime }<maxB}}\mathfrak{e}(A\cap B^{\prime })+\sum _{\genfrac{}{}{0pt}{}{B^{\prime }\in {\mathbf{S}}_{\mathbf{q}}}{max{B}^{\prime }>maxB}}\mathfrak{e}(A\cap B^{\prime })\right)-2\mathfrak{e}\left(B\right).\hfill \end{array}$$

Now, consider an alternative segmentation in which A is subdivided into $A^{\prime }\dot{\cup }{A}^{″}$ at some position x inside B. Then, A contributes

$$\begin{array}{ccc}\hfill d_q\left(A^{\prime }\right)+d_q\left(A^{″}\right)& =& \mathfrak{e}\left(A^{\prime }\right)+\mathfrak{e}\left(A^{″}\right)-2\left(\mathfrak{e}(A^{\prime }\cap B)+\mathfrak{e}(A^{″}\cap B)\right)\hfill \\ & & -2\left(\sum _{\genfrac{}{}{0pt}{}{B^{\prime }\in {\mathbf{S}}_{\mathbf{q}}:}{max{B}^{\prime }<maxB}}\mathfrak{e}(A^{\prime }\cap B^{\prime })+\sum _{\genfrac{}{}{0pt}{}{B^{\prime }\in {\mathbf{S}}_{\mathbf{q}}}{max{B}^{\prime }>maxB}}\mathfrak{e}(A^{″}\cap B^{\prime })\right).\hfill \end{array}$$

The terms corresponding to the segments $B^{\prime }\ne B$ that intersect A are the same as before since either $B^{\prime }\cap A=B^{\prime }\cap A^{\prime }$ or $B^{\prime }\cap A=B^{\prime }\cap A^{″}$, depending on whether $B^{\prime }$ comes before or after B in ${\mathbf{S}}_{\mathbf{q}}$. Thus, we have $d_q\left(A\right)>d_q\left(A^{\prime }\right)+d_q\left(A^{\prime }\right)$ if and only if Equation (16) is satisfied. Since ${\Delta }\left(A\right)$, ${\Delta }\left(A^{\prime }\right)$, and ${\Delta }\left(A^{″}\right)$ are convex linear combinations of the $d_q\left(A\right)$, $d_q\left(A^{\prime }\right)$, and $d_q\left(A^{″}\right)$, respectively, it is sufficient for ${\Delta }\left(A\right)>{\Delta }\left(A^{\prime }\right)+{\Delta }\left(A^{″}\right)$ that $d_q\left(A\right)>d_q\left(A^{\prime }\right)+d_q\left(A^{\prime }\right)$ holds for all ${\mathbf{S}}_{\mathbf{q}}$.    □

In other words, if A satisfies the condition of Lemma 2, then $\tilde{f}\left(\mathbf{C}\right)$ strictly decreases when A is subdivided into $A^{\prime }$ and $A^{″}$. Thus, we conclude:

Corollary 2.

An interval A satisfying the conditions specified in Lemma 2 cannot appear in a consensus segmentation.

Our goal is now to show that sufficiently long intervals A always satisfy the conditions of Lemma 2 and, thus, can never be part of the consensus segmentation. Here, we need that $\mathfrak{e}$ is superadditive, i.e., $\mathfrak{e}\left(A\right)>e\left(A_1\right)+e\left(A_2\right)$ for all $A=A_1\dot{\cup }{A}_2$ and $A_1,A_2\ne Ø$. This is the case particularly for the polynomial potentials. It fails for the negentropy potential, Equation (17), however, because this function is not monotonically increasing with the segment length $\left|A\right|$.

Theorem 3.

Let $\mathfrak{e}$ be a superadditive potential. Let B be the longest segment in the input segmentations and denote by ${\ell }^{*}$ the length of the shortest interval A such that $\mathfrak{e}\left(A\right)-2\mathfrak{e}\left(A^{\prime }\right)>2\mathfrak{e}\left(B\right)-2{min}_{B^{\prime },B^{″}:B^{\prime }\cup B^{″}=B}(\mathfrak{e}\left(B^{\prime }\right)+\mathfrak{e}\left(B^{″}\right))$, where $|A^{\prime }|=\lceil |A|/2\rceil$ and $|B^{\prime }|=\lfloor |B|/2\rfloor$. Then every segment of the consensus segmentation is shorter than $L^{*}:=max\left(2\right|B|,{\ell }^{*})$.

Proof. 

If $\mathfrak{e}$ is superadditive, the l.h.s. of Equation (16) is maximal if $|A^{\prime }|=|A^{″}|$ (for even $\left|A\right|$) or $|A^{\prime }|=|A^{″}|\pm 1$ for odd $\left|A\right|$, i.e., we assume that x is located in the middle of A. In order to ensure that segments containing x are completely contained in A we need $\left|A\right|\ge 2\left|B\right|$. If this condition is satisfied, Equation (16) applies. We obtain a sufficient condition by replacing the r.h.s. with the maximal possible contribution of the subdivided interval B. By superadditivity, this term monotonically increases with the size of B. The assumption that x equally divides A fixed the l.h.s. of the inequality. Since $\mathfrak{e}$ is strictly superadditive $\mathfrak{e}\left(A\right)-2\mathfrak{e}\left(A^{\prime }\right)$ is strictly monotonically increasing with $\left|A\right|$, thus, there is a unique smallest value ${\ell }^{*}$ of $\left|A\right|$ unless $\mathfrak{e}\left(A\right)-2\mathfrak{e}\left(A^{\prime }\right)\le 2$ for all A, in which case no bound ${\ell }^{*}$ exists.    □

Corollary 3.

The consensus segmentation $\mathbf{C}$ with superadditive potential $\mathfrak{e}$ for m input segmentations with length bound $L^{*}$ as specified in Theorem 3 can be computed in $O\left(nm{L}^{*}\right)$ time.

Proof. 

We observe that for each k, only values of j between $k-2{\ell }^{*}$ and $k-1$ appear in Equation (8) since longer segments by Theorem 3 cannot be part of an optimal consensus segmentation. The corollary now follows immediately from Corollary 1.    □

The length bound on consensus segments, thus, leads to a reduction of the computational efforts. Although ${\ell }^{*}$ in Theorem 3 may be inconvenient to compute for some choices of the potential $\mathfrak{e}$, we shall see below that a simple, uniform bound can be obtained for an interesting class of potentials.

2.5. Special Potential Functions

Let us now consider plausible distance functions. The disagreement distance between segmentation was introduced in Reference [24] using the potential $\mathfrak{e}\left(A\right):={\left(\right|A|/n)}^2/2$. A natural generalization is $\mathfrak{e}\left(A\right)={\left(\right|A|/n)}^{1+\alpha }/(1+\alpha )$ for $0<\alpha \le 1$. We note that a linear potential $\mathfrak{e}\left(\right|A\left|\right)=\left|A\right|/n$, i.e., $\alpha =0$, yields a constant value of $\tilde{f}\left(\mathbf{C}\right)$ because the sum of all segment lengths adds up to n; thus, $E\left(\mathbf{S}\right)=1$ is independent of the segmentation $\mathbf{S}$.

Recall that the entropy of a discrete distribution is defined as $H=-{\sum }_i{p}_{i}ln{p}_i$. Given a segmentation $\mathbf{S}$, we consider the probabilities $p_i$ of randomly picking a point from a segment, i.e., $p_i=\left|A_i\right|/n$ is the relative length of a segment $A_i\in \mathbf{S}$, where n denotes the total length of the segmented genome or time series. The information distance is the symmetrized conditional entropy, which can also be computed as $D({\mathbf{S}}^{\prime },{\mathbf{S}}^{″})=2H({\mathbf{S}}^{\prime }\wedge {\mathbf{S}}^{″})-H\left({\mathbf{S}}^{\prime }\right)-H\left({\mathbf{S}}^{″}\right)$ [24,34]. It corresponds to the potential function

$$\mathfrak{e}\left(A\right):=\left(\right|A|/n)ln\left(\right|A|/n),$$

(17)

given by the negative of the entropy (negentropy) contribution of the interval A.

It has been shown in References [24,33] that the union segmentation $\widehat{\mathbf{S}}$ refines the unweighted consensus segmentation for both the disagreement distance and the information distance. This result generalizes to the weighted case and the $\alpha$-disagreement distances with $0<\alpha \le 1$.

Corollary 4.

The consensus segmentation $\mathbf{C}$ is refined by the union segmentation $\widehat{\mathbf{S}}$ for the disagreement distance, its α generalization with $0<\alpha \le 1$, as well as the information distance.

Proof. 

It suffices to show that the potentials $\mathfrak{e}\left(z\right)$ are boundedly convex. For the disagreement distance, we have $\mathfrak{e}\left(z\right)=z^2/2$, and we have ${\mathfrak{e}}^{″}\left(z\right)=1$ and ${\mathfrak{e}}^{‴}\left(z\right)=0$; for $\mathfrak{e}\left(z\right)=z^{1+\alpha }/(1-\alpha )$, we have $e^{″}\left(z\right)=\alpha z^{\alpha -1}>0$ and $e^{‴}\left(z\right)=\alpha (\alpha -1)z^{\alpha -2}<0$ for $0<\alpha \le 1$. For the negentropy, $\mathfrak{e}\left(z\right)=zlnz$, we have ${\mathfrak{e}}^{″}\left(z\right)=1/z>0$ and ${\mathfrak{e}}^{‴}\left(z\right)=-1/z^2<0$, where $z:=\left(\right|A|/n)$. The scaling by $1/n$ obviously does not affect the signs.    □

Is does not seem possible to generalize the result to potentials that grow faster than quadratically.

Let us finally consider the consequence of Theorem 3. Reusing the convexity results above we can replace $2{min}_{B^{\prime },B^{″}:B^{\prime }\cup B^{″}=B}(\mathfrak{e}\left(B^{\prime }\right)+\mathfrak{e}\left(B^{″}\right))$ by $4\mathfrak{e}\left(B^{\prime }\right)$ where $|B^{\prime }|=\lfloor |B|/2\rfloor$. A short computation then shows that the inequality in Theorem 3 is satisfied for $\left|A\right|>\sqrt[1+\alpha ]{2}\left|B\right|$. Since $\sqrt[1+\alpha ]{2}\le 2$, we have

Corollary 5.

The consensus segmentation $\mathbf{C}$ of a collection of segmentations ${\mathbf{S}}_{\mathbf{q}}$ with respect to the α-disagreement potentials contains no segment longer than twice the length of the longest input segment.

This allows us immediately to limit the range of the indices in recursion (8) to $j_i>j_k-2max\left|B\right|$.

2.6. Generalization: Symmetrized Boundary Mover’s Distance

Equation (7) highlights the fact that the cost function $\tilde{f}\left(\mathbf{C}\right)$ measures, for each segment $A\in \mathbf{C}$, how well A conforms to the input segmentations. As noted above, the additive structure of Equation (7) is sufficient to enable minimization by dynamic programming for arbitrary choices of ${\Delta }$. If we retain the idea of weighted contributions for each input segmentation, we may write ${\Delta }\left(A\right)={\sum }_q{w}_q{\Delta }\left(A\right|{\mathbf{S}}_q)$, where ${\Delta }\left(A\right|{\mathbf{S}}_q)$ measures how well the segment A “fits” into the segmentation ${\mathbf{S}}_q$. As a minimal requirement, for any given interval A, the score ${\Delta }\left(A\right|{\mathbf{S}}_q)$ must attain its minimum value if the interval A is a segment in ${\mathbf{S}}_q$. Since two segmentations in general do not have segments or breakpoints in common, measures are required that are more fine-grained than the distinction between identical and distinct segments or breakpoints. ${\Delta }\left(A\right|{\mathbf{S}}_q)$, thus, are similar to a measure of overlap, between A and the segments of ${\mathbf{S}}_q$ that are covered by A. Clearly, the potential-based measures can be understood in this manner.

An interesting class of dissimilarities utilizes the distance between break points instead of the lengths of intersections between segments. For a segmentation $\mathbf{S}$ with segments $S_i$, $i=1,\cdots ,n$, we define $s_i=max{S}_i$ and set $s_0=0$, i.e., the segments are $S_i=[s_{i-1}+1,s_i]=:(s_{i-1}..s_i)$. By slight abuse of notation, we write $\mathbf{S}=(s_0,s_1,\cdots ,s_m)$, i.e., we now specify a segmentation in terms of its breakpoints. Moreover, we write $s\in \mathbf{S}$ to mean that s represents a breakpoint in the segmentation $\mathbf{S}$.

The “boundary movers distance” was introduced in References [24,33] as

$$D_B\left(\mathbf{S}\right|\mathbf{C}):=\sum _{s\in \mathbf{S}}\underset{c\in \mathbf{C}}{min}d(s,c),$$

(18)

where $d(.,.)$ is some distance function between the positions s and c on $[1,\cdots ,n]$. The dissimilarity measure $D_B$ is not symmetric and satisfies $D_B\left(\mathbf{S}\right|\mathbf{C})=0$ whenever $\mathbf{C}$ is a refinement of $\mathbf{S}$. The segmentation aggregation problem that minimizes ${\sum }_q{w}_q{D}_B\left({\mathbf{S}}_{\mathbf{q}}\right|\mathbf{C})$, therefore, is solved by the union segmentation $\mathbf{C}=\widehat{\mathbf{S}}$, while ${\sum }_q{w}_q{D}_B\left(\mathbf{C}\right|{\mathbf{S}}_{\mathbf{q}}\left|\right)$ is minimized by the indiscrete segmentation $\left\{\right[1,n\left]\right\}$. As noted in References [24,33], these measures, thus, are only useful with additional constraints on the number or size of allowed segments.

The symmetrized version of $D_B$, however, has attractive properties for our purposes, as we shall see: Clearly, ${min}_{c\in \mathbf{C}}d(s,c)=min\{d(s,c^{\prime }),d(s,c^{″})\}$, where $c^{\prime }$ and $c^{″}$ delimit the segment of $\mathbf{C}$ within which s resides. If $s=c^{\prime }$ or $s=c^{″}$, the contribution vanishes; hence, we can write

$$D_B\left(\mathbf{S}\right|\mathbf{C}):=\sum _{(c^{\prime }...c^{″})\in \mathcal{C}}\sum _{s\in (c^{\prime }...c^{″})}min\{d(s,c^{\prime }),d(s,c^{″})\}.$$

(19)

This term individually penalizes a segment $(c^{\prime },c^{″})$ of $\mathbf{C}$ for containing boundary points of $\mathbf{S}$ in its interior. On the other hand, we can rewrite $D_B\left(\mathbf{C}\right|\mathbf{S})$ in terms of segments of $\mathbf{C}$ by simply splitting the contribution of each boundary between the two adjacent segments:

$$D_B\left(\mathbf{C}\right|\mathbf{S})=\sum _{(c^{\prime },c^{″})\in \mathbf{C}}\frac{1}{2}\left(\underset{s\in \mathbf{S}}{min}d(s,c^{\prime })+\underset{s\in \mathbf{S}}{min}d(s,c^{″})\right).$$

(20)

Here, we have used that the lower bounds of the first segments and the upper bounds of the last segments necessarily coincide (they are the boundary of the interval on which our segmentations live); therefore, they do not contribute to the distance. Again, the minima are only taken over two alternative breakpoints of $\mathbf{S}$ for each given value of $c^{\prime }$ or $c^{″}$, namely those delimiting the segments of $\mathbf{S}$ harboring the breakpoints $c^{\prime }$ and $c^{″}$ of the consensus segment. It is not difficult to see that $D(\mathbf{S},\mathbf{C})=D_B\left(\mathbf{S}\right|\mathbf{C})+D_B\left(\mathbf{C}\right|\mathbf{S})$ vanishes only if $\mathbf{S}=\mathbf{C}$. Furthermore, $D_B\left({\mathbf{S}}_q\right|\mathbf{C})+D_B\left(\mathbf{C}\right|{\mathbf{S}}_{\mathbf{q}})$ can be written as a sum of contributions

$${{\Delta }}_q\left((c^{\prime }...c^{″})\right):=\sum _{\genfrac{}{}{0pt}{}{s\in S_q}{s\in (c^{\prime }...c^{″})}}min\{d(s,c^{\prime }),d(s,c^{″})\}+\frac{1}{2}\underset{s\in {\mathbf{S}}_q}{min}d(s,c^{\prime })+\frac{1}{2}\underset{s\in {\mathbf{S}}_q}{min}d(s,c^{″})$$

(21)

for each of the segments $(c^{\prime },c^{″})\in \mathbf{C}$ and each segment of the input segmentations ${\mathbf{S}}_q$. Clearly, ${\Delta }\left((c^{\prime }...c^{″})\right)={\sum }_q{w}_q{{\Delta }}_q\left((c^{\prime }...c^{″})\right)$ depends only on the input segmentations ${\mathbf{S}}_q$ and the boundary breakpoints $c^{\prime }$ and $c^{″}$, i.e., an individual segment in the consensus $\mathbf{C}$. The segmentation aggregation problem with the symmetrized boundary movers distance, therefore, can again be solved by dynamic programming recursion Equation (8). A more in-depth analysis of this distance function is the subject of ongoing research.

3. Computational Results

3.1. Implementation

The consensus segmentation algorithm is available as an R package consseg, where the dynamic programing recursion is implemented in C++ via Rcpp (≥0.12.18) and RcppXPtrUtils (≥0.1.1) to allow user-defined potential functions. A CRAN package accompanying this contribution will be made available. The development version is available at https://github.com/Bierinformatik/consseg (accessed on 1 December 2020).

The input segmentations are converted into an index returning for each position k the minimum position $min{B}_q$ and the maximum position $max{B}_q$ of the segment $B_q$ containing k. With their help, ${\delta }_{<}\left(k\right)$ is obtained by adding $w_q\mathfrak{e}\left(B_q\right)$ to ${\delta }_{<}(k-1)$ if $k=max{B}_q$. Analogously, ${\delta }_{\le }\left(k\right)$ is computed by adding $w_q\mathfrak{e}\left(B_q\right)$ to ${\delta }_{\le }\left(k\right)$ whenever $k=min{B}_q$. The terms ${\delta }_{<}^{\cap }\left(k\right)$ and ${\delta }_{>}^{\cap }\left(k\right)$ are evaluated as defined in Equation (10). These computations are interleaved with the evaluation of $F_k$, Equation (8). Since the expensive part in the algorithm is the evaluation of the segment cost ${\Delta }[j+1,k]$, we avoid their recomputation in the backtracking step by storing instead the values $J\left[k\right]$ of the segment boundary j that realizes the minimum in Equation (8) for position k. Thus, the last segment of the optimal segmentation on $[1,k]$ is $\left[J\right[k]+1,k]$. Backtracking then proceeds on $[1,J[k\left]\right]$. The segment boundaries of the optimal segmentation are, therefore, obtained as $j_{i+1}=J\left[j_i\right]$, starting from $j_0=n$ and continuing until $j_k=0$ is reached. It is worth noting that the fast, incremental Equation (12) is prone to rounding errors for large n and very fast-growing potentials $\mathfrak{e}$ due to the computation of difference ${\delta }_{<}\left(k\right)-{\delta }_{\le }\left(j\right)$ of two sums.

3.2. Consensus Segmentation of Yeast Transcriptome Data

To demonstrate the usefulness of consensus segmentations, we explored the yeast transcriptome time series mentioned in the introduction. We computed the consensus of segmentations obtained with widely different parameter choices [11]. We found that the consensus segmentation appears to produce a robust representation of the transcriptome and seems to fit better to the current annotation of the yeast genome than any particular choice of segmentation parameters. The example in Figure 3 also shows that distinct non-coding components, such as SRG1, are readily detectable. Short segments with very low coverage are likely gaps between transcriptome units without relevant RNA products. Consistently detectable elements, even if lowly expressed, on the other hand, may be of interest for closer inspection.

In order to evaluate the usefulness of consensus segmentations in a more quantitative manner, we quantify the overlap of segments with annotated coding sequences. To this end, we determine for each CDS C the segment $B\left(C\right)$ with the largest Jaccard index and then record the ratio $r\left(C\right)$ of segment length and annotation length. In symbols:

$$r\left(C\right)\left|B\left(C\right)\right|/\left|C\right|\mathrm{with}B\left(C\right)\underset{B\in \mathsf{S}}{arg\; max}\frac{|B\cap C|}{|B\cup C|}.$$

(22)

The cumulative distribution function $cdf\left(r\right)$ computed over a large set of known transcripts C quantifies the congruence between segmentation and annotation. As a reference we use here the transcripts harboring coding sequences annotated in Reference [36]. A perfect overlap between a consensus segment and the annotated transcript is indicated by $r=1$, $r<1$ indicates a segment that is shorter and $r>1$ a segment that is longer then the annotated transcript. Figure 4a shows the cumulative distribution function $cdf\left(r\right)$ for the 5171 CDSs of S. cerevisae IFO 0233 for five segmentations with widely different parameters computed with segmenTier [11]. The red curve is the consensus over these segmentations. It shows that the consensus segmentation is a robust method: computed from a small sample of distinct segmentations, some of which do not perform particularly well, it performs at least as well as the best individual segmentation obtained from an extensive search of the parameter space in Reference [11]. Discrepancies between annotation and consensus are not only limitations on the segmentation approach but also derive from inaccuracies of the annotation, processing of transcripts, and the complexity of the yeast transcriptome, which harbors abundant overlapping and polycistronic transcripts [37]. The consensus performs as well as the best individual segmentation (according to the benchmark in Reference [11]). Figure 4b shows that the individual segmentations share between about 30% and 70% of the segments with the consensus (corresponding to the height of the vertical jump at $r=1$), i.e., the consensus does not simply recapitulate any individual input segmentation. We, therefore, advocate to use the consensus segmentation as a robust and essentially parameter insensitive method for transcriptome analysis in compact genomes.

The consensus segmentation of the transcriptome of S. cerevisae IFO 0233 comprises 74,091 segments. After filtering for spacers using the input segmentations [11] and very short segments that most likely correspond to small overlaps and noise in the RNA-seq data, we retained 32,480 segments. Figure 5 shows the distribution of median coverage. Not surprisingly, segments overlapping known protein-coding sequences (CDS) show higher expression levels than other segments. Many segments overlap various types of long non-coding RNAs, such as CUTs and SUTs [38,39]. We also observe many segments with substantial expression levels that so far have remained unannotated, providing a large pool of candidates for novel ncRNAs. Transcriptome segmentation is only the first step towards an accurate and reliable genome annotation. The subsequent processing of the segmentation data, however, is beyond the scope of this present contribution and will be addressed in forthcoming work.

3.3. Consensus Segmentations of Growth Curves

The usefulness of consensus segmentations is by no means limited to transcriptome data or segmentations of genomes. We, therefore, include here also a very different application. The growth of a population of bacteria over time can be quantified by measuring the apparent absorption, usually referred to as OD (optical density), in a spectrophotometer. Growth curves typically show distinct growth regimes: an initial time lag that precedes a phase of exponential growth, which is followed by a deceleration phase that finally settles into saturation, see, e.g., Reference [40]. These can be separated by approximating the time course of $logOD$ by a sequence of line segments, i.e., as a continuous, piece-wise linear function. The corresponding approximation problem is again a segmentation problem that can be solved by dynamic programming [28]. Empirically, one observes that resulting segmentations are quite sensitive to small difference in the growth curves. We show here that the consensus segmentation is a convenient way to extract a robust estimate for the duration of the different phases.

In Figure 6, we compare the growth curves of four Escherichia coli cultures grown in minimal glucose medium at 37 ${}^{°}$C. The R package dpseg [41] was used to segment the individual growth curves. This package also uses a dynamic programming approach. Instead of fixing the number of segments as in Reference [28], it uses a penalty parameter to adjust the resolution of the segmentation. Considering the consensus of the individual segmentations and the variation of the breakpoints across the replicates provides a more realistic view of that data compared to the segmentation of an averaged growth curve.

3.4. Refinement Conjecture

Theorem 2 states that the union segmentation $\widehat{\mathbf{S}}$ refines the consensus segmentation $\mathbf{C}$ for the class of boundedly convex consensus functions. Numerical simulations strongly suggest that this is true also for many potentials with increasing positive curvature, despite the failure of the proof technique for the general case. We simulated 10 segmentations of length 50 and a maximum of 10 segments, using the base-R sample function to randomly choose breakpoints in a given range. Figure 7 shows consensus segmentations for six potential functions from negentropy to exponential.

We found that the consensus segmentation only contained breakpoints that are present in at least one of the input segmentation. This suggests:

Conjecture 1.

$\widehat{\mathbf{S}}$ appears to refine $\mathbf{C}$ for all superadditive potentials and possibly even for all convex potentials.

This “Refinement Conjecture” is of considerable practical use. If true, (assumed as a heuristic), it reduces the computational effort to $O\left(s^2\right)$ where s is the number of break points in the input segmentations. We further observed a trend for faster-growing potentials to yield more and, thus, shorter consensus segments. This is consistent with the fact that the bound $\sqrt[1+\alpha ]{2}\left|B\right|$ on the length of the consensus intervals in the argument leading up Corollary 5 decreases with the exponent $\alpha$ in polynomial potentials.

4. Concluding Remarks

In this work, we have extended and generalized previous work [24,33] on the segmentation aggregation problem. We showed that for the class of boundedly convex potential functions, including negentropies and powers $z^{1+\alpha }$ with $0<\alpha \le 1$, all consensus breakpoints are breakpoints in at least one of the contributing segmentations. Furthermore, we showed that for all superadditive potentials, consensus segments cannot be longer than twice the longest input segment. This bound allows a further reduction of the computational effort.

Consensus segmentations as described here pertain to two major application scenarios: (i) Reconciliation of segmentations of multi-dimensional data, comprising, e.g., independent measurements, such as biological or technical replicates, or measurements of different quantities, e.g., different histone modifications. (ii) Reconciliation of segmentations of the same data set produced with different similarity measures. In principle, it is also possible to compute the consensus segmentation of different segmentations produced, e.g., with randomized algorithms or different heuristics using the same similarity measures. A major advantage of consensus segmentations is that they can be computed without specific information about the data underlying the input segmentations. Such knowledge is not needed because the segmentation aggregation problem depends only on the distance function D as a “parameter”. Empirically, we found that variations of the distance functions have only very moderate consequence on the consensus segmentation.

In simulations, we found strong support for the Refinement Conjecture. This provides support for approaches that utilize a dynamic programming segmentation method to select be best segmentation from the union of segmentations that are computed with different heuristics. Such a scheme has been proposed in Reference [42]. In this manner, one can potentially achieve a substantial gain in computational efficiency compared to the full dynamic programming segmentation. The C-KS approach [12] also restricts itself to the union segmentation.

We considered two very different application scenarios. In applications to transcriptome data consensus segmentations have the potential to substantially improve annotations. A particular strength of the consensus approach is, by highlighting differences that are consistent between data tracks, the ability to identify processing-related boundaries. This is of particular interest in organisms with operons, poly-cistronic primary transcripts, or no expression-free gaps between genes. In all these cases, it becomes difficult and often impossible to distinguish transcriptional units from patterns of mapped RNA-seq reads alone. Here, we have used data from yeast strain IFO 0233, which has been used previously to illustrate transcriptome segmentation in Reference [11]. We have seen that the consensus segmentation provides a robust prediction of transcriptomic units from a moderate number of individual segmentations with very different parameters. We obtained thousands of segments that may correspond to the non-coding transcripts in S. cerevisae IFO 0233. Since the present contribution is intended to describe the method of consensus segmentation and its mathematical justification, we will report elsewhere on a comprehensive analysis of the IFO 0233 transcriptome.

Consensus formation is also of use to aggregate data from biological replicates. As an example, we showed that consensus segmentations of growth curves can be used to robustly determine distinct growth phases.

The consensus segmentation methods incorporate weights that refer to input segmentations. This feature can be used, for instance, to weight individual transcriptome data by coverage. In the case of growth functions, weights may be chosen to decrease with average measurement error, quantified, e.g., as average deviation from the linear fit. It would also be of interest to associate weights with individual segments. This can certainly be done in the context of the Boundary Mover’s distance. Whether this can be also be achieved in the potential-based approach, and to what extent the mathematics results of this contribution will remain intact, however, is a question for future research.

We observed that consensus segmentations are quite robust w.r.t. to the choice of the potential on real data, while we observed a trend towards shorter consensus segments with increasing $\alpha$ for the power potentials $e\left(z\right)=z^{\alpha -1}$ on i.i.d. random data. Conceptually, consensus segmentations based on the comparison of segments are designed to handle essentially arbitrary heterogeneity along the time or genome coordinate, while breakpoint-centered approaches, such as C-KS, need to rely on statistical regularities of true breakpoints. In order to assess the utility of different potentials $\mathfrak{e}(.)$ and dissimilarity measures $D(.,.)$, and to compare the segment-centered dynamic programming approach with breakpoint-centered alternatives, a principled way of benchmarking consensus segmentation methods will be necessary. This will require, in particular, the development of a simulator for correlated segmentations that mimic characteristics of different types of underlying data. At present, such tools are not available.

Our analysis of consensus segmentations suggests several avenues for future research. From a theoretical point of view, the most immediate open problem is the Refinement Conjecture, and the characterization of the potential function and—more generally—the dissimilarity measures for which the union segmentation refines the consensus segmentation. This is also of practical relevance since the recursions can be restricted to breakpoints of the union segmentation in such cases; see Equation (15). In addition, a more detailed understanding of the consensus segmentation would be useful. For instance, are there potentials or dissimilarities that guarantee a breakpoint in the consensus within every interval that contains a breakpoint in every input segmentation, or in the (weighted) majority of the input segmentations? Such results could immediately be used to limit the scope of j in Equation (8). More generally, one may ask whether the idea of consensus segmentation can give rise to useful ways of measuring the local accuracy or reliability of consensus and/or input break points.