The Entropy of Progressively Censored Samples

In many life-testing and reliability studies, the experimenter might not always obtain complete information on failure times for all experimental units. Among the different censoring schemes, the progressive censoring scheme has received a considerable attention in the last few years. The aim of this paper is simplifying the entropy of progressively Type II censored samples. We propose an indirect approach using a decomposition of the entropy in progressively Type II censored samples to simplify the calculation. Some recurrence relations for the entropy in progressively Type II censored samples are derived to facilitate this calculation. An efficient computational method is derived that simplifies computation of the entropy in progressively Type II censored samples to a sum; entropy in collections order statistics. We compute the entropy in a collection of progressively Type II censored samples for some known distributions.


Introduction
Information theory provides an intuitive tool to measure the uncertainty of random variables and the information shared by them, in which the entropy and the mutual information are two critical concepts.
Let X be a random variable with a cumulative distribution function (cdf) F (x) and probability density function (pdf) f (x).The differential entropy H(X) of the random variable is defined by Cover and Thomas [1] to be Let us consider a life-testing experiment where n units is kept under observation until failure.These units could be some system, components, or computer chips in reliability study experiments, or they could be patients put under certain drug or clinical conditions.Suppose the life lengths of these n units are independent identical random variables with a common cdf F (x) and pdf f (x).Data collected from such experiments called the order statistics sample where X r:n is called the rth-order statistics (OS).
For some reason, suppose that we have to terminate the experiment before all items have failed.For example, individuals in a clinical trial may drop out of the study, or the study may have to be terminated for lack of funds.In an industrial experiment, units may break accidentally.There are, however, many situations in which the removal of units prior to failure is pre-planned.One of the main reasons for this is to save time and cost associated with testing.Data obtained from such experiments are called censored data.
The most common censoring schemes are Type I and Type II censoring.In conventional Type I censoring, the experiment continues up to a prespecified time T .Any failures that occur after T are not observed.The termination point T of the experiment is assumed to be independent of the failure times.In conventional Type II censoring, the experimenter decides to terminate the experiment after a prespecified number of items r ≤ n fail.In this scenario, only the smallest lifetimes are observed.In Type I censoring, the number of failures observed is random and the endpoint of the experiment is fixed, whereas in Type II censoring the endpoint is random, while the number of failures is fixed.
Park [2] studied the entropy of Type II censored sample.Park [3] considered testing exponentiality based on the Kullback-Leibler information with the Type II censored data.The entropy of a single X r:n , and a complete order statistic sample has been studied in Wong and Chen [4] and Ebrahimi et al. [5].
Here we considers progressive Type II censored schemes.Among the different censoring schemes, the progressive censoring scheme has received a considerable attention in the last few years, particularly in reliability analysis.It is a more general censoring mechanism than the traditional Type I and Type II censoring [6].The recent review article by Balakrishnan [7] provide details on progressive censoring schemes and on its different applications.This paper is concerned with simplifying calculation of the entropy in progressively Type II censored data from the i.i.d.random sample of size n.However, the extension to progressively Type II censored data is not so straightforward, because the joint entropy of progressively Type II censored data is an n-dimensional integral.Besides, removals cause additional complications.
Following Balakrishnan and Aggarwala [8], progressively Type II censored samples can be described as follows.Let n units be placed in test at time zero.
are removed from experiment, so the life testing stops at the mth failure..

The observed failure times
then m = n which corresponds to the usual order statistics.
Thus, usual OS and the Type II censoring become a special cases of progressively Type II censored samples.So any result established for progressively Type II censoring becomes a generalization of the corresponding result for OS and the Type II censoring.
The likelihood function may be written as [8] f 1:m:n,....,m:m:n where To our knowledge, Balakrishnan et al. [9] generalized the result of Park [3] testing exponentiality based on the Kullback-Leibler information with the type II censored data to a progressively Type II censored data and obtained an approximate to the joint entropy in progressively Type II censored samples based on nonparametric estimation.Hence, the exact values of the joint entropy in progressively Type II censored samples has not been obtained.Several applications for entropy such as characterization, tests for goodness-of-fit based on censored data, parameter estimation and quantization theory are known, for example see [3,9].In the case of H 1....i:m:n , difficulty arise from the removal as well as the expression of H 1....i:m:n , which involves integration over i random variables, so simplifying the calculation of H 1....i:m:n is more attractive.In this article we focus on the study of the properties of the joint entropy in progressively Type II censored OS.In Section 2 we developed the idea of Park [2] about the decomposition of entropy in OS to introduce an indirect approach for decomposition of entropy in progressive Type II censored OS.In Section 3 we derive a recurrence relations for the entropy in progressively Type II censored samples, which will prove helpful in calculating the entropy.In Section 4 we derive an efficient computational method to reduce r-dimensional integrals in the calculation of H 1•••r:m:n to no integral where the computation of the entropy in progressively Type II censored samples simplifies to a sum; entropy of the smallest OS of varying sample size.In Section 5 we apply our results for computing the entropy in collections of a progressively Type II censored samples from normal and exponential distributions.

Decomposition of the Joint Entropy
Park [2] and Wong and Chen [4] have shown that the total entropy of i.i.d.random sample of size n is decreased if the sample is ordered.Park [2] showed how much the entropy of i.i.d.random sample of size n is decreased if the sample is ordered through, the following identity about the entropy of the ordered data h 1....:n:n h 1....:n In view of Equation ( 4) and noting that progressive Type II censored sample can be seen as an ordered sample we have the following result for the entropy of the progressive Type II censored OS sample.
, and Since the progressively Type II censored sample form a Markov chain [8], we have the following results.3), and can be written as where f 1....i:m:n (x 1:m:n , • • • , x i:m:n ) is the joint pdf of the first i order statistics of the progressively Type II censored sample.
Using the Markov chain property of the order statistics from progressive Type II censored samples, we have the following decomposition for the score function: The following decomposition follows from the strong additivity of the entropy On the other hand, in view of the result of Balakrishnan and Aggarwala [8], the f i+1•••m:n|i:m:n is the joint density of the progressively Type II censored sample of size (m − i), with censoring scheme •m:m:n|i:m:n can be written as the double integral where and f i:m:n (x) is defined by where and Since we already know about the entropy of the complete sample H 1•••m:m:n , the entropy H 1•••i:m:n can be now easily derived from Equations ( 6) and (8).EXAMPLE 2.1.For the exponential density exp(−x), we can show that g(w) = 1 so that Thus in that case where H 1:1:1 = 1 is the entropy in a single observation from the exponential density exp(−x).
REMARK 2.1.We note that all of Park's results concerning the entropy for the minimum order statistics X 1:n works for the case of progressively Type II censored sample, since f 1:n = f 1:m:n .

Recurrence Relations
Recurrence relations between the cdf (pdf) of OS and progressive Type II censored OS have been studied by many authors for the purpose of simplifying the calculation of moments of OS and progressive Type II censored OS.
The standard recurrence relation for the moments of OS was obtained by Cole [11], and can be written as where μ k i:j is the moments of the usual OS X i:j .This result can be directly derived from the corresponding recurrence relation between the cdf's of OS.Kamps and Cramr Lemma 4 [12] obtained the corresponding recurrence relation for generalized OS as Since the generalized OS includes the progressive Type II censored OS, it is clear that the case of progressive Type II censoring is subsumed in the above result.By setting Using Equation ( 14) and the decomposition of the entropy in Equation ( 8) we have the following results for the entropy in the progressive censoring scheme.RELATION 3.1 where PROOF.From Equation ( 8) we have on the other hand Equation ( 14) yields combining Equations ( 16) and ( 17) and noting that (m then the Lemma follows.The following relation shows that the entropy of the first r of the progressive Type II censored OS of sample size n − 1 can be obtained as a linear combination of the first r and r + 1 of the progressive Type II censored OS of sample size n. where PROOF.For a sample of size n − 1 the general decomposition of the entropy of progressive Type II censoring takes the form By applying RELATION 3.1 on Equation (20) we get where C 1 defined above.Equation ( 21) can be written, by using Equations ( 5) and ( 6), as After some simplifications the result follows.
0 all results of Sections 2 and 3 reduce to corresponding results for the entropy in a collections usual OS.

Computational Method for Calculating H 1•••i:m:n
In this section we provide another approach to simplify the calculation of the entropy in a collection of progressively Type II censored OS.We reduce r integrals in the calculation of H 1•••r:m:n to no integral where the computation of the entropy in progressively Type II censored samples simplifies to a sum; entropy of the smallest OS of varying sample size h 1:n .Lemma 4.1.Let X 1 , X 2 • • • X n be i.i.d.random sample of size n from pdf f (x) with cdf F (x) and hazard function h(x) = f (x) 1−F (x) , and let X 1:n , X 2:n , • • • , X n:n be OS corresponding to this sample.Park [2] obtained the entropy in the smallest order statistics as The entropy in the r collection of progressively Type II censored sample (X 1:m:n , X 2:m:n • • • X r:m:n ) can be written as where in which empty products are defined as 1.

PROOF. By the Markov chain properties of progressive Type II censored samples, one can write
where f i+1|i:m:n (x i+1 |x i ) is the conditional pdf of X i+1:m:n given X i:m:n = x i , which also is the density of the first order statistic of a sample of size where H i+1|i:m:n is the expected entropy in X i+1:m:n given X i:m:n = x i i.e., By Lemma 4.1.and noting that, condition on X i:m:n = x i , X i+1:m:n has the same pdf as the first order statistic from a random sample of size . Equation ( 27) can be written as where By changing integrals and noting that X i:m:n < X i+1:m:n , we have Therefore Equation (31), can be written as Thus by using Equations ( 26) and (31) H 1....r:m:n can be expressed as a summation of single integral as where c (r) is defined above.From Theorem 1 in Balakrishnan et al. [13], we have the following relation for f s:m:n are defined above.We reexpress Equation (33) as where f 1:R i is the usual smallest order statistics in a sample of size R i .If we use Equations ( 23) and (34) in Equation (32) the result follows.
We have written program in the algebraic manipulation package, MATHEMATICA [14], for computing Theorem 4.1 and Lemma 4.1 calculated above.For a pre-determined progressively Type II censoring scheme (n, m, R 1 , R 2 , • • • R m ) the program return the numerical values of the entropy.The electronic version of the computer program can be obtained by contacting the corresponding author.REMARK 4.1.The entropy of the smallest usual order statistics are known for well-known distributions for example see Park [2] and Asadi et al. [15].

Illustrative Examples
The entropy of the smallest OS h 1:n has the expression [2].
2 , the entropy of the smallest OS h 1:n takes the form where μ (2)  r:n is the second moment of X r:n of the standard normal distribution, see Park [2].We use Theorem 4.1 and Equation (36) to calculate the H 1•••r:m:n given in Table 1.

1 .
The m, and the R i and i = 1 • • • , m − 1 are fixed prior to the test.2.At the first failure, R 1 units are randomly removed from the remaining n − 1 surviving units.3.At the second failure, R 2 units are randomly removed from the remaining n − R 1 − 2 units.4. The test continues until the mth failure, when all remaining

Table 1 .
Table gives the values of the entropy in a collection r of OS progressive Type II censored sample.For r = m, the Table gives the values of entropy in a complete progressive Type II censored sample.The entropy in a collection of order statistics from a progressive Type II censored sample from a normal distribution with unit standard deviation.

Table 2 .
The entropy in a collection of order statistics from a progressive Type II censored sample from logistic distribution.n m Censoring scheme r OS of Proressive samples Entropy , • • • , X 10:10 ) 7.62962