ConRegR: Extrapolative recalibration of the empirical distribution of pvalues to improve false discovery rate estimates
 Juntao Li^{1, 2},
 Puteri Paramita^{1},
 Kwok Pui Choi^{2} and
 R Krishna Murthy Karuturi^{1}Email author
DOI: 10.1186/17456150627
© Li et al; licensee BioMed Central Ltd. 2011
Received: 3 December 2010
Accepted: 20 May 2011
Published: 20 May 2011
Abstract
Background
False discovery rate (FDR) control is commonly accepted as the most appropriate error control in multiple hypothesis testing problems. The accuracy of FDR estimation depends on the accuracy of the estimation of pvalues from each test and validity of the underlying assumptions of the distribution. However, in many practical testing problems such as in genomics, the pvalues could be underestimated or overestimated for many known or unknown reasons. Consequently, FDR estimation would then be influenced and lose its veracity.
Results
We propose a new extrapolative method called Constrained Regression Recalibration (ConRegR) to recalibrate the empirical pvalues by modeling their distribution to improve the FDR estimates. Our ConRegR method is based on the observation that accurately estimated pvalues from true null hypotheses follow uniform distribution and the observed distribution of pvalues is indeed a mixture of distributions of pvalues from true null hypotheses and true alternative hypotheses. Hence, ConRegR recalibrates the observed pvalues so that they exhibit the properties of an ideal empirical pvalue distribution. The proportion of true null hypotheses (π_{0}) and FDR are estimated after the recalibration.
Conclusions
ConRegR provides an efficient way to improve the FDR estimates. It only requires the pvalues from the tests and avoids permutation of the original test data. We demonstrate that the proposed method significantly improves FDR estimation on several gene expression datasets obtained from microarray and RNAseq experiments.
Reviewers
The manuscript was reviewed by Prof. Vladimir Kuznetsov, Prof. Philippe Broet, and Prof. Hongfang Liu (nominated by Prof. Yuriy Gusev).
Background
In highthroughput biological data analysis, multiple hypothesis testing is employed to address certain biological problems. Appropriate tests are chosen for the data, and the pvalues are then computed under some distributional assumptions. Due to the large number of tests performed, error rate controls (which focus on the occurrence of false positives) are commonly used to measure the statistical significance. False discovery rate (FDR) control is accepted as the most appropriate error control. Other useful error rate controls include conditional FDR (cFDR) [1], positive FDR (pFDR) [2] and local FDR (lFDR) [3] which have similar interpretations as that of FDR. However, appropriate FDR estimation depends on the precise pvalues from each test and the validity of the underlying assumptions of the distribution.
where g_{0}(p) = 1 denotes the probability density function of a uniform distribution over (0, 1) and g_{1}(p) ≈ 0 for p close to 1. Therefore, g(p) will be close to a constant (i.e. π_{0}) for p close to 1.
where β is typically chosen to be 0.25, 0.5 or 0.75. These estimates are reasonable under the uniform distribution assumption of a component in this mixture model [6].
However, in many applied testing problems, the pvalues could be underestimated or overestimated for many known or unknown reasons. The violation of pvalue distribution assumptions may lead to inaccurate FDR estimation. There are many factors influencing FDR estimation in the analysis of highthroughput biological data such as microarray and sequencing studies. Dependence among the test statistics is one of the major factors [7, 8]. Usually in microarray data, there are many groups of genes having similar expression patterns and the test statistics (for example, tstatistic) are not independent within one group. The global effects in the array may also influence the dependence in the data. For example, batch and cluster effects [9, 10] always occur in the experiments and sometimes they may be the major cause of incorrectly estimated FDR.
Further, due to the "large p, small n" problem [11] for the gene expression data, some parameters such as mean and variance for each gene cannot be well estimated, or the test assumptions are not satisfied or the distribution of the statistic under null hypotheses may not be accurate. Therefore, many applied testing methods modified the standard testing methods (for example, modifying tstatistic to moderated tstatistic [12]) to increase their usability. As the modified test statistics only approximately follow some known distribution, the approximate pvalue estimation may influence the FDR estimation. Resampling strategies may better estimate the underlying distributions of the test statistics. However, due to small sample size and data correlation, the limited number of permutations and resampling bias [13] also influence the FDR estimation.
To address the above problems, we propose a novel extrapolative recalibration procedure called Constrained Regression Recalibration (ConRegR) which models the empirical distribution of pvalues in multiple hypothesis testing and recalibrates the imprecise pvalue calculation to better approximated pvalues to improve the FDR estimation. Our approach focuses on pvalues as the pvalues from true null hypotheses are expected to follow the uniform distribution and the interference from the distribution of pvalues from alternative hypotheses is expected to be minimal towards p = 1. In contrast, the estimation of the empirical null distributions of test statistics may not be accurate as their parametric form may not be known beforehand and their accuracy may depend on the data and the resampling strategy used. ConRegR first maps the observed pvalues to predefined uniformly distributed pvalues preserving their rank order and estimates the recalibration mapping function by performing constrained polynomial regression to the k highest pvalues. The constrained polynomial regression is implemented by quadratic programming solvers. Finally, the pvalues will be recalibrated using the normalized recalibration function. FDR is estimated using the recalibrated pvalues and the can be determined during ConRegR procedure. We demonstrate that our ConRegR procedure can significantly improve the estimation of FDR on simulated data, and also the environmental stress response time course microarray datasets in yeast and a human RNAseq dataset.
Methods
Under the null hypotheses, the pvalues are uniformly distributed. Hence, ConRegR first generates the uniformly distributed pvalues within [0, 1] range.
Uniformly distributed pvalue generation
Let p_{ i } denotes the pvalue of the i^{ th } test (i = 1, ..., n), without loss of generality, we assume p_{1} ≥ p_{2} ≥ ... ≥ p_{ n }. If we choose a suitable k < n such that the i^{ th } null hypothesis is most likely true, then p_{1}, ..., p_{ k }correspond to the order statistics of k independent uniformly distributed random variables provided p_{ i } 's i(i = 1, ..., k) are correctly estimated.
Therefore, are uniformly distributed over .
By StoneWeierstrass theorem [14], polynomial functions can well approximate any continuous function in the interval [0,1]. Therefore we use polynomial regression to estimate the recalibration function f(·) satisfying appropriate boundary and monotone constraints.
Constrained Regression Recalibration (ConRegR)
The constraints f (0) = 0, f (1) = 1 and f' (x) > 0 should be imposed to ensure the orders of the pvalues remain the same after the transformation. Furthermore, the constraint for either f" (x) > 0 or f" (x) < 0 indicates the function f should also be a monotonic convex or monotonic concave function to deal with the situations with underestimated or overestimated pvalues separately and helps in good extrapolation. The constraints f (0) = 0 and f (1) = 1 can be easily met by scaling and shifting the regression function. Therefore, the regression function only depends on the other two constraints which can be combined into one constraint during the regression procedure.
is a 2l × (t + 1) matrix, where a_{1}, ..., a_{ l } are l randomly generated numbers following U(0, 1) to guarantee
this constraint is valid in (0, 1), and c is chosen to be 0 (or 1) if f is desired to be convex (or concave respectively).
under Aβ ≥ b, where Q = X^{ T } X and q = X^{ T } y. Therefore, the constrained polynomial regression problem can be reformulated as a quadratic programming problem.
Two further modifications
We use QuadProg package in R to solve the quadratic programming problem [16]. Due to floating point errors [17], Q = X^{ T } X tends to be positive semidefinite instead of being positive definite. To get around this, we add a sufficiently small positive value (λ = 10^{10}) to the diagonal of Q to guarantee
Q' = Q + λ I_{t+1 }is positive definite and Q' replaces Q in (10).
Furthermore, the polynomial function may not accurately fit the data due to the limitation of the polynomial maximal power (usually set the maximal power t = 10). We can add the fraction of the power (i.e. a noninteger power) to increase the accuracy of the fit. For example, let , where m = 1, 2 or more.
Computational procedure
 1.
For each (v is the interval over k and default setting is v = [n/100]), let .
 2.
Use equation (5) to compute .
 3.
Use quadratic programming to obtain regression function h_{ k } , where c can be predefined or estimated by checking whether more than half of points for are above the diagonal (line from origin to (1, 1)) (c = 1) or below the diagonal (c = 0).
 4.
Transform h_{ k } to to satisfy constraints f_{ k } (0) = 0, and f_{ k } (1) = 1.
 5.
Repeat steps 24 for all k, and compute the and for each k. Let k_{ best } be the maximal of k which locally minimizes under the constraint of small , where the cutoff of and local minimization criteria should be predefined.
 6.
Choose the final regression function f (.) under k_{ best } and output recalibrated pvalues.
 7.
Reestimate the FDR using recalibrated pvalues and .
Rcode for ConRegR is attached as Additional file 1.
Results
Dependence simulation
where b_{ i } denotes the biological effect, d_{ ij } denotes the dependence effect. Set b_{ i } = 1, if i ≤ n(1  π_{0}) and b_{ i } = 0 if i > n(1  π_{0}). d_{ i } . = (1, 1, 1, 0, 0, 0, 0, 1, 1, 1) if and d_{ i } . = (1, 1, 1, 0, 0, 0, 0, 1, 1, 1) if ). ε_{ ij } ~ N(0, 1) is the background noise.
Combined pvalues simulation
In many analyses, more than one dataset are involved and a metaanalysis by combining pvalues from different studies or datasets is needed to estimate the overall significance for each gene. For example, (i) to find genes which are significant in at least one experiment, minimal pvalues will be of interest; (ii) to identify genes which are significant across all the experiments, the maximal pvalues will be of interest; and (iii) in order to detect genes which are significant on average, the product of pvalues will be appropriate. The distribution of combined pvalues will not be uniform even under true null hypotheses [18]. For currently used metaanalysis methods, such as "minimal", "maximal" or "product", we can obtain the transformation functions to recalibrate the combined pvalues to satisfy the condition of pvalues are uniform distributed under true null hypotheses. However, for other more complicated metaanalysis methods, the transformation function cannot be determined accurately leading to under or overestimation of significance, and ConRegR can provide the polynomial function approximation for the unknown transformation.
Suppose for gene i, the pvalues p_{ ij } (j = 1, 2, ..., L) follow the uniform distribution over (0, 1), then 1  (1  p_{min}) ^{ L } ~ U(0, 1) and , where p_{min} = min(p_{i1}, p_{i2}, ..., p_{ iL }) and p_{max} = max(p_{i1}, p_{i2}, ..., p_{ iL }). For the pvalues from "product" method, according to Fisher's method [19].
where b_{ i } (b_{ i } = 1, if i ≤ n(1  π_{0}) and b_{ i } = 0 if i > n(1  π_{0})) denotes the biological effect, both δ_{ ij ~ } N(0, 1) and ε_{ ij } N(0, 1) are the background noise. The individual pvalues are computed from twosample ttest and the combined pvalues are calculated by L(L = 3) simulations.
Combined pvalues methods [18].
Method  Formula  Transformation 

Min  p_{min} = min(p_{i1}, p_{i2}, ..., p_{ iL })  1(1p_{ min })^{ L } 
Max  p_{man} = max(p_{i1}, p_{i2}, ..., p_{ iL }) 

Square 


Sqroot 


Prod 


Yeast Environmental Response Data
Yeast environmental stress response data generated by [20, 21] for nearly 6000 genes of yeast (S. cerevisiae) was aimed at understanding how yeast adopts or reacts to various stresses present in its environment. We selected 10 datasets: (1) Heat shock from 25°C to 37°C response; (2) Hydrogen peroxide treatment; (3) Menadione exposure; (4) DTT exposure response; (5) Diamide treatment response; (6) Hyperosmotic shock response; (7) Nitrogen source depletion; (8) Diauxic shift study; and, (910) two nearly identical experiments on stationary phase. We used Limma (Linear Models for Microarray Data) [12] package in R to compute pvalues for responsiveness of genes for each dataset.
The pvalue distribution for each dataset is shown in Additional File 2, Figure S4. As can be seen in Figure S4, the majority of the pvalue histograms do not have similar frequency after p = 0.5, and the density near p = 1 is less than π_{0} = 0.5. This implies that the pvalues were underestimated and the number of significantly responsive genes under these environmental stresses should be less than observed. We applied ConRegR on the pvalues of each dataset. Our result shows that the histograms of recalibrated pvalues obtained by applying ConRegR are better than without recalibration, and π_{0} estimations are all above 0.5 (Figure S4).
where (respectively, ) is the estimated FDR by recalibration (respectively, input) pvalues for gene i(i = 1 ... n); and is the true FDR for gene i.
Significance Analysis of Differential Expression from RNAseq Data
The nextgeneration sequencing technologies have been used for gene expression measurement. In [25], the authors compared RNAseq and Affymetrix microarray experiments and claimed that the sequencing data identified many more differentially expressed genes between human kidney and liver tissue samples than microarray data using the same FDR cutoff. In total, 11,493 significant genes were identified by RNAseq (3380 more genes than Affymetrix), only 6534 (56.9%) genes were also identified by Affymetrix experiments. By checking the pvalue histograms for RNAseq dataset, we found that majority of pvalues are very significant and its frequencies are very nonuniform for p > 0.5. However, the pvalue histogram for Affymetrix datasets is close to uniform for p > 0.5 (Additional File 2, Figure S5).
Discussion
ConRegR focuses on the uniformity of pvalues under null hypotheses and uses constrained polynomial regression to recalibrate the empirical pvalue distribution to more welldefined pvalue distribution. Therefore, the FDR estimation can be improved after the recalibration since the assumption of FDR estimation is that the input pvalues should follow such an ideal empirical pvalue distribution under null hypothesis. If the input pvalues follow the properties of ideal empirical pvalues distribution, the regression function tends to be diagonal line (i.e., y = x) and the pvalues do not change considerably after recalibration.
Though our method is discussed in the context of global FDR control, it is equally applicable to the other FDR like controls such as local FDR. Our method does not provide any new FDR control, but inputs better calibrated pvalues to the existing FDR estimators to improve their efficacy.
In ConRegR, the cutoff of and local minimization criteria can be changed for choosing the suitable k after checking the plots of and . From the combined pvalues simulation, the regression function may not fit the data well for the "Square" case. The fractional power, such as 1/2, can be added in the polynomial function to improve the fit.
The assumption that the regression function is convex or concave can be useful to deal with the imprecise pvalues whose distribution is biased towards 1 or 0 respectively. These are the most common cases of the pvalues being underestimated or overestimated. However, in some cases, the pvalues can be underestimated in one range of pvalues and overestimated in the remaining range of pvalues. These pvalue distributions may have peak or valley in the middle of the pvalue range or even have multiple peaks. The regression function will then no longer be convex nor concave. Regression function to handle this situation is currently under study. Our ConRegR can be generalized to an iterative weighted least squares method (e.g. decreasing weights from 1 to n). The weight program in the current version of the ConRegR is assigning a weight of 1 for all pvalues from 1 to k and a weight of 0 for the rest. Furthermore, different optimization schemes also need to be experimented. These will be explored in our future work. The distribution of pvalues from multiple testing can be modeled by the mixture of uniform distribution and some other welldefined distribution such as Beta distribution [5]. The parametric recalibration method is under development. The discrete pvalues from some nonparametric tests cannot be modeled by mixture model and new procedure should be explored to resolve this kind of problem.
Reviewers' comments
Reviewer's report 1
Prof. Vladimir Kuznetsov, Bioinformatics Institute, A*STAR, Singapore
Review
Major: Summary. Mathematical part of the manuscript is major, but it is not convicted. Choosing the model (2) is not justified. The used model assumes a uniform distribution of the tail of FDR distribution is not correct due to experimental data; the most empirical FDR distribution exhibits the nonhomogeneous fat tail as it was showed in Suppl. 1.
Authors' response: Thanks for your comment. The pvalue distribution under true null hypotheses follows the uniform distribution if the data satisfy all the assumptions of the hypothesis testing method [4]. However, the FDR distribution may not have similar property. The model (2) is based on the properties of the pvalue distribution, but not on the FDR distribution. For experimental data, many factors influence those assumptions and the pvalue distribution may not have uniformly distributed tail which is central to the paper. We clarified this in paragraphs 2, 3 & 4 in page 2.
The final FDR distribution of combined pvalue function is not be uniform "even under true null hypotheses", because it is a mixture distribution of the samples from different populations. If the samples are taken from significantly different distributions with different sample size and possible batch effect (not removed), does not allowed you to combine the datasets. If the samples are taken from the same distribution, why should not the pvalue distributions be uniform (assuming that they were uniform in the individual datasets before merging)?
Authors' response : In combined pvalues simulation, we assume pvalues follow the uniform distribution for each dataset under true null hypotheses. For each gene, the overall pvalue was obtained by combining its pvalues across experiments using different combination methods which are typical nonparametric metaanalysis methods. The combined pvalue distributions are not uniform which explains why another distribution is used to derive the overall pvalue [18]. We clarified this in paragraphs 1 and 2 in page 9.
Questionable statements
p.3
"Dependence among the test statistics is one of the major factors". It would be nice if you give examples of the test statistics you are mentioning.
Authors' response : We added the "tstatistic" as an example in text (paragraph 1 in page 3). "large p, small n problem". Could you describe the meaning this problem?
Authors' response : "large p, small n" problem is the problem that the number of variables (p) is much bigger than sample size (n) which always occurs in gene expression studies. We added one reference [11]for this problem (paragraph 2 in page 3). The n and p used in this context are different from that used in the discussion throughout the paper.
"ConRegR first maps the observed pvalues to uniformly distributed pvalues". From the context of this paragraph it is unclear where the set of 'uniformly distributed pvalues' is obtained from.
Authors' response : We changed to "predefined uniformly distributed pvalues" (paragraph 3 in page 3).
p.4
"Let pi' denote the ideal pvalues". How did you decide that the ideal distribution of the pvalues is a linear function, as defined in eq2 and is data and testindependent?
Authors' response: is the "ideal" pvalues. By our definition in equation 2 , their distribution is exactly uniform distribution. The "ideal" pvalues are predefined by equation 5 and not related to any data or test. We clarified this in paragraph 3 in page 4.
"the recalibration functions f() between p':(1..k) and p:(1..k) and apply it to all input pvalues (i.e p:(1..n))". However, you mentioned before (p.3) that the distribution of p:(1..n) is a mixture of functions. Hence, how technically valid is mapping the whole set p:(1..n) with a function empirically estimated on a limited subset p:(1..k) ? How do you ensure that the parameters of mapping function f() estimated on the subset (1..k) are also valid in the subset (k+1..n) ?
Authors' response : This is the reason that we emphasized throughout the manuscript that our procedure is extrapolative in nature. The constraints and the choice of k in our procedure help the extrapolation reasonable as shown in our simulations and real data. This is discussed in the methods section. However, we cannot guarantee that the mapping function is 100% valid for all the data.
"By StoneWeirstrass theorem, polynomial functions can well approximate (a small typo here) any continuous functions". Before you defined p:(1..n) as a discrete set. How do you ensure that the function defined on this set continuous? Why quadratic programming is chosen for polynomial approximation, but not linear? Why not other function, e.g. cubic splines, sum of exponents with least squares fitting?
Authors' response : Since pvalues are continuous, the mapping function from pvalues to pvalues should be continuous function. The discrete set p(1..n) is only a sample set from pvalues. We used quadratic programming because the constraints are needed in approximation. We clarified it on last three paragraphs in page 4. We agree that different optimization procedures also can be chosen. But they are subject of our future study.
p.5
"where ai,(i = 1,...,l) are l randomly generated numbers following U(0,1)". Why do they need to be generated randomly from U(0,1)? Since eq9 and eq10 are solved for beta under A * beta >= b, the solution beta depends on the random constrains A. "we add a sufficiently small positive value (delta(Q) = 1010) to the diagonal of Q". Since Q = XXT, this is equivalent to delta(X) = sqrt(delta(Q)) = 10^{5}.
Authors' response : Since the range of × is (x_{ k }, 1), we generated a_{ i } from U(0,1) to guarantee the constraint A * beta >= b is valid in (0,1). The points of constraints can also be chosen uniformly throughout the range [0,1]. As we chose 10000 points for constraints, the uniform selection or random selection may not make significant difference. We clarified it on paragraph 3 in page 5. The small positive value δ = 10^{ 10 } is not the determinant value of Q. we changed the notation to λ (paragraph 2 in page 6).
p.6
"the polynomial function may not accurately fit the data due to the limitation of the polynomial maximal power (usually set the maximal power t = 10)". How do you ensure that your approximating polynomial is not overparameterized?
Authors' response : Since we approximate the polynomial function with monotonic convex/concave constraints, the approximation is not overparameterized. In Additional File 2, figures S2 and S3, the quadratic and cubic polynomial functions were well fitted by a polynomial of t = 10.
"estimated pi0(k) may oscillate near the real pi0". What is the source of the oscillations? Are they periodic?
Authors' response : The oscillation is due to the extrapolation may be unreliable if less number of pvalue are used for regression. It may not be periodic. To better reflect it, we changed the word to "fluctuate". We clarified it on paragraph 4 in page 6.
"the k closest to 1 that gives stable estimate". What is the definition of stability in this context? What numerical criterion do you use to detect the stability?
Authors' response : We used error to define the stability of the estimation, smaller error implies more stable estimate. In results, we used error < 0.05 to detect the stability. We clarified it on paragraph 1 in page 7 and paragraph 2 in page 8.
p.7
"As can be seen in the panels B and D in Figure 2". On Figure 2 the panels are not named. On Figure 3 all the axes need to be labeled and legends for the blue and the red lines should be added. Overall, the description of the results presented on Figure 3 is not comprehensible.
Authors' response : We changed "panels B and D" to "plots B and D" (paragraph 3 in page 7) and updated in Figure 2. We added the labels and legends for Figure 3.
p.8
"Combined pvalues simulation". The whole subsection needs to be described in the "Methods" section, since it introduces novel numerical techniques. In this subsection when you state "FDR improved by XX%", it could be good to show the particular FDR values before the improvement and after the improvement as well. The pvalue distributions for the sets of experimental data would better be shown in the paper, rather than in supplementary materials.
Authors' response : The methods section described the ConRegR method and the combined pvalue simulation subsection shows the simulation result by applying ConRegR. So we think it is suitable in the result section. The particular FDR values before and after applying ConRegR have large numbers and it is not possible to summarize by one table. So we used improvement percentage to show the efficacy of ConRegR. The Figure S1 now becomes Figure 5in the paper.
"In many analyses ...the product of pvalues will be appropriate". Where do these examples come from? Especially, since p1 * p2 < min(p1, p2), it's not logical that the pvalue of significance across, at least, half of the experiments is lower than the pvalue in all the experiments.
Authors' response : The product of pvalues is based on Fisher's method for combining independent tests of significance (reference [19]). The combined pvalues for product and minimal method are not comparable before transformation.
"The distribution of combined pvalues will not be uniform even under true null hypotheses". If the samples are taken from significantly different distributions (i.e the batch effect was not removed), what allowed you to combine the datasets? If the samples are taken from the same distribution, why should not the pvalue distributions be uniform (assuming that they were uniform in the individual datasets before merging)?
Authors' response : Please see our response to the 2nd comment.
Reviewer's report 2
Prof. Philippe Broet, JE2492, University ParisSud, France
This reviewer provided no comments for publication.
Reviewer's report 3
Prof. Hongfang Liu, Department of Biostatistics, Bioinformatics, and Biomathematics, Georgetown University Medical Center, NW, USA
This reviewer provided no comments for publication.
Declarations
Acknowledgements
The authors thank Edison T. Liu for his constant encouragement and support during this work. We appreciate Huaien Luo and Cyril Dalmasso for their valuable comments. We thank the reviewers Prof. Vladimir Kuznetsov, Prof. Philippe Broet and Prof. Hongfang Liu for their valuable comments which improve the manuscript. This work was supported by the Genome Institute of Singapore, Biomedical Research Council, Agency for Science, Technology and Research (A*STAR), Singapore.
Authors’ Affiliations
References
 Tsai C, Hsueh H, Chen JJ: Estimation of false discovery rates in multiple testing: application to gene microarray data. Biometrics. 2003, 59: 10711081. 10.1111/j.0006341X.2003.00123.x.PubMedView ArticleGoogle Scholar
 Storey JD: A direct approach to false discovery rates. Journal of The Royal Statistical Society Series B. 2002, 64: 479498. 10.1111/14679868.00346.View ArticleGoogle Scholar
 Efron B, Tibshirani R, Storey JD, Tusher V: Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association. 2001, 96 (456): 11511160. 10.1198/016214501753382129.View ArticleGoogle Scholar
 Lehmann E, Romano JP: pvalues. Testing Statistical Hypotheses. 2005, New York: Springer, 6365. 3Google Scholar
 Pounds S, Morris SW: Estimating the occurrence of false positives and false negatives in microarray studies by approximating and partitioning the empirical distribution of pvalues. Bioinformatics. 2003, 19: 12361242. 10.1093/bioinformatics/btg148.PubMedView ArticleGoogle Scholar
 Pawitan Y, Karuturi RKM, Michiels S, Ploner A: Bias in the estimation of false discovery rate in microarray studies. Bioinformatics. 2005, 21 (20): 38653872. 10.1093/bioinformatics/bti626.PubMedView ArticleGoogle Scholar
 Efron B: Correlation and LargeScale Simultaneous Significance Testing. Journal of the American Statistical Association. 2007, 102 (477): 93103. 10.1198/016214506000001211.View ArticleGoogle Scholar
 Qiu X, Klebanov L, Yakovlev A: Correlation Between Gene Expression Levels and Limitations of the Empirical Bayes Methodology for Finding Differentially Expressed Genes. Statistical Applications in Genetics and Molecular Biology. 2005, 4:Google Scholar
 Johnson WE, Li C, Rabinovic A: Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 2007, 8: 118127.PubMedView ArticleGoogle Scholar
 Li J, Liu J, Karuturi RKM: Stepped linear regression to accurately assess statistical significance in batch confounded differential expression analysis. Bioinformatics Research and Applications. 2008, 481491.View ArticleGoogle Scholar
 Ochs MF: Knowledgebased data analysis comes of age. Briefings in Bioinformatics. 2010, 11: 3039. 10.1093/bib/bbp044.PubMedPubMed CentralView ArticleGoogle Scholar
 Smyth GK: Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Statistical Applications in Genetics and Molecular Biology. 2004, 3:Google Scholar
 Efron B, Tibshirani R: On testing the significance of sets of genes. The Annals of Applied Statistics. 2007, 1: 107129. 10.1214/07AOAS101.View ArticleGoogle Scholar
 Bishop E: A generalization of the StoneWeierstrass theorem. Pacific Journal of Mathematics. 1961, 11 (3): 777783.View ArticleGoogle Scholar
 Nocedal J, Wright S: Numerical Optimization. 2000, SpringerGoogle Scholar
 Goldfarb D, Idnani A: A numerically stable dual method for solving strictly convex quadratic programs. Mathematical Programming. 1983, 27: 133. 10.1007/BF02591962.View ArticleGoogle Scholar
 Press WH, Teukolsky SA, Vetterling WT, Flannery BP: Numerical Recipes 3rd Edition: The Art of Scientific Computing. 2007, Cambridge University Press, 3Google Scholar
 Hedges LV, Olkin I: Test of Statistical Significance of Combined Results. Statistical methods for metaanalysis. 1985, Academic Press, 2846. 6Google Scholar
 Fisher RA: Combining independent tests of significance. American Statistician. 1948, 2 (5): 3010.2307/2681650.Google Scholar
 DeRisi JL, Iyer VR, Brown PO: Exploring the metabolic and genetic control of gene expression on a genomic scale. Science. 1997, 278 (5338): 680686. 10.1126/science.278.5338.680.PubMedView ArticleGoogle Scholar
 Gasch AP, Spellman PT, Kao CM, CarmelHarel O, Eisen MB, Storz G, Botstein D, Brown PO: Genomic expression programs in the response of yeast cells to environmental changes. Mol Biol Cell. 2000, 11 (12): 42414257.PubMedPubMed CentralView ArticleGoogle Scholar
 Chen D, Toone WM, Mata J, Lyne R, Burns G, Kivinen K, Brazma A, Jones N, Bähler J: Global transcriptional responses of fission yeast to environmental stress. Molecular Biology of the Cell. 2003, 14: 214229. 10.1091/mbc.E02080499.PubMedPubMed CentralView ArticleGoogle Scholar
 Han X, Sung W, Feng L: Identifying differentially expressed genes in timecourse microarray experiment without replicate. Journal of Bioinformatics and Computational Biology. 2007, 05 (02a): 28110.1142/S0219720007002655.View ArticleGoogle Scholar
 Li J, Liu J, Karuturi R: Datadriven smoothness enhanced variance ratio rest to unearth responsive genes in 0time normalized timecourse microarray data. Bioinformatics Research and Applications. 2007, 2536.View ArticleGoogle Scholar
 Marioni JC, Mason CE, Mane SM, Stephens M, Gilad Y: RNAseq: An assessment of technical reproducibility and comparison with gene expression arrays. Genome Research. 2008, 18 (9): 15091517. 10.1101/gr.079558.108.PubMedPubMed CentralView ArticleGoogle Scholar
Copyright
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.