ConReg-R: Extrapolative recalibration of the empirical distribution of p-values to improve false discovery rate estimates
© Li et al; licensee BioMed Central Ltd. 2011
Received: 3 December 2010
Accepted: 20 May 2011
Published: 20 May 2011
False discovery rate (FDR) control is commonly accepted as the most appropriate error control in multiple hypothesis testing problems. The accuracy of FDR estimation depends on the accuracy of the estimation of p-values from each test and validity of the underlying assumptions of the distribution. However, in many practical testing problems such as in genomics, the p-values could be under-estimated or over-estimated for many known or unknown reasons. Consequently, FDR estimation would then be influenced and lose its veracity.
We propose a new extrapolative method called Constrained Regression Recalibration (ConReg-R) to recalibrate the empirical p-values by modeling their distribution to improve the FDR estimates. Our ConReg-R method is based on the observation that accurately estimated p-values from true null hypotheses follow uniform distribution and the observed distribution of p-values is indeed a mixture of distributions of p-values from true null hypotheses and true alternative hypotheses. Hence, ConReg-R recalibrates the observed p-values so that they exhibit the properties of an ideal empirical p-value distribution. The proportion of true null hypotheses (π0) and FDR are estimated after the recalibration.
ConReg-R provides an efficient way to improve the FDR estimates. It only requires the p-values from the tests and avoids permutation of the original test data. We demonstrate that the proposed method significantly improves FDR estimation on several gene expression datasets obtained from microarray and RNA-seq experiments.
The manuscript was reviewed by Prof. Vladimir Kuznetsov, Prof. Philippe Broet, and Prof. Hongfang Liu (nominated by Prof. Yuriy Gusev).
In high-throughput biological data analysis, multiple hypothesis testing is employed to address certain biological problems. Appropriate tests are chosen for the data, and the p-values are then computed under some distributional assumptions. Due to the large number of tests performed, error rate controls (which focus on the occurrence of false positives) are commonly used to measure the statistical significance. False discovery rate (FDR) control is accepted as the most appropriate error control. Other useful error rate controls include conditional FDR (cFDR) , positive FDR (pFDR)  and local FDR (lFDR)  which have similar interpretations as that of FDR. However, appropriate FDR estimation depends on the precise p-values from each test and the validity of the underlying assumptions of the distribution.
where g0(p) = 1 denotes the probability density function of a uniform distribution over (0, 1) and g1(p) ≈ 0 for p close to 1. Therefore, g(p) will be close to a constant (i.e. π0) for p close to 1.
where β is typically chosen to be 0.25, 0.5 or 0.75. These estimates are reasonable under the uniform distribution assumption of a component in this mixture model .
However, in many applied testing problems, the p-values could be under-estimated or over-estimated for many known or unknown reasons. The violation of p-value distribution assumptions may lead to inaccurate FDR estimation. There are many factors influencing FDR estimation in the analysis of high-throughput biological data such as microarray and sequencing studies. Dependence among the test statistics is one of the major factors [7, 8]. Usually in microarray data, there are many groups of genes having similar expression patterns and the test statistics (for example, t-statistic) are not independent within one group. The global effects in the array may also influence the dependence in the data. For example, batch and cluster effects [9, 10] always occur in the experiments and sometimes they may be the major cause of incorrectly estimated FDR.
Further, due to the "large p, small n" problem  for the gene expression data, some parameters such as mean and variance for each gene cannot be well estimated, or the test assumptions are not satisfied or the distribution of the statistic under null hypotheses may not be accurate. Therefore, many applied testing methods modified the standard testing methods (for example, modifying t-statistic to moderated t-statistic ) to increase their usability. As the modified test statistics only approximately follow some known distribution, the approximate p-value estimation may influence the FDR estimation. Resampling strategies may better estimate the underlying distributions of the test statistics. However, due to small sample size and data correlation, the limited number of permutations and resampling bias  also influence the FDR estimation.
To address the above problems, we propose a novel extrapolative recalibration procedure called Constrained Regression Recalibration (ConReg-R) which models the empirical distribution of p-values in multiple hypothesis testing and recalibrates the imprecise p-value calculation to better approximated p-values to improve the FDR estimation. Our approach focuses on p-values as the p-values from true null hypotheses are expected to follow the uniform distribution and the interference from the distribution of p-values from alternative hypotheses is expected to be minimal towards p = 1. In contrast, the estimation of the empirical null distributions of test statistics may not be accurate as their parametric form may not be known beforehand and their accuracy may depend on the data and the resampling strategy used. ConReg-R first maps the observed p-values to predefined uniformly distributed p-values preserving their rank order and estimates the recalibration mapping function by performing constrained polynomial regression to the k highest p-values. The constrained polynomial regression is implemented by quadratic programming solvers. Finally, the p-values will be recalibrated using the normalized recalibration function. FDR is estimated using the recalibrated p-values and the can be determined during ConReg-R procedure. We demonstrate that our ConReg-R procedure can significantly improve the estimation of FDR on simulated data, and also the environmental stress response time course microarray datasets in yeast and a human RNA-seq dataset.
Under the null hypotheses, the p-values are uniformly distributed. Hence, ConReg-R first generates the uniformly distributed p-values within [0, 1] range.
Uniformly distributed p-value generation
Let p i denotes the p-value of the i th test (i = 1, ..., n), without loss of generality, we assume p1 ≥ p2 ≥ ... ≥ p n . If we choose a suitable k < n such that the i th null hypothesis is most likely true, then p1, ..., p k correspond to the order statistics of k independent uniformly distributed random variables provided p i 's i(i = 1, ..., k) are correctly estimated.
Therefore, are uniformly distributed over .
By Stone-Weierstrass theorem , polynomial functions can well approximate any continuous function in the interval [0,1]. Therefore we use polynomial regression to estimate the recalibration function f(·) satisfying appropriate boundary and monotone constraints.
Constrained Regression Recalibration (ConReg-R)
The constraints f (0) = 0, f (1) = 1 and f' (x) > 0 should be imposed to ensure the orders of the p-values remain the same after the transformation. Furthermore, the constraint for either f" (x) > 0 or f" (x) < 0 indicates the function f should also be a monotonic convex or monotonic concave function to deal with the situations with under-estimated or over-estimated p-values separately and helps in good extrapolation. The constraints f (0) = 0 and f (1) = 1 can be easily met by scaling and shifting the regression function. Therefore, the regression function only depends on the other two constraints which can be combined into one constraint during the regression procedure.
is a 2l × (t + 1) matrix, where a1, ..., a l are l randomly generated numbers following U(0, 1) to guarantee
this constraint is valid in (0, 1), and c is chosen to be 0 (or 1) if f is desired to be convex (or concave respectively).
under Aβ ≥ b, where Q = X T X and q = -X T y. Therefore, the constrained polynomial regression problem can be reformulated as a quadratic programming problem.
Two further modifications
We use QuadProg package in R to solve the quadratic programming problem . Due to floating point errors , Q = X T X tends to be positive semidefinite instead of being positive definite. To get around this, we add a sufficiently small positive value (λ = 10-10) to the diagonal of Q to guarantee
Q' = Q + λ It+1 is positive definite and Q' replaces Q in (10).
Furthermore, the polynomial function may not accurately fit the data due to the limitation of the polynomial maximal power (usually set the maximal power t = 10). We can add the fraction of the power (i.e. a non-integer power) to increase the accuracy of the fit. For example, let , where m = 1, 2 or more.
For each (v is the interval over k and default setting is v = [n/100]), let .
Use equation (5) to compute .
Use quadratic programming to obtain regression function h k , where c can be predefined or estimated by checking whether more than half of points for are above the diagonal (line from origin to (1, 1)) (c = 1) or below the diagonal (c = 0).
Transform h k to to satisfy constraints f k (0) = 0, and f k (1) = 1.
Repeat steps 2-4 for all k, and compute the and for each k. Let k best be the maximal of k which locally minimizes under the constraint of small , where the cutoff of and local minimization criteria should be predefined.
Choose the final regression function f (.) under k best and output recalibrated p-values.
Re-estimate the FDR using recalibrated p-values and .
R-code for ConReg-R is attached as Additional file 1.
where b i denotes the biological effect, d ij denotes the dependence effect. Set b i = 1, if i ≤ n(1 - π0) and b i = 0 if i > n(1 - π0). d i . = (1, 1, 1, 0, 0, 0, 0, -1, -1, -1) if and d i . = (-1, -1, -1, 0, 0, 0, 0, 1, 1, 1) if ). ε ij ~ N(0, 1) is the background noise.
Combined p-values simulation
In many analyses, more than one dataset are involved and a meta-analysis by combining p-values from different studies or datasets is needed to estimate the overall significance for each gene. For example, (i) to find genes which are significant in at least one experiment, minimal p-values will be of interest; (ii) to identify genes which are significant across all the experiments, the maximal p-values will be of interest; and (iii) in order to detect genes which are significant on average, the product of p-values will be appropriate. The distribution of combined p-values will not be uniform even under true null hypotheses . For currently used meta-analysis methods, such as "minimal", "maximal" or "product", we can obtain the transformation functions to recalibrate the combined p-values to satisfy the condition of p-values are uniform distributed under true null hypotheses. However, for other more complicated meta-analysis methods, the transformation function cannot be determined accurately leading to under- or over-estimation of significance, and ConReg-R can provide the polynomial function approximation for the unknown transformation.
Suppose for gene i, the p-values p ij (j = 1, 2, ..., L) follow the uniform distribution over (0, 1), then 1 - (1 - pmin) L ~ U(0, 1) and , where pmin = min(pi1, pi2, ..., p iL ) and pmax = max(pi1, pi2, ..., p iL ). For the p-values from "product" method, according to Fisher's method .
where b i (b i = 1, if i ≤ n(1 - π0) and b i = 0 if i > n(1 - π0)) denotes the biological effect, both δ ij ~ N(0, 1) and ε ij N(0, 1) are the background noise. The individual p-values are computed from two-sample t-test and the combined p-values are calculated by L(L = 3) simulations.
Combined p-values methods .
pmin = min(pi1, pi2, ..., p iL )
1-(1-p min ) L
pman = max(pi1, pi2, ..., p iL )
Yeast Environmental Response Data
Yeast environmental stress response data generated by [20, 21] for nearly 6000 genes of yeast (S. cerevisiae) was aimed at understanding how yeast adopts or reacts to various stresses present in its environment. We selected 10 datasets: (1) Heat shock from 25°C to 37°C response; (2) Hydrogen peroxide treatment; (3) Menadione exposure; (4) DTT exposure response; (5) Diamide treatment response; (6) Hyper-osmotic shock response; (7) Nitrogen source depletion; (8) Diauxic shift study; and, (9-10) two nearly identical experiments on stationary phase. We used Limma (Linear Models for Microarray Data)  package in R to compute p-values for responsiveness of genes for each dataset.
The p-value distribution for each dataset is shown in Additional File 2, Figure S4. As can be seen in Figure S4, the majority of the p-value histograms do not have similar frequency after p = 0.5, and the density near p = 1 is less than π0 = 0.5. This implies that the p-values were under-estimated and the number of significantly responsive genes under these environmental stresses should be less than observed. We applied ConReg-R on the p-values of each dataset. Our result shows that the histograms of recalibrated p-values obtained by applying ConReg-R are better than without recalibration, and π0 estimations are all above 0.5 (Figure S4).
where (respectively, ) is the estimated FDR by recalibration (respectively, input) p-values for gene i(i = 1 ... n); and is the true FDR for gene i.
Significance Analysis of Differential Expression from RNA-seq Data
The next-generation sequencing technologies have been used for gene expression measurement. In , the authors compared RNA-seq and Affymetrix microarray experiments and claimed that the sequencing data identified many more differentially expressed genes between human kidney and liver tissue samples than microarray data using the same FDR cutoff. In total, 11,493 significant genes were identified by RNA-seq (3380 more genes than Affymetrix), only 6534 (56.9%) genes were also identified by Affymetrix experiments. By checking the p-value histograms for RNA-seq dataset, we found that majority of p-values are very significant and its frequencies are very non-uniform for p > 0.5. However, the p-value histogram for Affymetrix datasets is close to uniform for p > 0.5 (Additional File 2, Figure S5).
ConReg-R focuses on the uniformity of p-values under null hypotheses and uses constrained polynomial regression to recalibrate the empirical p-value distribution to more well-defined p-value distribution. Therefore, the FDR estimation can be improved after the recalibration since the assumption of FDR estimation is that the input p-values should follow such an ideal empirical p-value distribution under null hypothesis. If the input p-values follow the properties of ideal empirical p-values distribution, the regression function tends to be diagonal line (i.e., y = x) and the p-values do not change considerably after recalibration.
Though our method is discussed in the context of global FDR control, it is equally applicable to the other FDR like controls such as local FDR. Our method does not provide any new FDR control, but inputs better calibrated p-values to the existing FDR estimators to improve their efficacy.
In ConReg-R, the cutoff of and local minimization criteria can be changed for choosing the suitable k after checking the plots of and . From the combined p-values simulation, the regression function may not fit the data well for the "Square" case. The fractional power, such as 1/2, can be added in the polynomial function to improve the fit.
The assumption that the regression function is convex or concave can be useful to deal with the imprecise p-values whose distribution is biased towards 1 or 0 respectively. These are the most common cases of the p-values being under-estimated or over-estimated. However, in some cases, the p-values can be under-estimated in one range of p-values and over-estimated in the remaining range of p-values. These p-value distributions may have peak or valley in the middle of the p-value range or even have multiple peaks. The regression function will then no longer be convex nor concave. Regression function to handle this situation is currently under study. Our ConReg-R can be generalized to an iterative weighted least squares method (e.g. decreasing weights from 1 to n). The weight program in the current version of the ConReg-R is assigning a weight of 1 for all p-values from 1 to k and a weight of 0 for the rest. Furthermore, different optimization schemes also need to be experimented. These will be explored in our future work. The distribution of p-values from multiple testing can be modeled by the mixture of uniform distribution and some other well-defined distribution such as Beta distribution . The parametric recalibration method is under development. The discrete p-values from some nonparametric tests cannot be modeled by mixture model and new procedure should be explored to resolve this kind of problem.
Reviewer's report 1
Prof. Vladimir Kuznetsov, Bioinformatics Institute, A*STAR, Singapore
Major: Summary. Mathematical part of the manuscript is major, but it is not convicted. Choosing the model (2) is not justified. The used model assumes a uniform distribution of the tail of FDR distribution is not correct due to experimental data; the most empirical FDR distribution exhibits the non-homogeneous fat tail as it was showed in Suppl. 1.
Authors' response: Thanks for your comment. The p-value distribution under true null hypotheses follows the uniform distribution if the data satisfy all the assumptions of the hypothesis testing method . However, the FDR distribution may not have similar property. The model (2) is based on the properties of the p-value distribution, but not on the FDR distribution. For experimental data, many factors influence those assumptions and the p-value distribution may not have uniformly distributed tail which is central to the paper. We clarified this in paragraphs 2, 3 & 4 in page 2.
The final FDR distribution of combined p-value function is not be uniform "even under true null hypotheses", because it is a mixture distribution of the samples from different populations. If the samples are taken from significantly different distributions with different sample size and possible batch effect (not removed), does not allowed you to combine the datasets. If the samples are taken from the same distribution, why should not the p-value distributions be uniform (assuming that they were uniform in the individual datasets before merging)?
Authors' response : In combined p-values simulation, we assume p-values follow the uniform distribution for each dataset under true null hypotheses. For each gene, the overall p-value was obtained by combining its p-values across experiments using different combination methods which are typical non-parametric meta-analysis methods. The combined p-value distributions are not uniform which explains why another distribution is used to derive the overall p-value . We clarified this in paragraphs 1 and 2 in page 9.
"Dependence among the test statistics is one of the major factors". It would be nice if you give examples of the test statistics you are mentioning.
Authors' response : We added the "t-statistic" as an example in text (paragraph 1 in page 3). "large p, small n problem". Could you describe the meaning this problem?
Authors' response : "large p, small n" problem is the problem that the number of variables (p) is much bigger than sample size (n) which always occurs in gene expression studies. We added one reference for this problem (paragraph 2 in page 3). The n and p used in this context are different from that used in the discussion throughout the paper.
"ConReg-R first maps the observed p-values to uniformly distributed p-values". From the context of this paragraph it is unclear where the set of 'uniformly distributed p-values' is obtained from.
Authors' response : We changed to "predefined uniformly distributed p-values" (paragraph 3 in page 3).
"Let pi' denote the ideal p-values". How did you decide that the ideal distribution of the p-values is a linear function, as defined in eq2 and is data- and test-independent?
Authors' response: is the "ideal" p-values. By our definition in equation 2 , their distribution is exactly uniform distribution. The "ideal" p-values are predefined by equation 5 and not related to any data or test. We clarified this in paragraph 3 in page 4.
"the recalibration functions f() between p':(1..k) and p:(1..k) and apply it to all input p-values (i.e p:(1..n))". However, you mentioned before (p.3) that the distribution of p:(1..n) is a mixture of functions. Hence, how technically valid is mapping the whole set p:(1..n) with a function empirically estimated on a limited subset p:(1..k) ? How do you ensure that the parameters of mapping function f() estimated on the subset (1..k) are also valid in the subset (k+1..n) ?
Authors' response : This is the reason that we emphasized throughout the manuscript that our procedure is extrapolative in nature. The constraints and the choice of k in our procedure help the extrapolation reasonable as shown in our simulations and real data. This is discussed in the methods section. However, we cannot guarantee that the mapping function is 100% valid for all the data.
"By Stone-Weirstrass theorem, polynomial functions can well approximate (a small typo here) any continuous functions". Before you defined p:(1..n) as a discrete set. How do you ensure that the function defined on this set continuous? Why quadratic programming is chosen for polynomial approximation, but not linear? Why not other function, e.g. cubic splines, sum of exponents with least squares fitting?
Authors' response : Since p-values are continuous, the mapping function from p-values to p-values should be continuous function. The discrete set p(1..n) is only a sample set from p-values. We used quadratic programming because the constraints are needed in approximation. We clarified it on last three paragraphs in page 4. We agree that different optimization procedures also can be chosen. But they are subject of our future study.
"where ai,(i = 1,...,l) are l randomly generated numbers following U(0,1)". Why do they need to be generated randomly from U(0,1)? Since eq9 and eq10 are solved for beta under A * beta >= b, the solution beta depends on the random constrains A. "we add a sufficiently small positive value (delta(Q) = 10-10) to the diagonal of Q". Since Q = XXT, this is equivalent to delta(X) = sqrt(delta(Q)) = 10-5.
Authors' response : Since the range of × is (x k , 1), we generated a i from U(0,1) to guarantee the constraint A * beta >= b is valid in (0,1). The points of constraints can also be chosen uniformly throughout the range [0,1]. As we chose 10000 points for constraints, the uniform selection or random selection may not make significant difference. We clarified it on paragraph 3 in page 5. The small positive value δ = 10 -10 is not the determinant value of Q. we changed the notation to λ (paragraph 2 in page 6).
"the polynomial function may not accurately fit the data due to the limitation of the polynomial maximal power (usually set the maximal power t = 10)". How do you ensure that your approximating polynomial is not over-parameterized?
Authors' response : Since we approximate the polynomial function with monotonic convex/concave constraints, the approximation is not over-parameterized. In Additional File 2, figures S2 and S3, the quadratic and cubic polynomial functions were well fitted by a polynomial of t = 10.
"estimated pi0(k) may oscillate near the real pi0". What is the source of the oscillations? Are they periodic?
Authors' response : The oscillation is due to the extrapolation may be unreliable if less number of p-value are used for regression. It may not be periodic. To better reflect it, we changed the word to "fluctuate". We clarified it on paragraph 4 in page 6.
"the k closest to 1 that gives stable estimate". What is the definition of stability in this context? What numerical criterion do you use to detect the stability?
Authors' response : We used error to define the stability of the estimation, smaller error implies more stable estimate. In results, we used error < 0.05 to detect the stability. We clarified it on paragraph 1 in page 7 and paragraph 2 in page 8.
"As can be seen in the panels B and D in Figure 2". On Figure 2 the panels are not named. On Figure 3 all the axes need to be labeled and legends for the blue and the red lines should be added. Overall, the description of the results presented on Figure 3 is not comprehensible.
"Combined p-values simulation". The whole subsection needs to be described in the "Methods" section, since it introduces novel numerical techniques. In this subsection when you state "FDR improved by XX%", it could be good to show the particular FDR values before the improvement and after the improvement as well. The p-value distributions for the sets of experimental data would better be shown in the paper, rather than in supplementary materials.
Authors' response : The methods section described the ConReg-R method and the combined p-value simulation subsection shows the simulation result by applying ConReg-R. So we think it is suitable in the result section. The particular FDR values before and after applying ConReg-R have large numbers and it is not possible to summarize by one table. So we used improvement percentage to show the efficacy of ConReg-R. The Figure S1 now becomes Figure 5in the paper.
"In many analyses ...the product of p-values will be appropriate". Where do these examples come from? Especially, since p1 * p2 < min(p1, p2), it's not logical that the p-value of significance across, at least, half of the experiments is lower than the p-value in all the experiments.
Authors' response : The product of p-values is based on Fisher's method for combining independent tests of significance (reference ). The combined p-values for product and minimal method are not comparable before transformation.
"The distribution of combined p-values will not be uniform even under true null hypotheses". If the samples are taken from significantly different distributions (i.e the batch effect was not removed), what allowed you to combine the datasets? If the samples are taken from the same distribution, why should not the p-value distributions be uniform (assuming that they were uniform in the individual datasets before merging)?
Authors' response : Please see our response to the 2nd comment.
Reviewer's report 2
Prof. Philippe Broet, JE2492, University Paris-Sud, France
This reviewer provided no comments for publication.
Reviewer's report 3
Prof. Hongfang Liu, Department of Biostatistics, Bioinformatics, and Biomathematics, Georgetown University Medical Center, NW, USA
This reviewer provided no comments for publication.
The authors thank Edison T. Liu for his constant encouragement and support during this work. We appreciate Huaien Luo and Cyril Dalmasso for their valuable comments. We thank the reviewers Prof. Vladimir Kuznetsov, Prof. Philippe Broet and Prof. Hongfang Liu for their valuable comments which improve the manuscript. This work was supported by the Genome Institute of Singapore, Biomedical Research Council, Agency for Science, Technology and Research (A*STAR), Singapore.
- Tsai C, Hsueh H, Chen JJ: Estimation of false discovery rates in multiple testing: application to gene microarray data. Biometrics. 2003, 59: 1071-1081. 10.1111/j.0006-341X.2003.00123.x.PubMedView ArticleGoogle Scholar
- Storey JD: A direct approach to false discovery rates. Journal of The Royal Statistical Society Series B. 2002, 64: 479-498. 10.1111/1467-9868.00346.View ArticleGoogle Scholar
- Efron B, Tibshirani R, Storey JD, Tusher V: Empirical Bayes analysis of a microarray experiment. Journal of the American Statistical Association. 2001, 96 (456): 1151-1160. 10.1198/016214501753382129.View ArticleGoogle Scholar
- Lehmann E, Romano JP: p-values. Testing Statistical Hypotheses. 2005, New York: Springer, 63-65. 3Google Scholar
- Pounds S, Morris SW: Estimating the occurrence of false positives and false negatives in microarray studies by approximating and partitioning the empirical distribution of p-values. Bioinformatics. 2003, 19: 1236-1242. 10.1093/bioinformatics/btg148.PubMedView ArticleGoogle Scholar
- Pawitan Y, Karuturi RKM, Michiels S, Ploner A: Bias in the estimation of false discovery rate in microarray studies. Bioinformatics. 2005, 21 (20): 3865-3872. 10.1093/bioinformatics/bti626.PubMedView ArticleGoogle Scholar
- Efron B: Correlation and Large-Scale Simultaneous Significance Testing. Journal of the American Statistical Association. 2007, 102 (477): 93-103. 10.1198/016214506000001211.View ArticleGoogle Scholar
- Qiu X, Klebanov L, Yakovlev A: Correlation Between Gene Expression Levels and Limitations of the Empirical Bayes Methodology for Finding Differentially Expressed Genes. Statistical Applications in Genetics and Molecular Biology. 2005, 4:Google Scholar
- Johnson WE, Li C, Rabinovic A: Adjusting batch effects in microarray expression data using empirical Bayes methods. Biostatistics. 2007, 8: 118-127.PubMedView ArticleGoogle Scholar
- Li J, Liu J, Karuturi RKM: Stepped linear regression to accurately assess statistical significance in batch confounded differential expression analysis. Bioinformatics Research and Applications. 2008, 481-491.View ArticleGoogle Scholar
- Ochs MF: Knowledge-based data analysis comes of age. Briefings in Bioinformatics. 2010, 11: 30-39. 10.1093/bib/bbp044.PubMedPubMed CentralView ArticleGoogle Scholar
- Smyth GK: Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Statistical Applications in Genetics and Molecular Biology. 2004, 3:Google Scholar
- Efron B, Tibshirani R: On testing the significance of sets of genes. The Annals of Applied Statistics. 2007, 1: 107-129. 10.1214/07-AOAS101.View ArticleGoogle Scholar
- Bishop E: A generalization of the Stone-Weierstrass theorem. Pacific Journal of Mathematics. 1961, 11 (3): 777-783.View ArticleGoogle Scholar
- Nocedal J, Wright S: Numerical Optimization. 2000, SpringerGoogle Scholar
- Goldfarb D, Idnani A: A numerically stable dual method for solving strictly convex quadratic programs. Mathematical Programming. 1983, 27: 1-33. 10.1007/BF02591962.View ArticleGoogle Scholar
- Press WH, Teukolsky SA, Vetterling WT, Flannery BP: Numerical Recipes 3rd Edition: The Art of Scientific Computing. 2007, Cambridge University Press, 3Google Scholar
- Hedges LV, Olkin I: Test of Statistical Significance of Combined Results. Statistical methods for meta-analysis. 1985, Academic Press, 28-46. 6Google Scholar
- Fisher RA: Combining independent tests of significance. American Statistician. 1948, 2 (5): 30-10.2307/2681650.Google Scholar
- DeRisi JL, Iyer VR, Brown PO: Exploring the metabolic and genetic control of gene expression on a genomic scale. Science. 1997, 278 (5338): 680-686. 10.1126/science.278.5338.680.PubMedView ArticleGoogle Scholar
- Gasch AP, Spellman PT, Kao CM, Carmel-Harel O, Eisen MB, Storz G, Botstein D, Brown PO: Genomic expression programs in the response of yeast cells to environmental changes. Mol Biol Cell. 2000, 11 (12): 4241-4257.PubMedPubMed CentralView ArticleGoogle Scholar
- Chen D, Toone WM, Mata J, Lyne R, Burns G, Kivinen K, Brazma A, Jones N, Bähler J: Global transcriptional responses of fission yeast to environmental stress. Molecular Biology of the Cell. 2003, 14: 214-229. 10.1091/mbc.E02-08-0499.PubMedPubMed CentralView ArticleGoogle Scholar
- Han X, Sung W, Feng L: Identifying differentially expressed genes in time-course microarray experiment without replicate. Journal of Bioinformatics and Computational Biology. 2007, 05 (02a): 281-10.1142/S0219720007002655.View ArticleGoogle Scholar
- Li J, Liu J, Karuturi R: Data-driven smoothness enhanced variance ratio rest to unearth responsive genes in 0-time normalized time-course microarray data. Bioinformatics Research and Applications. 2007, 25-36.View ArticleGoogle Scholar
- Marioni JC, Mason CE, Mane SM, Stephens M, Gilad Y: RNA-seq: An assessment of technical reproducibility and comparison with gene expression arrays. Genome Research. 2008, 18 (9): 1509-1517. 10.1101/gr.079558.108.PubMedPubMed CentralView ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.