On the necessity of different statistical treatment for Illumina BeadChip and Affymetrix GeneChip data and its significance for biological interpretation

Background The original spotted array technology with competitive hybridization of two experimental samples and measuring relative expression levels is increasingly displaced by more accurate platforms that allow determining absolute expression values for a single sample (for example, Affymetrix GeneChip and Illumina BeadChip). Unfortunately, cross-platform comparisons show a disappointingly low concordance between lists of regulated genes between the latter two platforms. Results Whereas expression values determined with a single Affymetrix GeneChip represent single measurements, the expression results obtained with Illumina BeadChip are essentially statistical means from several dozens of identical probes. In the case of multiple technical replicates, the data require, therefore, different stistical treatment depending on the platform. The key is the computation of the squared standard deviation within replicates in the case of the Illumina data as weighted mean of the square of the standard deviations of the individual experiments. With an Illumina spike experiment, we demonstrate dramatically improved significance of spiked genes over all relevant concentration ranges. The re-evaluation of two published Illumina datasets (membrane type-1 matrix metalloproteinase expression in mammary epithelial cells by Golubkov et al. Cancer Research (2006) 66, 10460; spermatogenesis in normal and teratozoospermic men, Platts et al. Human Molecular Genetics (2007) 16, 763) significantly identified more biologically relevant genes as transcriptionally regulated targets and, thus, additional biological pathways involved. Conclusion The results in this work show that it is important to process Illumina BeadChip data in a modified statistical procedure and to compute the standard deviation in experiments with technical replicates from the standard errors of individual BeadChips. This change leads also to an improved concordance with Affymetrix GeneChip results as the spermatogenesis dataset re-evaluation demonstrates. Reviewers This article was reviewed by I. King Jordan, Mark J. Dunning and Shamil Sunyaev.


Background
Microarrays that rely on hybridization with DNA probes pioneered large-scale expression studies. After the introduction of spotted array technology in the mid 90s, microarrays have steadily gained popularity for exploratory gene expression analysis. A spotted array experiment requires both the treated and control samples to be labeled with different dyes and to be competitively hybridized on the same array. The expression level is expressed as a ratio between the intensities between the two labels. Spotted arrays are plagued by accuracy and sensitivity problems that are only partly remedied by the measuring only relative expression. Dye bias and repeatability remain unsatisfactory.
In recent years, Affymetrix GeneChip and Illumina Bead-Chip have emerged as two of the most popular microarray platforms. From the experimental design viewpoint, the GeneChip and BeadChip offer flexibility in terms of their ability to measure absolute expression values for each experimental sample independently. The growing amount of publicly available microarray data has prompted researchers to explore ways to compare results between experiments across the different platforms. This task signifies the first step in producing consistent and trusted results to support meaningful biological discovery. Yet, it is more difficult than it appears superficially. Even a simple variant of the problem like comparing results from the same sample across different platforms is not trivial. The first step requires the statistically significant changes in gene expression to be determined between treatment conditions for each platform. The platform-specific gene lists generated by applying the same significance threshold are finally compared. In general, the concordance between these gene lists is disappointingly low. Nevertheless, recent works have shown that concordance improvements can be made by filtering for gene nucleotide sequence identity [1][2][3][4], by suppressing lower intensity genes [5] or by aligning gene lists with continuous measures of differential gene expression [6].
During our evaluation of cross-platform comparison between Affymetrix and Illumina, we stumbled upon another, quite surprising reason for the low concordance. Given the specific design of Illumina arrays [7][8][9][10], it appears that the data derived from them requires specific statistical treatment different from that of more classical microarrays. Notably, the Affymetrix GeneChip and Illumina BeadChip have one stark difference in their designs. In a nutshell, many instances of a unique probe design are synthesized onto a group of adjacent discrete features or cells on the GeneChip. Consequently, each group of cells will target a particular gene. In the case of Illumina Bead-Chip, a unit of bead coated with hundred of thousands of probes is analogous to a group of cells on GeneChip. Fur-thermore, multiple beads of a probe design are immobilized onto randomized positions on the BeadChip. Therefore, given a probe design, a gene is only measured once on the GeneChip, whereas it is measured typically about 30 times on the Beadchip. But instead of delivering the individual bead intensities (possible with appropriate scanner modifications), the mean and standard error (i.e., the standard deviation divided by the square root of the number of beads) of the bead intensities, known as the summary data, are usually reported.
Thus, Affymetrix GeneChips provide individual measurement results but the Illumina BeadChips generate means and standard errors for subsets of bead intensity measurements. Therefore, the summary data of a BeadChip experiment requires a different statistical interpretation compared with the individual measurements in the case of GeneChip data, especially in cases of multiple technical replicates. If the average of the bead intensities delivered by a single Illumina BeadChip is fed into standard expression profile analysis software (for example, GeneSpring), the standard deviation over technical replicates is calculated from the deviations of the subset means from the overall mean. But more correctly, the overall standard error is to be computed by taking into account also the standard deviations obtained from the individual Bead-Chips.
In this paper, we will first present a derivation for the correct summary statistic applicable to Illumina BeadChips data. Furthermore, this summary statistic will be applied to one control experiment with artificial spikes and, also, it will be used for the re-evaluation of two published biological experiments. In all cases, the modified treatment is contrasted against the standard one. In the control experiment [11], we will demonstrate a dramatic improvement in recognizing the spike sequence selection if the corrected summary statistic is applied. In the example of the MT1-MMP mammary epithelium dataset [7], cell cycle pathway involvement can be shown with statistical confidence only after applying the correct summary statistic. Interestingly, cell cycle gene involvement was suggested by the authors, although their analysis of the data did not provide strong arguments for it. Then with the spermatogenesis cross-platform data [8], we demonstrate that considerably improved concordance between the Affymetrix and Illumina platforms can be achieved with the correct summary statistic. Our analysis also provides new evidence for the transcriptional regulation of the N-glycan biosynthesis, the tight and the adherens junction pathways in this context, a finding that is supported also by independent experimental evidence.

Statistics of Illumina BeadChip & Affymetrix GeneChip datasets
The bead intensity of a given gene in a BeadChip is described with the random variable X. The expression profile experiment is supposed to consist of K technical replicates (independent measurement of arrays on the same biological sample). Each bead intensity x k,n is an instance of the random variable X (where k = 1...K replicates, n = 1...N k beads, N k is the number of beads in the k-th technical replicate). We assume that the first M k beads are retained after outlier removal (see below). The summary data includes the mean μ k , the standard error (where σ k is the standard deviation) and the number of beads M k (the typical value of M k is about 30).
The observed BeadChip intensities of gene in the k-th array are denoted as (Table   1). However, a typical BeadChip experiment does not report these individual bead intensities. Instead, the Illumina BeadStudio software first performs an outlier removal on the bead intensities. Instances with intensities above three median absolute deviations from the median are removed. Upon the outlier removal, the mean and standard error of the bead intensities as well as the number of beads used in summarization for each gene are reported (the AVG_signal, BEAD_STDERR, Avg_NBEADS columns in the Illumina Beadstudio output file).
Using the means and standard errors of all the technical replicates, the mean μ total and standard deviation σ total of bead intensities of a gene across the K technical replicates are given as A proof for equation (2) is supplied in the Appendix 1.
The standard deviation σ total is composed of two components each carrying a different meaning. Given that each of the K technical replicates represents the same, identical and independent distribution, one expects the K mean estimates μ k to be relatively similar and, hence, σ μ would be small relative to σ total and, ideally, close to zero. Since averaging for each replicate is carried out over about 30 individual measurements, it can be assumed that each individual μ k is likely a good estimate of the population mean μ if there were no batch variations. Therefore, a large σ μ can be interpreted as batch variation or noise among the replicates, a considerable part of which, apparently, has systematic origin such as variations in the total amount of hybridization-ready nucleic acids, etc. Ideally, K (typically 2-4) should be much larger in order to obtain a good estimate of σ μ . However, this is impractical due to the high cost of performing large number of microarrays.
Therefore, we suggest to assume σ μ ≈ 0 for the case of no batch variation in equation (5) and to use σ wtrep as a good lower estimate of the summary statistic in testing instead of σ μ (see Appendix 2 for proof).
σ 1 1 This proposed summary statistic is supported by observations communicated in two recent publications, which have leveraged on the variation in bead intensities. Dunning et al. [12] showed that differentially expressed gene detection experienced an increase in statistical power by using the inverse of as weights in their linear model.
On the other hand, Lin et al. [13] proposed a variance stabilization transformation that incorporated bead intensities variation and showed an improvement in differentially expressed gene detection. Beyond this point, we shall refer to σ total with respect to equation (5) instead of (2).
In the case of a Affymetrix GeneChip experiment, the measurement for a gene is taken only once in each replicate (see 4 th row of Table 1). Consequently, the mean and standard deviation of a gene across K technical replicates are given as The summary statistic [μ total , σ total ] and [ν total , ω total ] are the parallels between the Illumina and Affymetrix platforms.
However, σ total has an advantage over ω total . Due to multi-ple copies of the same probe within a single Illumina array, the standard deviation can be computed for each array individually. As a result, σ total offers more protection against any systematic error than ω total (see Appendix 2 for proof). The lack of systematic error as a confounding factor in σ total increases the chance of detecting true biological differences from the statistical tests.
In any case, the more important concern related to the analysis of Illumina data is the mistake of treating the mean estimates of bead intensities as instances of the bead intensities. Standard gene expression profile analysis software (as applied in several published studies [7][8][9][10]) assumes that the imported data are bead intensities rather than mean estimates of bead intensities. Such a software plainly computes the mean and standard deviation for the incoming data and the corresponding summary statistic for the control and the treatment group would be [μ total ,   The first column indicates the 11 test cases of pair-wise treatment conditions. The second and third columns indicate the number of TP and FP genes respectively for using σ μ as the summary statistic. Similarly, column four and five indicate the results for using σ wtrep . The last column indicates the result for the number of common TP genes found by both σ μ and σ wtrep .
hypotheses. Ideally, the P-values of the 34 alternate hypotheses will appear on one extremity of the Schweder-Spojotvoll plot [15]. Upon computing and sorting the Pvalues of the pair-wise t-tests (after array normalization as in Materials and methods), each of the 34 smallest P-values was examined to see if it belongs to a true positive (TP) or false positive (FP) gene.
The number of identified TP genes by the statistic σ wtrep is generally higher than that by σ μ ( Table 2). In particular, an improvement from 7 (0.1 and 0.03 pM comparison) or 14 (0.3 versus 0.1 pM) to 9 and 22 recovered spikes in the low concentration range of 0.03-0.3 pM is encouraging. Note that this region spans the endogenous gene expression level and, hence, it is critical to obtain good differentially expressed gene identification here. An improvement was also achieved in the high concentration region. But in practice, gene expression will not reach such level to leverage on it. Note that the detection limit was 0.25 pM while the saturation point was about 300 pM [11].
Most importantly, the TP genes found by σ μ is a subset of those found by σ wtrep . This means that more TP genes found by σ wtrep had moved into the first 34 ranks to dis-place only other FP genes. For that to happen, the P-values must have been re-ranked by the statistic so that the TP genes are more statistically significant than the FP genes.
This means that σ μ is not a good estimate for the standard deviation. Figure 1 shows a plot of mean bead intensities μ total against the standard deviation of mean estimates σ μ and variation in bead intensities σ wtrep of the human background cRNA (i.e. 0 pM spike data). 'Heteroskedasticity' means that the larger intensities tend to have larger variations, a common observation with many types of microarray data. The 'heteroskedasticity' nature of the relationship between the mean bead intensities μ total and the variation in bead intensities σ wtrep is apparent (in red in Figure 1). On the other hand, a trend of growth in the standard deviation of mean estimates σ μ with increase of mean intensities across the dynamic range is not certain (in blue in Figure 1) and even σ μ = const cannot be excluded (the case of purely systematic error). Given that the number of technical replicates K is only four, obtaining good estimates for σ μ especially in the higher intensities region is impossible. As such, we strongly advocate the use of equation (5), which only relies on σ wtrep for Relationship between σ μ , σ wtrep and μ total Figure 1 Relationship between σ μ , σ wtrep and μ total . We show the plot of mean estimates of bead intensities μ total against standard deviation in mean estimates σ μ and bead intensities σ wtrep of 0 pM spike concentration data.
computing the correct summary statistic, instead of equation (2).

MT1-MMP data: proof for cell cycle pathway involvement
Golubkov et al. [7] published the expression profiles of mammary epithelial cells without and after transfection with a plasmid carrying the membrane type-1 matrix metalloproteinase (MT1-MMT) gene recorded with the Illumina platform. Originally, the expression data was first normalized using the "normalize.quantiles" [16] routine of Bioconductor and then imported into GeneSpring for Welch's t-test (thus, using σ μ as the summary statistic). A total of 207 differentially expressed genes were determined with cutoff criteria of p ≤ 0.05 and absolute fold change (FC) of at least 2.
In this work, the original expression data was first normalized (see Array normalization procedure section) prior to statistical treatment. Welch's t test was then performed for both σ μ and σ wtrep , which yielded 215 and 218 differentially expressed genes respectively upon applying the same cutoff criteria. For the three lists consisting of 207, 215 and 218 gene candidates, RefSeq IDs were extracted. The resulting 202, 200 and 203 RefSeq IDs were then separately submitted to NIH DAVID [17] for KEGG pathway mapping. Furthermore, 19815 RefSeq IDs were extracted from the Illumina Human-6 Expression BeadChip annotation file and submitted to DAVID as the background list.
With reference to Table 3, only KEGG pathways with EASE [18] score ≤ 0.05 and count ≥ 5 are shown. In a nutshell, the EASE score is a P-value of a more conservative version of the Fisher's exact test while count denotes the number of genes in the differentially expressed gene list that belongs to a particular pathway. Notably, our analysis with the statistic σ μ essentially repeats the outcome of the work by Golubkov et al. with respect to the pathways (Table 3) and regulated genes (the overlap between the two lists includes 176 genes out of 200 and 202 genes respectively); thus, the differences in the normalization had a small effect.
Application of the statistic σ wtrep dramatically influences the result. Suddenly, additional cell cycle genes are significantly regulated in transfected cells and the cell cycle pathway pops up in the DAVID analysis. Table 4 highlights the two genes out of a total of six significantly upregulated genes from the elucidated cell cycle pathway. These two genes, Cyclin A1 and CDC45L, are found by σ wtrep but not by σ μ . Consequently, the addition of these two genes resulted in an improved EASE score of 0.04 (from 0.29). The elucidation of this pathway has substantiated the authors' claim with statistically significant expression arguments that the cell cycle is disrupted with observable mitotic spindle aberrations and aneuploidy in the 184B5-MT cells [7].

Human spermatogenesis data: proof for the N-glycan, the tight and the adherens junction pathway involvement
Platts et al. [8] studied RNA expression in ejaculates of normal and zoospermic men both with the Affymetrix and the Illumina platforms. The Affymetrix expression data of 13 normal and 8 teratozoospermic men was processed by the MBEI (PM-MM) algorithm after invariant set normalization to obtain the gene expression values using the DChip software [19]. The Illumina BeadChip study included only 5 out of the 13 normal but all zoospermic examples. The authors used the same procedure for elucidating differentially expressed genes in both cases [8].
In this work, 5 out of the 13 normal and the 8 teratozoospermic samples from the Affymetrix experiment that were used by Platts et al. in their Illumina experiment (N1, N5, N6, N11, N12) were re-analyzed. The gene-level data was normalized (see Materials and methods), followed by a pair-wise t-test with ν and ω as the summary statistic (equations 6 and 7). This resulted in a total of 11932 differentially expressed genes (6861 RefSeq IDs) after applying cutoff criteria of p ≤ 0.01 and |FC| ≥ 2. In a similar fashion, the expression data from the corresponding 5 normal and 8 teratozoospermic of the Illumina experiment was normalized and statistically treated for both σ μ and σ wtrep . Using the same cutoff criteria, the two analyses yielded 2464 DEGs (2109 RefSeq IDs) and 4149 DEGs (3316 RefSeq IDs) respectively. Since the number of differentially expressed genes for σ wtrep is increased for the same cutoff criteria, this statistic exhibited a higher statistical power.
The three RefSeq ID lists were submitted to DAVID for KEGG pathway mapping. For Affymetrix, the background list was set to a list of 39647 ReqSeq IDs that was extracted from the HG-U133 (version 2) annotation file. For Illumina, the same list of 19815 RefSeq IDs from MT1-MMP example (see previous section) was submitted as the background.  Table 3, only KEGG pathways with EASE score ≤ 0.05 and count ≥ 5 are shown in Table 5. The elucidation of the proteosome and ubiquitin mediated proteolysis pathways by the re-analyzed Affymetrix expression data is consistent with the authors' finding that there is a severe suppression of the proteosomal RNAs associated with the ubiquitin-proteasomal pathway (UPP) in the teratozoospermic samples. On the other hand, the Illumina analysis revealed the proteosome but not the ubiquitin mediated proteolysis pathway. Even though the count for the ubiquitin-mediated proteolysis pathway had increased from 11 to 15 when σ wtrep was used instead of σ μ , this increase only improved the EASE score from 0.07 to 0.066. This marginally missed the significance cutoff of ≤ 0.05. But more interestingly, the Illumina analysis was able to elucidate a few pathways The first, second and last column indicate the gene symbol, the RefSeq ID and the description of each cell cycle gene respectively. The third and fourth columns indicate the logarithm fold change and the P-value of each gene found by Golubkov V.S. et. al. [7], which is analogous to using σ μ as the summary statistic. The fifth and sixth columns indicate the logarithm fold change and the P-value of each gene found by using σ wtrep as the summary statistic. Cutoff criteria of p ≤ 0.05 and |FC| ≥ 2 were applied. The genes in the upper section were found only by the σ wtrep statistic, whereas the genes in the lower section were detected with significance by both statistics. involved in spermatogenesis [20], like the N-glycan biosynthesis [21][22][23], adherens and tight junction [24][25][26] when σ wtrep was used as the summary statistic. Table 6 highlights 8 out of 17 genes from the elucidated N-glycan pathway that are found by σ wtrep but not σ μ . Using σ wtrep as the summary statistic, the P-values for these 8 genes were improved. With the addition of these 8 genes, the EASE score improved from 0.143 to 0.003. More notably, genes with biological evidence on the role of N-glycan biosynthesis pathway in spermatogenesis begin to surface with the application of the correct summary statistic. There is independent experimental evidence that proves the involvement of the detected genes. For example, beta-1,4-galactosyltransferase-I (B4GALT1) is found to bind with ZP3 receptors on the sperm surface [27]. Also, there is a reported increase in dehydrodolichyl diphosphate synthase (DHDDS) activity in prepuberal rats during early stages of spermatogenesis [28,29]. MAN2A2 is also found to be implicated in male infertility of the alpha-mannosidase IIx (MX) gene knockout mouse [21][22][23].

Analogous to
In the case of the tight junction pathway, 17 out of 37 genes from this elucidated pathway are found by σ wtrep but not σ μ ( Table 7). The improvement in the P-values of these additional 17 genes effected a marked improvement in EASE score from 0.206 to 0.012. For the adherens junc-tion pathway, 14 out of 26 genes are found only by σ wtrep (Table 8). Consequently, the EASE score improved dramatically from 0.527 to 0.037. As a result of applying the correct summary statistic, the genes CLDN1, CSNK2A2, CTNNA1, JAM3, and TJP1 have surfaced from the analysis. Claudin-1 (CLDN1) is involved in the developmental regulation of the tight junctions in mouse testis [30], while casein kinase 2, alpha prime polypeptide (CSNK2A2) null male mice are infertile and their cells from spermatogonia to early spermatids suffer nuclear envelope protrusions, outer membrane swelling and inner membrane disruption [31]. Also, the assembly of adherens junctions between Sertoli and germ cells was associated with a transient induction in the steady-state mRNA and protein levels of cadherins and catenins. In particular, alpha-catenin (CTNNA1) expression was seen in semi-quantitative reverse transcription polymerase chain reaction and immunoblotting [32]. Furthermore, the expression of zona occludens 1 (TJP1) in rats is regulated in vitro during the assembly of inter-Sertoli tight junctions during spermatogenesis [33] while JAM-3 is a protein required for spermatid differentiation [34].
The concordance between the three RefSeq lists was next investigated. The results are shown in Figure 2. Through the derivation of a parallel summary statistic to the Affymetrix one, an addition of 423 concordance RefSeq IDs was recovered. This is an increase of 45% (423 out of The first, second and last column indicate the gene symbol, the RefSeq ID and the description of each N-glycan biosynthesis gene respectively. The third and fourth columns indicate the logarithm fold change and the P-value of each gene found by using σ μ as the summary statistic. The fifth and sixth columns indicate the logarithm fold change and the P-value of each gene found by using σ wtrep as the summary statistic. The cutoff criteria p ≤ 0.01 and |FC| ≥ 2 were applied. The first 8 genes are found to be statistically significant by σ wtrep only, the rest was found with both statistics. The first, second and last column indicate the gene symbol, the RefSeq ID and the description of each Tight junction gene respectively. The third and fourth columns indicate the logarithm fold change and the P-value of each gene found by using σ μ as the summary statistic. The fifth and sixth columns indicate the logarithm fold change and the P-value of each gene found by using σ wtrep as the summary statistic. A cutoff criteria of p ≤ 0.01 and |FC| ≥ 2 was applied. The first 17 genes are found to be statistically significant by σ wtrep only. The last row contain a gene excluded by σ wtrep due to |FC| ≤ 2 although p ≤ 0.01. The genes upper in the section were found only by the σ wtrep statistic, the remainder was found with both approaches except of the last gene entry recovered only by the σ μ statistic (it was filtered out due the fold change criterion).
942) in the concordance region. The correct summary statistic also improved gene discovery. Out of the 916 RefSeq IDs that were unique to the analysis by σ wtrep , 8 (B4GALT1, DHDDS, MAN2A2, CLDN1, CSNK2A2, CTNNA1, JAM3, TJP1) were validated spermatogenesis genes from the Nglycan biosynthesis, tight and adherens junction pathways. These pathways were not reported by Affymetrix. Another interesting observation is that the RefSeq list obtained with σ μ as a statistic almost formed a subset of the list yielded by σ wtrep . In total, 93.7% of its ReqSeq IDs were found within the list generated with σ wtrep . At first glance, it seems that the effect of using σ μ rather than σ wtrep does not seem detrimental. However, the rankings of the P-values in this overlapped region were not preserved, similar to the case of our Illumina spike experiments analysis. As such, the top 100 candidate list for example will be quite different when σ μ instead of σ wtrep is used.

Conclusion
Due to the specific statistical nature of the Illumina Bead-Chip summary data as means and standard deviations of subsets of measurements, the typical statistical workflow The first, second and last column indicate the gene symbol, the RefSeq ID and the description of each Tight junction gene respectively. The third and fourth columns indicate the logarithm fold change and the P-value of each gene found by using σ μ as the summary statistic. The fifth and sixth columns indicate the logarithm fold change and the P-value of each gene found by using σ wtrep as the summary statistic. Cutoff criteria of p ≤ 0.01 and |FC| ≥ 2 were applied. The first 14 genes are found to be statistically significant by σ wtrep only. The lower part of the table lists genes found by both statistics except for the last row that contains a gene excluded by σ wtrep due to |FC| ≤ 2 (although p ≤ 0.01).
of finding differentially expressed genes cannot be applied to this data directly. To remedy this situation, σ wtrep is proposed as correct summary statistic of the Illumina Bead-Chip. Our work has shown that the same Illumina BeadChip data from published experiments churns out better differentially expressed gene selection after applying our proposed summary statistic.
This was particularly evident in the low concentration range of the Illumina spike experiment [11,14]. Given that this range is typical for the endogenous gene expression, the improvement should also be observed in biological experiments as well. Indeed, the superior statistical significance contributed markedly to more successful biological pathway elucidations. This was demonstrated with the MT1-MMP [7] data as well as the human spermatogenesis [8] data. For these two examples, more relevant differentially expressed genes were revealed when our proposed summary statistic was applied. In fact, a number of these genes has already been independently validated in Venn diagram of gene list overlap the literature [21,[27][28][29][30][31][32][33][34]. Their biological significance was demonstrated through functional studies like gene knockout, mutagenesis and quantification studies like RT-PCR and immunoblotting. Finally in the context of cross-platform comparison between Affymetrix and Illumina, more concordant results were recovered for the spermatogenesis expression profile [8]. This should not be surprising because our summary statistic is a close parallel to that of Affymetrix.
To conclude, our work is most relevant and imperative to any investigator who wants to derive more accurate differentially expressed gene lists from Illumina data.

The Illumina spike experiment
We exploited the dataset from a published artificial spike experiment [11,14]; the complete dataset was obtained as a personal communication by Semyon Kruglyak [See additional file 1]. In total, 4 versions for each of eight artificial polyadenylated RNAs (bla, cat, cre, e1a, gfp, gst, gus, lux) were generated by the authors. Although it was not mentioned in [11,14],

The MT1-MMT (membrane type-1 matrix metalloproteinase) experiment
This dataset available as NCBI GEO GSE5095 was complemented with replicate-specific standard errors and number of beads in a private communication by Vladislav S. Golubkov. In the experiment, 184B5 human normal mammary epithelial cells were transfected with MT1-MMP [7]. Total RNA was then isolated from the 184B5-MT and 184B5 cell culture, following DNA-chip RNA expression profiling using Illumina Human-6 Expression BeadChips.

The human spermatogenesis experiment
The published expression profile dataset NCBI GEO GSE6969 from human ejaculates was used [8]. In the experiment, the samples were collected from 17 normal fertile men and 14 teratozoospermic men aged between 21 to 57. Upon RNA isolation of the spermatozoa, RNA expression profiling was carried out on both the Affymetrix and Illumina platform. 4 out of 17 normal and 6 out of 14 teratozoospermic samples are profiled by the Illumina Human-8 Expression BeadChips while the remain-ing 13 normal and 8 teratozoospermic samples were profiled by the Affymetrix HG-U133 (version 2) Gene-Chips. In addition, 5 out of these 13 normal samples and the same 8 teratozoospermic samples were profiled again by the Illumina Human-6 Expression BeadChips.

Array normalization procedure
Our proposed procedure is inspired by quantile normalization and by the scaling method used by Affymetrix [16]. The quantile normalization variant is applied to groups of technical replicates with the goal to achieve equal spread in the distribution of the bead intensities for each array. Then, the scaling method is applied to all the arrays to ensure that the medians of all arrays are equal.
For the sake of simplicity, the normalization procedure illustrated below will be based on only one treatment condition. The same steps will be repeated for other treatment conditions. Therefore, an arbitrary gene g from an array consisting of a total of G genes with K technical replicates each will have summary data μ g,k , σ g,k , M g,k where g = 1,...,G and k = 1,...,K.
Log-transformation is first applied on the mean bead intensities for all readings. The k-th technical replicate of gene g after undergoing log-transformation is depicted as log 2 (μ g,k ).
First, normalization within the replicate is performed. For the k-th technical replicate, the median and standard deviation for the log mean bead intensities within replicate are calculated as The median-of-medians within replicate and its corresponding standard deviation, hereby depicted as MOM wtrep and respectively, are given as Therefore, the normalized log mean bead intensities within replicate for the k-th technical replicate of gene g is defined as The median for the normalized log mean bead intensities within the k-th replicate is then re-calculated in a similar fashion as before, where The median-of-medians across replicates, depicted as MOM acrep , is defined as Therefore, the normalized log mean bead intensities across replicates will be The normalized log mean bead intensities across replicates are then transformed back to the original scale, where and the corresponding standard deviation is now σ g,k,norm = scale σ * σ g,k Therefore, after undergoing the array normalization procedure, an arbitrary gene g from an array consisting of a total of G genes with K technical replicates each will now have the summary data μ g,k,norm , σ g,k,norm , M g,k where g = 1,..., G and k = 1,...,K.
Assume that one treatment condition is replicated K times in the Beadchip, i.e., k = 1,...,K. When the mean estimates μ g,k are being treated as the intensity values x g,k,1 , the mean and the incorrect summary statistic of the variance of this treatment condition are given as Note that is equivalent to equation (3).
On the other hand, when no misinterpretation of the summary statistic has occurred, the mean and variance are given as μ g = median(μ g,1,norm ,...,μ g,K,norm ) Note that is now equivalent to equation (5).

Statistical test procedure
A pair-wise t-test [35] infers if differences exist between the two populations sampled. Furthermore, given 2 treatment conditions c1 and c2, an arbitrary gene g will have two pair of readings μ g,c1 , σ g,c1 , n g,c1 and μ g,c2 , σ g,c2 , n g,c2 .
Then if the two treated samples came from normal populations and if both have equal variances, then the t value for the difference between the two treatment conditions is given as where , and the degree of freedom is given as ν = n g,c1 + n g,c2 -2. However, if the two treatment samples do not have equal variances, the t value is given as The degree of freedom is given as . This is known as the Welch's t test. For a 2-sided alternate hypothesis H A : μ g,c1 ≠ μ g,c2 , reject H 0 : μ g,c1 = μ g,c2 if |t|≥

Reviewer's report 1 I. King Jordan, School of Biology, Georgia Institute of Technology
Wong, Loh and Eisenhaber present a novel statistical method for evaluating gene expression data produced using the Illumina BeadChip technology. The fundamental insight that led to the new statistical method is their appreciation that Affymetrix GeneChip microarrays produce single gene expression measurements, while use of Illumina BeadChips yields mean expression values from dozens of identical probes. Therefore, Illumina BeadChip data must be treated differently. Specifically, when technical replicates are available, the standard deviations across replicates for Illumina BeadChip data are best computed as weighted means of the square of the standard deviations of individual measures. In other words, the standard deviations for data sets with technical replicates should be computed from standard errors of individual Illumina BeadChip measures. When this adjustment is applied to several test data sets, the performance of the Illumina BeadChips improves markedly.
While I am not qualified to evaluate the statistical details of their method, the results of its application to the three test data sets appear to be quite convincing. As such, this work represents an important technical development with direct relevance to any study that uses Illumina BeadChip technology.
One of the measures used by the authors to indicate the success of their statistic is increased concordance between lists of differentially expressed genes uncovered by Illumina BeadChip and Affymetrix GeneChip experiments on a spermatogenesis dataset. However, it would seem that the increased replicates of the Illumina BeadChip technology provides for an inherent advantage over the singlemeasure technology employed with Affymetrix Gene-Chips. If this is indeed the case, then one may expect improved performance for the Illumina platform relative to Affymetrix and not merely increased concordance as was demonstrated for the spermatogenesis dataset. Do the authors have any sense, or evidence, as to whether the increased sampling of Illumina provides more resolution than Affymetrix? For instance, are the new pathways identified by the Illumina BeadChip analysis of the spermatogenesis dataset a function of the superiority of the platform? Or are the methods complementary, i.e. does the Affymetrix analysis uncover pathways missed by Illumina irrespective of the use of the statistical innovations introduced herein?

Authors' response
There is no reason to assume a superiority of either platform given the same quality of probe sequence design. For example, one might imagine several Affymetrix chips to be mounted on the same glass slide and to be hybridized simultaneously (to resemble the situation of several beads per array). In this case, both platforms can be used in the mode of exclusion of the batch-specific constant shift error as described in the text.
I am wondering about the availability of their method. The authors conclude that the work is relevant, even imperative, to any investigator looking for differentially expressed genes in Illumina data? How are those investigators to use this method -on a web server, as a BioPerl object, as an R routine? The investigation into the performance of σ wtrep is well presented. However, a detail in the re-analysis of Golubkov et al. seems to be missing. In the original paper, genes were filtered using the detection scores obtained from Illumina's software. It does not seem that these scores were supplied in GEO, so were these scores also available to Wong et al. as part of their re-analysis? If not, how did they go about reproducing the filtering performed in Golubkov et al.?

Authors' response
Authors' response Indeed, the data stored in GEO is insufficient to carry out the calculations both in the paper of Golubkov et al. and  The results supplied in the paper were enough to convince me that the summary statistic σ wtrep is better than current alternatives. However, I'm afraid I was a bit unsure of the connection between σ wtrep and the normalization method proposed by the authors. Can I still use σ wtrep in my differential expression analysis if I use the usual quantile normalization?

Authors' response
The summary statistic σ wtrep can be used with the usual quantile normalization or any normalization methods.
One only has to ensure that the standard deviation σ k of the corresponding μ k be adjusted by a transformation factor i.e. σ k(normalized) = Aσ k where . After which, σ wtrep is computed using equation (4).
What motivated the authors to propose this method of normalization? However, I feel that the description of the normalization procedure was not that easy to follow and would benefit from a small worked example if possible. Do the authors plan to make any of the methodology presented in the paper available in open-source software?
Authors' response In a typical Illumina BeadChip experiment, different treatment conditions should be hybridized within a chip, while their corresponding technical replicates should be distributed across chips. The treatment conditions within a chip shall be exposed to similar systematic and random error. Hence, the differences in spreads among the arrays or treatment conditions should ideally reflect true biological differences. The motivation of our normalization method is to create a two-step normalization procedure whereby the first step forces the same median and spread only among technical replicates while the second step simply ensures that the medians across all the arrays are common. As such, the spreads among the various treatment conditions need not be the same, thus preserving true differences. The software (as a Matlab program) is available on request.
Aside from these questions, and suggestions to improve readability supplied separately to the authors, I am happy for this manuscript to be published.
Specific Comments for the authors: • Bottom of Page 2: "But instead of delivering the individual bead intensities, the mean and standard error (i.e. the standard deviation divided by the square root of the number of beads) of the bead intensities, known as the summary data, are reported." This statement is possibly a bit misleading as the individual bead intensities are available with appropriate scanner modifications (see Dunning et. al). I suggest this statement be changed to acknowledge this, although the summary data are usually the starting points for analysis when using Illumina's software.
• Page 3 paragraph 3 -"Furthermore, this summary statistic will..." This should be changed to either "this summary statistic" or "these summary statistic". I suggest the manuscript be checked for other similar errors.  The manuscript is well written. The statistical approach is straightforward but has been convincingly demonstrated to produce biologically meaningful results. The authors show that the corrected standard deviation estimate helps obtaining better results for the spike dataset (Chudin et al.) and also reveal more biologically relevant genes in the human spermatogenesis dataset and mammary epithelial dataset.
As an outsider in this field, I do not understand why the analysis is based on the t-test, which heavily depends on sample estimates of variance and assumes normality. It seems that non-parametric method may suite the problem better.

Authors' response
First of all, it is important to note that the summary data is computed using the arithmetic mean and the standard deviation formulas. These formulas are the maximum likelihood estimates (MLEs) of the normal distribution. In other words, normality is innately assumed on the summary data. Furthermore, the typical sample size for each gene measurement in Illumina is about 30 and t-test is known to be robust when sample size is large. More notably, t-test is robust against assumption violations as long as the sample sizes are almost equal and that only two-tailed hypotheses are considered. These were the conditions for all our test cases.

Proof for
With reference to Table 1, if the individual bead intensities are available, the mean μ total and squared standard deviation of the total set of bead intensities across K technical replicates can be computed as where, for convenience of notation, we introduce On the other hand, only the summary data are given.
Then the mean μ total and the squared standard deviation of the average bead intensities across K technical replicates are given as Clearly, the weighted average of the K bead intensity averages is equal to μ total (equation 8 and 11). For later usage, the equation (12) can be transformed by representing μ k by the average of actual bead intensities .
On the other hand, the variance in bead intensities across K technical replicates is defined as weighted aver- Obviously, the second term in equation (16) and the first in equation (13) are identical except for the sign. Together with equations (8) and (9), this proves that . Note that, in practice, the number of beads for each replicate is roughly equal. Hence when M k ≈ M for k = 1,...,K, the weighted arithmetic averages in the above equations can be justifiably by normal averages.

Computation of σ total under the condition of no systematic error
We assume the k-th batch-specific expression vector in Table 1 to be composed of a random, batch-independent component (with standard deviation σ k and batchindependent mean μ) and batch-specific systematic shift vector having equal components s k (thus, with mean s k and zero standard deviation). Whereas the standard deviation of the k-th replicate is not affected by the constant systematic error, the mean μ k is given as μ k = μ + s k . Therefore, the mean μ sys across replicates is where s denotes the average of the systematic shifts. The standard deviation of the means from the K replicates is essentially the standard deviation σ s of the systematic shifts as is shown with the derivation Consequently, the standard deviation of bead intensities is plagued by the systematic error. On the other hand, the standard deviation σ k of the k-th replicate is not affected by the batch-specific shift and, therefore, the batch-specific systematic error does not affect σ wtrep . Rightfully, the systematic error should not be present after array normalization. Hence, under the assumption of no systematic error after array normalization, usage of σ wtrep as reliable (lower) estimate of σ total is justified.