A simulation study was performed to evaluate the behavior of the proposed index, denoted and compare it to Allison's index 16 , a modified version of Allison's index 18 , Nagelkerke 17 and Xu and O'Quigley's 19 indices denoted and respectively (see the Methods Section for the description of the five indices) under proportional and non-proportional hazards regression models, using different values of the regression parameter, different covariate distributions and different sample sizes. Scenarios with various independent censoring distributions were also considered.

The simulation protocol was as follows. For each subject i, i = 1,⋯, n, we considered one covariate Z with either a discrete (Bernoulli ℬ(0.5)) or a continuous (uniform [0, ]) distribution. These two distributions of Z were standardized to have the same variance. Survival times T were generated with the survival function S(t; z) = exp(-te ^βZ ) (proportional hazards model) or S(t; z) = (1 + t · e ^βZ )^-1 (proportional odds model). For these two survival distributions, the hazard ratios were HR = e ^β for the proportional hazard model and HR = [1 + (e ^β - 1)S ₀(t)]^-1 for the proportional odds model, S ₀(t) refering to the baseline survivor function. In our simulation scheme, e ^β was set to 1 (null effect), to small values; 1.25, 1.5, 1.75, medium values; 2, 3, and high values; 4, 5. The sample sizes n of the data were taken equal to 50, 100, 500 and 1, 000.

The censoring mechanism was assumed to be independent from T given Z and the distribution of the censoring variable C _i , i = 1,⋯, n was either uniform C _i ~ {0, r} or exponential C _i ~ {γ }. The calculation of the parameters r and γ as functions of the expected overall percentage of censoring p _c is described in Additional file 1. The percentage of censoring was taken equal to 0%, 25% and 50%. For each configuration 1,000 repetitions were generated.

The table in Additional file 2 displays the results of the simulations for for four different sample sizes and two different covariate distributions, considering a Cox proportional hazards model. As seen from Additional file 2, when β = 0, i.e. in the absence of covariates, our index approaches 0 for n = 50 to 1, 000; the separability is close to 0. The index increases towards 1 with |β|, the separability increases with the effect of the covariate. When β≠ 0, the value of for the different sample sizes is fairly stable, in particular for moderate or high effects (e ^β ≥ 1.5). The mean values of our index for n = 50 to 500 are close to the mean values obtained for n = 1, 000 which is assumed to approach its asymptotic limit. The standard errors of (indicated in brackets in Additional file 2) are small even when censored, and, as expected, decrease when n increases. Our index is slightly sensitive to the censoring rate, especially for high values of hazard ratio. Similar comments can be made when dealing with an exponential censoring mechanism (results not shown).

Figures 1, 2, 3 and 4 display, for a Cox model, the differences δ between the mean of and the mean of and respectively, for n = 100, for different percentage of censoring p _c , different covariate distributions and with a uniform censoring mechanism. The means of the differences δ are always positive. They are close to zero for small hazard ratios and increase with higher hazard ratios. The differences between and increase with the percentage of censoring, which is not surprising since the Nagelkerke's index is known to be sensitive to censoring 15 . The two indices and have a similar behavior relatively to . This is expected since O'Quigley et al 18 propose to use as a simple working approximation of their index. The same results are obtained for n = 50, 500 and 1, 000 and for an exponential censoring mechanism (results not shown). For e ^β ≥ 2, the 95% confidence interval for the differences of the three graphs does not comprise 0, thus in each case the difference δ is significant. The table in Additional file 3 and Figures 5, 6, 7 and 8 display the results of the simulations under a proportional odds model. The mean values of the different indices are lower than in the case of a proportional hazards model. All indices are more sensitive to censoring. Our index shows higher mean values than the other indices, especially in case of a Bernoulli distribution.

In this subsection, using a basic example we evaluated the practical interest of our index when combining the information contained in two studies with different sample sizes. The method used to generate the two datasets was inspired by Bair and Tibshirani 20 but modified in order to resemble the structure of real genomic data. The two datasets mimicked the analysis of the prognosis impact of transcriptional changes for a set of 1, 000 genes. The two datasets were of unequal size and composed of n = 150 and 50 individuals, respectively. To each individual i, i = 1,⋯, n; n = 150 or 50, we associated a survival time T _i , a censoring time C _i and vector of 1,000 quantitative values Z _i = { ; g = 1,⋯, 1, 000} (e.g. expression measurement).

To perform a fair evaluation of our index, we simulated survival data with either an exponential distribution given by S(t) = exp(-te ^ξ ) (proportional hazard model) or a log-logistic survival distribution given by S(t) = (1 + t · e ^ξ )^-1 (non-proportional hazard model). For individuals i such as 1 ≤ i ≤ n/2 (n = 150 or 50), the parameter ξ was equal to 0. For individuals i such as n/2 + 1 ≤ i ≤ n (n = 150 or 50), e ^ξ was equal to 3 and 5. We defined individuals i = 1 to n/2 as belonging to the group of patients with low risk of occurrence of the event of interest and individuals from i = n/2 + 1 to n to the group with high risk of occurrence of the event.

For each dataset, censoring times C _i (i = 1,⋯, n; n = 150 or 50) were considered independent from survival times and with a uniform distribution on {0, r}, r chosen in order to have an expected percentage of censoring of 30%.

The observed time to follow-up (i = 1,⋯, n; n = 150 or 50) was equal to the minimum between the two previously defined times T _i and C _i .

For the two datasets, for each individual i, 1,000 gene expression values (i = 1,⋯, n; n = 150 or 50; g = 1,⋯, 1, 000) were generated, according to the simulation scheme shown on the figure in Additional file 4. Gene expression values from g = 1 to 50 for individuals i = 1 to n/2 (n = 150 or 50) followed a log-normal distribution Log- (μ = 4, σ = 1.5) with and . For the rest of the individuals (i = n/2 + 1,⋯, n; n = 150 or 50), gene expression values followed a distribution Log- (0, 1.5). Gene expression values from g = 51 to 100 for individuals i = 1 to n/2 (n = 150 or 50) followed a log-normal distribution with parameters μ = 3 and σ = 1.5 Log- (3, 1.5). For the rest of the individuals (i = n/2 + 1,⋯, n; n = 150 or 50), gene expression values followed a distribution Log- (0, 1.5). For gene expression values from g = 100 to 150 and for 40% individuals randomly selected among the n (n = 150 or 50), the (i ∈ N, g = 100,⋯, 150) followed a log-normal distribution Log- (1, 1.5), whereas for the remaining individuals, they followed a log-normal distribution Log- (0, 1.5). For gene expression values from g = 151 to 250 and for 50% individuals randomly selected among the n, the (i ∈ N; g = 151,⋯, 250) followed a log-normal distribution Log- (0.5, 1.5), whereas for the remaining individuals, they followed a log-normal distribution Log- (0, 1.5). For gene expression values from g = 251 to 350 and for 70% individuals randomly selected among the n, the (i ∈ N; g = 251,⋯, 350) followed a log-normal distribution Log- (0.1, 1.5), whereas for the remaining individuals, they followed a log-normal distribution Log- (0, 1.5). Finally, for gene expression values from g = 351 to 1, 000, the (i = 1,⋯, n; g = 351,⋯, 1, 000) followed a log-normal distribution Log- (0, 1.5) for all individuals.

As genes involved in the same or related pathway are likely to be coexpressed, we introduced correlations between genes. To evaluate the behavior of our index in the context of dependent data, we generated datasets with so-called "clumpy" dependence (gene measurements are dependent in small groups, but each group is independent from the others). We applied the following protocol 21 22 . For each group of ten genes indexed by l, l = 1,⋯, 100, a random vector A = a _il , i = 1,⋯, n, was generated from a standard log-normal distribution Log- (0, 1). The data matrix Z was then built so that with ρ equal to 0.25, 0.5 or 0.75. Finally and in order to show the behavior of our index in situations close to real genomic data analysis, we standardized the dataset using classical quantile normalization 23 .

In this simulation scheme, the first hundred genes were differentially expressed between the low and high risk group of patients. The other 250 genes were not linked to the low and high risk status, but were distributed differentially according to a binary factor (with various means) unlinked to the low/high risk status. The remaining genes were not linked to the low and high risk status.

For a given threshold, we calculated the number of genes common to the two simulated datasets with the five indices, for the different survival distributions, the different hazards ratio values and the different correlations between genes. We estimated the true positive fraction (TPF, number of true positives found divided by the number of truly prognostic genes) and the true negative fraction (TNF, number of true negatives divided by the number of truly non-prognostic genes) obtained with the five indices, , and as a function of the threshold target value. These criteria were estimated by the mean over one hundred iterations of: (i) the proportion of correct selection (i.e. when the selected genes g belonged to {0,⋯, 100}) among the modified genes; (ii) the proportion of correct 'non-selection' (i.e. when the selected genes g belonged to {101,⋯, 1, 000}) among the non-modified genes, respectively.

Considering this procedure, the most successful criterion was the one that achieve the best operating characteristics.

Figure 9 displays the true positive fraction versus the false negative fraction (number of false positives found divided by the number of truly non-prognostic genes) for four configurations: ρ = 0.5, e ^ξ = 3 and 5 and for a proportional and non-proportional model. For the five indices, higher operating characteristics are obtained under a proportional hazards model (Figure 9.a and 9.b) as compared to a proportional odds model (Fig 9.c and 9.d). Moreover, for a given distribution and a given threshold, our index gives the best results with higher true positive and true negative fractions. Results for the four other indices are very close to each other. Results with other levels of correlation (ρ = 0.25 and 0.75) are very close to those obtained with ρ = 0.5 (curves not shown here).

In this section, we exemplify the use of the proposed index by identifying transcriptomic prognostic factors common to eight studies corresponding to five different solid tumors such as breast, lung, bladder cancer, glioma and melanoma. We compared our index to the four indices and two classical test-based criteria (q-values derived from the log-likelihood ratio and robust score statistics). The data consisted of eight independent genomic studies 24 25 26 27 28 29 30 31 , with different survival outcomes and different sample sizes which samples were hybridized on a same platform (Affymetrix HU133 Plus 2.0 or HU133A ; Affymetrix, Santa Clara, CA, USA). The datasets are publicly available on the GEO site under the labels GSE2034, GSE1456, GSE11121, GSE4573, GSE5287, GSE4271, GSE4412 and GSE19234, respectively, and they are briefly described below.

The proposed index was calculated for the 22,283 gene expression measures for the eight datasets. The intersection procedure 32 led to a threshold equal of 0.07 for .

For ≥ 0.07 (which corresponds from our simulations to a hazard ratio value around 1.5), we selected 5 transcripts related to four genes (Table 1).

We identified HJURP and LRIG1 genes that are directly involved in tumorigenesis. HJURP encodes an indispensable factor for chromosomal stability in immortalized cancer cells. It is up-regulated in lung cancer 33 . LRIG1 encodes a protein that acts as a growth suppressor in breast cancer 34 . Its expression decreases in human breast cancer and the majority of ErbB2+ breast tumors show under-expression of LRIG1. In our series, the increase of HJURP and decrease of LRIG1 gene expressions are associated with a worse prognosis.

Our selection process also brought two genes involved in cell cycle regulation. Gene KIF4A encodes a protein critical for mitotic regulation including chromosome condensation, spindle organization and cytokinesis. It possesses a functional and physical link with the gene product of BRCA2 (breast cancer 2, early onset) 35 . Gene ESPL1 plays a central role in chromosome segregation at the onset of anaphase. Its over-expression induces aneuploidy and tumorigenesis 36 . The article of Zhang et al 36 showed that the ESPL1 transcript is over-expressed in human breast tumors. It is worth noting that ESPL1 and KIF4A, have been previously discussed in a meta-analysis conducted by Carter et al 37 . For these two genes, over-expression, leading to a cell proliferation, is associated with a worse prognosis.

Finally, for each gene from our selection, the hazard ratios were in the same direction in each of the eight studies.

With a same threshold of 0.07, Allison's index selected 3 transcripts corresponding to genes KIF4A and ESPL1 and Xu and O'Quigley's index selected 2 transcripts corresponding to gene ESPL1. The transcripts identified with these two indices are all included in our selected subset. For and with a threshold of 0.07, no transcript was selected. No transcript was selected relying on the q-value calculated with the robust score or the log-likelihood ratio statistics with a threshold of 0.40.