MeDiChISeq processes mapped sequence files in different formats (e.g. BED, BAM). Read-count intensity profiles are reconstructed from mapped read files by elongating each read to a user-defined length (default read elongation: 150nt) and counting the elongated read overlaps within a defined window (default wiggle-format files resolution: 10nt). While the read elongation parameter is generally provided by the user, we have incorporated in MeDiChISeq a function that predicts a suitable read elongation from the information retrieved in the ChIP-seq profile itself (see Additional file 2 and below).

In ChIP-chip the reconstructed signal intensity is generated by comparing the immunoprecipitated information (IP) with the control dataset, while in ChIP-seq the IP and control datasets are processed separately. Therefore, MeDiChISeq takes as an additional file (when available) a control dataset for improving the confidence assessment of the identified binding events (see below).

One of the main advantages of MeDiChISeq is its capacity of inferring a representative binding pattern (referred to as “kernel”) from the provided ChIP-Seq dataset. As illustrated in the Additional file 3, this is performed by fitting a binding pattern model to a reduced number of genomic regions, which are selected by applying a read-count intensity cutoff criterion. This cutoff can be defined as a given read-count intensity or by a quantile intensity parameter.

Model fitting is performed in an iterative manner by evaluating in each round the number of peaks that fit the best to the current model and adjusting its parameters (shape and scale of Gamma distribution) by minimizing the regression residuals. The formalism of this procedure is extensively described in 15 and its implementation for ChIP-seq datasets is detailed in the Additional file 1.

In ChIP-seq assays the reconstruction of factor binding/chromatin modification profiles is currently performed by applying a computational elongation of the sequenced reads prior read-count intensities assessment. This elongation step is performed because each sequenced read corresponds to the 5’-ends of the fragmented immunoprecipitated chromatin. Importantly, the applied elongation distance, which corresponds generally to the fragmented chromatin length, is important for proper assignment of a factor binding site or chromatin modification.

While the read elongation parameter is generally provided by the user (based on the experimental conditions), we have incorporated in MeDiChISeq a function that could predict a suitable read elongation from the information retrieved in the ChIP-seq profile itself. In fact, while previous studies assessed the read elongation distance by evaluating the distance between the forward and reverse enriched reads 7 , MeDiChISeq applies the iterative linear regression model fitting in a strand-specific manner without read elongation. This preliminary step infers the DNA fragment length per strand (ideally both strand-specific fragment lengths are the same); subsequently these are combined to define the read elongation parameter (see Additional file 2).

Binding site identification by MeDiChISeq is based on the assumption that all binding patterns associated to a given immunoprecipitation assay might present similar peak shape characteristics. Thus, the representative binding pattern or kernel modeled by the iterative regression approach is used to deconvolve binding events over the entire ChIP-seq profile. For this, the dataset is subdivided into overlapping windows (default parameter 20,000 nt window length; a contiguous window overlap is defined to cover at least one peak length) and the pre-computed kernel is used to identify those enrichment patterns that fit best.

Like in the case of MeDiChI, the likelihood of an enrichment event to match the trained kernel is related to the ChIP-seq background and is estimated by applying a non-parametric bootstrap approach 15 . MeDiChISeq compares for this purpose the putative binding sites identified by kernel fitting with the “kernel-fitting residuals” (i.e., those not complying with the model, and corresponding to the background). Moreover, these residuals are further deconvolved to identify potential patterns that would match with the operative kernel despite their possible background characteristics. Finally, each putative binding site is compared with its surrounding background in a local (default size of this centered surrounding windows is 5,000; 10,000 and 15,000nt) and global (genome-wide background) context.

The use of three different window sizes facilitates classifying the surrounding of potential binding sites as background. MeDiChISeq provides to each identified binding site local confidence p-values for all three evaluated windows and a global p-value. To provide an overall confidence estimate based on both global and local p-values, these descriptors were combined into a single confidence indicator (Fisher’s combined probability test).

When available, a control dataset (e.g., non-enriched sample or IP with non-specific IgG) is included during the binding site assessment. Indeed, whenever an enrichment event matches with the trained kernel, the kernel-fitting process is also performed in the control dataset for the corresponding genomic region. If in a given chromatin region both the enrichment and the control dataset comply with the trained kernel, the confidence of the identified binding site in the immunoprecipitated dataset is corrected as follows: Confidence Threshold CT = – log p ‒ value IP * Intensity IP – – log p ‒ value Control * Intensity Control

This approach enhances the confidence of the predicted binding event by evaluating its uncertainty from different perspectives, namely relative to a local background, relative to the identified patterns across the genome (global background) and relative to the enrichment seen in the control sample. Note that the described control sample-based confidence correction is based on the assumption that the compared datasets (IP and control) present comparable sequencing depth levels. It is important to mention that some methodologies apply in case of divergent sequencing depths linear scaling corrections; however we have shown in a previous study that important differences in sequencing depths may give rise to non-linear differences between compared datasets 16 .

In contrast to other methods, we do not suggest a default p-value threshold but provide a comprehensive list of all identified binding sites (complying with the kernel fitting) and their associated confidence descriptors such that the user can chose the optimal confidence threshold. In fact, defining a default p-value threshold may be misleading for inexperienced users, who may consider such reference as a gold standard rather than evaluating by other approaches the degree of false calls for a given p-value. Instead, we propose a graphical approach for estimating p-value levels, which may preferentially be associated to background behavior (described in the MeDiChISeq vignette; Additional file 4).

MeDiChISeq has been implemented in R and is designed to operate by multicore processing to accommodate computation requirements during linear regression fitting and bootstrapping.

For users who are interested in evaluating sites identified by another peak caller, MeDiChISeq offers an option in which only defined regions can be deconvolved. The R package, vignette and manual of MeDiChISeq are available as additional files (Additional files 4, 5 and 6). Note that these files can also be downloaded from our homepage http://igbmc.fr/Gronemeyer_MeDiChISeq.

Figure 1 illustrates that a comparison of peak caller performances under default parameters is unsatisfactory. In fact, default confidence thresholds that are too relaxed will increase the amount of false positive calls, while too stringent conditions will produce false negatives. To circumvent this problem, peak caller performance can be evaluated in the context of reproducibility assays by comparing binding site predictions for biological replicates and ranking them according to confidence descriptors. The underlying assumption is that true binding sites will be retrieved in both replicate datasets within a similar confidence ranking, while low confidence peaks, which are expected to contain also false positives, will show lower consistency in the reproducibility assay. The consistency between ranked peak confidence descriptors was previously formalized based on a copula mixture model, which estimates the probability that each pair of peaks is reproducible. This probability was described as “Irreproducibility Discovery Rate (IDR) 20 and has been used by the ENCODE consortium to identify a transition from signal to noise when peak caller binding site predictions were evaluated 21 .

Here we have compared peak caller performance in the context of reproducibility across replicate ChIP-seq datasets. Importantly, MeDiChISeq showed the highest number of reproducible peaks in CTCF and NRSF ChIP-seq datasets (Figure 2A). Also for broader patterns like H3K27ac and H3K4me3 MeDiChISeq identified the highest number of reproducible peaks at acceptable IDR thresholds (e.g., 0.1 or 90% reproducible discovery). Note that the IDR progression curve for the histone modification mark H3K4me3 continues to increase rather slowly above this threshold, suggesting retrieval of an important number of irreproducible events among the significant top-ranked peaks in the replicate dataset. That the other peak callers identify less than 5,000 H3K4me3 peaks with IDR levels below 10% supports the view that for broader binding patterns the assessment of IDRs by applying standard approaches becomes suboptimal 20 .

An important limitation of the above analysis is the potential variability between compared biological replicate datasets, as technical differences between the compared profiles may exist (e.g., sequencing depth differences; Peak caller deconvolution performance for broad patterns). To circumvent this limitation, we treated the predictions of two peak callers as “virtual replicates” for IDR analyses for a number of individual ChIP-seq datasets (Figure 2B). We thus ask if two peak callers identify binding events/marks in the same ChIP-seq dataset with similar confidence (i.e., if a top ranked peak of peak caller A is also top ranked by peak caller B). This novel type of comparison demonstrated that MeDiChISeq identifies higher numbers of reproducible peaks when compared with other methods. In fact, in the case of CTCF datasets, MeDiChISeq-MACS performed best for the first replicate, while MeDiChISeq-PeakRanger won in the case of the second replicate (Figure 2B). Importantly, evaluation of H3K27ac and H3K4me3 ChIP-seq datasets by this approach revealed large differences in reproducibility performance of the peak callers. PeakRanger and BayesPeak systematically performed worst, while MeDiChISeq versus any other peak caller gave the best scores in either biological replicate. Note that the particular IDR patterns observed for H3K4me3 in an inter-replicate comparison (right panel in Figure 2A) was not seen when the inter-peak caller performance for each replicate dataset was compared (right panels in Figure 2B), suggesting that it results from significant divergence between the “biological replicate datasets”. Overall these analyses showed that MeDiChISeq systematically identified the most reproducible events among biological replicates and peak caller annotations, thereby revealing the high sensitivity and reliability of this approach.

In addition to identifying the highest number of true binding events (sensitivity), a good peak caller algorithm is expected to produce the lowest amounts of false positives (specificity). As indicated above, IDR studies are expected to identify a transition from signal to noise when evaluating peak callers’ binding sites reproducibility. In this manner, the highest number of significant binding sites at the lowest IDR, as observed in the case of MeDiChISeq performance (Figure 2), reflects a high degree of sensitivity and specificity, at least in the context of reproducible binding site discovery in biological replicates or when comparing different peak callers’ performance per ChIP-seq dataset.

Previous studies have evaluated peak caller performance to distinguish false positives from “true” binding sites by using a manually curated collection of binding regions (and “false” enrichment sites) that cover typical variation in peak size and appearance for the TFs NRSF, SRF and MAX 14 17 . We have evaluated MeDiChISeq in the context of this benchmark, demonstrating in all three cases a high percentage of true binding site recovery (> 80%) and low false discovery rate (Figure 3). It is worth mentioning that, while its overall FDR performance is similar to that of MACS, MeDiChISeq generally retrieves more true binding sites. Furthermore, while using a background control dataset affected the false discovery rate of all other evaluated peak callers, MeDiChISeq performed equally well in identifying true binding events in presence and absence of this control. This is most likely due to the fact that the binding site identification relies on a pre-computed kernel and is thus less affected by artifactual enrichment events. This performance is well illustrated in the case of NRSF datasets, where in the absence of background control dataset, MACS and MeDiChISeq present a maximal percent of recovery of 90% but accompanied by high FDR levels (>0.5). Importantly, the incorporation of background control dataset in the analysis reduces the FDR levels but the percentage of true site recovery is also compromised for MACS (less than 80%), while MeDiChISeq manages to keep the percentage of recovery levels up to 90% even under these conditions (FDR < 0.4).

The rapid technological progress in the field of massive parallel sequencing provided over the years sequencing platforms with continuously increasing sequencing depths. In fact, while the first versions of the Illumina Genome analyzer had sequencing capacities in the range of several millions reads, the latest Hiseq2000/2500 versions provides more than 3 billion reads per flow cell. Following this continuous progress, the number of sequenced reads used per ChIP-seq assay has increased considerably. In fact, while early ChIP-seq assays generated <4 million total mapped reads (TMRs), current datasets comprise >20 million TMRs. Importantly, increasing the sequencing depth increased also the number of discovered binding sites 5 22 23 but only a few studies evaluated the peak caller performance for conditions with varying sequencing depths. Obviously, increasing the sequencing depth will increase both the signal and the noise levels, which could potentially affect peak caller performance.

To address this question, we created a high density ChIP-seq dataset by combining the datasets of the two biological CTCF replicates. This meta-profile comprised >36 M TMRs and was used for profile reconstruction from subsets generated by random sampling (80%; 60%; 40%; 20%; 10%; 5%) (Additional file 9A). To perform IDR evaluation, pseudo replicates were produced by two independent random samplings. As expected, the CTCF profile reconstructed from <2 M TMRs had unacceptably high IDR levels (Figure 4). In this condition MeDiChISeq and PeakRanger performed worst, followed by MACS and BayesPeak. This is readily explained by the fact that both MeDiChISeq and PeakRanger evaluate peak shapes, which are highly variable at low TMR levels (see pseudo replicates at 1.8 M TMRs in Additional file 9A). Importantly, with the increase in the TMR levels peak shapes consolidate and the performance of MeDiChISeq is enhanced accordingly (Figure 4A). Indeed, above 14 M TMRs MeDiChISeq outperforms all other peak callers with respect to the number of significant peaks with the lowest IDR levels. Note that above this TMR level all peak callers tend to reach saturation, a phenomenon generally referred to as the sequencing depth beyond which the number of newly discovered sites (in this case in a reproducible manner) reach a constant level (i.e. between 30 to 40 thousand sites for IDR levels lower than 0.1; Figure 4B).

This comparative study clearly demonstrates a direct correlation between TMR size and the degree of reproducible peak identification by any of the compared peak callers. In addition, it shows that MeDiChISeq, and to a certain degree also BayesPeak, tend to retrieve more reproducible binding sites than MACS and PeakRanger. This could be the direct consequence of different sensitivities and/or specificities of the peak callers under these conditions. To assess this issue we evaluated the extent of overlap between the retrieved sites by the different methods relative to MeDiChISeq. This analysis demonstrated that MeDiChISeq retrieved >88% of the binding sites identified by the other methods, but predicted an additional >10,000 specific sites (Additional file 9B).

These additional sites may originate from the use of the stringent comparative conditions (summit overlap +/−50nts distance). Indeed, a comparison between MACS and MeDiChISeq revealed that > 4,000 of the 15,000 MeDiChISeq-specific sites overlapped with MACS calls, when peak comparison settings were relaxed (Additional file 10). The remaining 10,000 binding sites that did not appear in the MACS-predicted site list were further evaluated for the presence of the previously described CTCF motif. Importantly, more than 4,000 MeDiChISeq-specific sites (40%) contained a CTCF motif, strongly suggesting that this population corresponds to bona fide CTCF binding sites that were ignored by MACS (Figure 4C). Of note, PeakRanger and/or BayesPeak identified nearly 3,000 of these bona fide CTCF binding sites (illustrated in Additional file 10C, right panel).