The reference set of NFS-Indels was constructed using the same approach as that described in the article presenting the PROVEAN method 18 . First, NFS-Indels were collected from the UniProtKB/Swiss-Prot database 27 , and annotated in-house as deleterious, neutral or unknown, based on keywords found in the UniProtKB annotations (e.g. variants described using words such as “inhibit'', ''affect'', etc. were classified as deleterious). Second, this set was enriched with NFS-Indels retrieved from the 1000 Genomes Project database 28 . Variants with average allele frequencies of >10% were collected and could thus be considered as common, i.e. non pathogenic or neutral, in the human population.

All the variants were then mapped to the human protein sequences in the SM2PH knowledge base. This resulted in a data set containing a total of 2163 NFS-Indels, mapped to 1535 distinct proteins. Of these, 1406 NFS-Indels were defined as polymorphisms/neutral (147 insertions and 1259 deletions) and 757 as disease-causing/deleterious (108 insertions and 649 deletions). Then, 757 neutral NFS-Indels (101 insertions and 657 deletions), were randomly selected from the pool of neutral NFS-Indels. Thus, the final data set was balanced for positive/negative examples (i.e. neutral/disease-causing NFS-Indels), an important characteristic to ensure appropriate statistical analysis and quality learning. The variants used in the final data set are provided in Additional file 1.

The annotations used to characterize the NFS-Indel data set were either extracted from the SM2PH knowledge base or calculated using existing tools for protein structural analysis and in-house developed Python scripts. The SM2PH knowledge base contains high quality multiple sequence alignments for all human proteins, which are annotated with structural and functional parameters derived from MACSIMS (Multiple Alignment of Complete Sequences Information Management System). MACSIMS combines knowledge-based methods with complementary ab initio sequence-based predictions to extract valuable information from multiple alignments 29 . These parameters are described in more detail in 25 and on the SM2PH help pages (decrypthon.igbmc.fr/sm2ph/cgi-bin/help). This information was complemented by additional structural parameters, including secondary structures and residue solvent accessible surface area (RSA) calculated using Spine-X 30 .

The different structure, functional and evolutionary parameters are shown in Table 1 and described in detail below.

The column ‘In final method?’ indicates whether the parameter is used in the final ILP rule set for prediction of deleterious NFS-Indels. *The numerical values range from −5 to +5 but, in order to reduce computational cost, we have regrouped values higher than ±2 into the semantic category two or more/two or less.

Machine learning, a branch of artificial intelligence, concerns the construction and study of computational systems that can learn from data. The development of a machine learning tool requires three steps. The first one is to obtain a data set that is large, representative and error-free, in order to ensure accurate automated learning. The second is to develop a strategy for the characterization of the objects or examples contained in this data set. This is called annotation of the data set and includes the parameters used to characterize the examples in the data set, as well as the categories or classes that are to be learned (in this case, deleterious or neutral variants). These two steps are done in the two previous paragraphs ‘Data set collection’ and ‘Data set annotation’. The third is to design an efficient machine learning strategy. This includes the pre-selection of the parameters that will be used during learning and the choice and optimization of the learning algorithm 42 .

Here, we have used a machine learning algorithm called Inductive Logic Programming (ILP), which infers hypotheses from experience (inductive learning) by means of logic programming. The approach is based on positive and negative examples, which combined with background knowledge, can be used to infer a hypothesis. Positive and negative examples are objects (here, NFS-Indels) from a training set that satisfy a condition (here, are deleterious or neutral) and the background knowledge consists of the set of parameters for these objects. ILP searches for a combination of parameter values that covers the maximum number of positive examples and the minimum number of negative examples. Such combinations are called ‘rules’. The process is then repeated until all positive examples are covered by at least one rule.

We used the ILP algorithm implemented in Aleph 26 . Aleph allows the user to set a number of program options, including the minimum number of positive examples (minpos) and the maximum number of negative examples for each rule (noise). Then ILP evaluates each rule based on the difference between the number positive examples covered and the number of negative examples. If the minpos and noise constraints are satisfied the rule is added to the hypothesis space. In the experiments described in the Results section we set minpos = 6 and noise = 0, in order to eliminate false positive predictions.

The rules produced by the ILP algorithm can be used to predict the status of unknown objects, i.e. to predict whether an unknown NFS-Indel is deleterious or neutral. To evaluate the prediction power of our algorithm, we used a 10-fold cross-validation strategy. Specifically, we randomly split the data set into 10 parts: 9 parts were used to train the ILP algorithm and 1 part was used to estimate the prediction performance of the method. The performance was assessed using the statistical parameters shown in Table 4 and defined in 42 . This process was repeated 10 times.

T=True, F=False, p=Positive, n=Negative, NPV=Negative predictive value, MCC=Matthews correlation coefficient.

Before running Aleph, we performed a pre-selection of parameters to avoid the effects of the ‘curse of dimensionality’ 43 due to the large number of parameters and the relatively small number of examples in the data set. Several approaches have been proposed for dimensionality reduction, including tests that are independent of the machine learning algorithm, such as Principal Component Analysis (PCA) 44 , which we used initially to investigate the level of redundancy in our parameter set. However, we were unable to clearly identify the most discriminative parameters using this approach. Therefore, we used an in-house ‘wrapper’ approach 44 , which incorporates the ILP learning process in the parameter selection process, and allows the identification of more complex relationships. As an example, we can cite the disorder probability is important for the characterization of deleterious mutations, but not for neutral mutations, and is in fact the most discriminative parameter in the ILP rules. We used the wrapper approach to evaluate the quality of the parameters.

First, we estimated the discriminative power of the parameters by calculating:

x = ∑ f n i − f p i

Where:

f n i = Relative frequency of negative examples for each value i of a given parameter

f p i = Relative frequency of positive examples for each value i of a given parameter

Second, we calculated the correlation matrix between all the parameters and defined a ‘correlation cutoff’, above which we considered two parameters to be significantly correlated. Third, we sorted the parameters by their discriminative power x, and for each parameter, we eliminated the significantly correlated parameters with lower x. Finally, we eliminated the parameters that had values of x lower than a given ‘x cutoff’, since these can be considered to be non-discriminating and eliminating them reduces the computational costs associated with large numbers of parameters. We tested different ‘correlation cutoffs’ in the range 0.5-0.9 with ‘x cutoff’ in the range 0.2-0.3, and measured the ILP prediction accuracy using a 10 fold cross-validation. The best performance (accuracy = 0.78) was achieved with a correlation cutoff of 0.6/0.7 and a x cutoff of 0.2 (data not shown). The correlation matrix and the reduced set of selected parameters are provided in Additional files 2 and 3.

One of the major concerns in such machine learning strategies is the problem of over-fitting. If the learned model is over-fitted to the training set, it will generally have poor predictive performance on the test data. In order to detect any potential over-fitting, we plotted the training accuracy and the testing accuracy as a function of the number of parameters in the model (Additional file 4). For a correlation cutoff <0.8, both the training and the testing accuracy increase with an increased number of parameters, indicating that the model is not over-fitted and thus, that the wrapper approach is effective. A small loss of training accuracy is observed however, for larger sets of parameters, corresponding to correlation cutoffs > =0.8.

The background knowledge used for the machine learning strategy consists of a large data set of positive and negative examples of NFS-Indels. The construction of a suitable background knowledge base is perhaps the most important step in the development of any prediction method. Our data set consists of 757 disease-causing/deleterious NFS-Indels (108 insertions and 649 deletions) and 757 neutral NFS-Indels (101 insertions and 657 deletions), collected from publically available databases.

In this section, we aim to deduce and describe some of the general reasons behind the pathogenicity of NFS-Indels in Mendelian diseases. In order to discover differences between deleterious and neutral NFS-Indels, we performed a chi-square test (95% confidence) for each parameter, where the null hypothesis is that there is no significant difference between the values of the parameter for deleterious and neutral variants. The results of these analyses are described below:

• Conservation: As shown in Figure 1, disease-causing NFS-Indels are more likely to occur at conserved sites than neutral NFS-Indels. This is true for both conserved single residue positions and conserved core blocks, although the difference is more significant for the ‘core block’ parameter extracted from the MACSIMS program.

• Functional annotations: As shown in Figure 2, disease-causing NFS-Indels are also more likely to occur at known functional sites. The most significant difference is observed for protein domains extracted from the Pfam database.

• Physico-chemical properties and their local perturbation: The 4 physico-chemical parameters associated with the amino acids in the NFS-Indel, as well as their perturbation in the local environment, are shown in Figures 3, 4, 5 and 6 (amino acid volume in Figure 3, hydrophobicity in Figure 4, polarity in Figure 5 and charge in Figure 6) for both average and total scores. The results are similar for the four properties and the two types of scores (average/total):

• Physico-chemical properties: Disease-causing NFS-Indels tend to be bigger, more hydrophobic, and more apolar than neutral NFS-Indels. The differences are more significant for the total values compared to the average values (higher p-values). For the total values, we also observe that deleterious NFS-Indels tend to be more positively charged than neutral Indels.

• Local Perturbation: The perturbation parameters reflect differences between the NFS-Indel and the flanking amino acids. Here, we consider the physico-chemical parameters of the NFS-Indel compared to: the site, the environment and the region (see Methods for details). Note that a negative perturbation value indicates that the average (total) value for the amino acids in the NFS-Indel is larger than for the surrounding residues and a positive value indicates a smaller average (total) NFS-Indel parameter. In general, disease-causing NFS-Indels tend to be more associated with positive and negative changes, while neutral NFS-Indels tend to have similar physico-chemical properties compared to the surrounding site. Nevertheless, we observe some exceptions, for example, for positive changes in the volume parameter, i.e. NFS-Indels that are smaller than the surrounding amino acids, the phenotype is more likely to be neutral. It should be noted that, in the case of polarity, the differences are less significant, compared to the other physico-chemical parameters.

• Structural annotations: As shown in Figure 7, neutral NFS-Indels tend to be exposed on the surface of the protein, in disordered regions and in coiled coils, while disease-causing NFS-Indels are more likely to be located in structured, buried regions and secondary structures. Concerning the type of secondary structure, deleterious NFS-Indels are more often found in strands than in helices.

• Others: As shown in Figure 8, deleterious NFS-Indels are more likely to be located in the central region of the protein and are longer than neutral NFS-Indels. No significant differences are observed when the NFS-Indel includes a proline, but deleterious NFS-Indels tend to contain a glycine more often than neutral NFS-Indels.

We used a 10-fold cross validation experiment to estimate the accuracy of our method, where the complete data set was randomly divided into 10 parts (nine parts for training, the rest for testing) and the process was repeated 10 times. For the 10 tests, our computational strategy achieves an average prediction accuracy on the test set of 79%, sensitivity of 89%, specificity of 69%, precision of 75%, negative prediction value of 85%, and Matthews Correlation Coefficient (MCC) of 0.59 (Table 5). We then identified the set of rules produced by the ILP algorithm that achieved the highest accuracy (83% on the test set, corresponding to fold 1). This final rule set is composed of 241 human-interpretable rules that represent new knowledge about the molecular perturbations underlying disease-causing NFS-Indels. The complete list of parameters used in this set are indicated in Table 1.

NPV: negative prediction value, MCC: Matthews Correlation Coefficient.

Finally, we compared the performance of KD4i with two recent methods for the prediction of the phenotypic effects of NFS-Indels, namely DDIG-in and SIFT-Indel. DDIG-in uses a Support Vector Machine (SVM) as the learning method, while SIFT-Indel uses a rule set derived from a decision tree algorithm. In order to provide a more direct comparison with the DDIG-in method, we have also trained a SVM on our dataset and parameters. The results are shown in Table 6. We observe that the accuracies obtained with DDIG-in and our SVM implementation are similar, despite the fact that the size of the data set used to train the DDIG-in algorithm is almost 4 times larger than the one used here, since we used only publically available data in our experiment. In comparison to SIFT-Indel, the KD4i ILP approach (based on the final rule set) achieves similar accuracy, but with higher sensitivity and lower specificity.

For the KD4i (ILP) implementation, the average performances in the 10-fold cross-validation and the performance of the final selected rule set (corresponding to fold 1 in Table 5) are provided. For DDIG-in and SIFT-Indel, the performances for combined insertions and deletions (as originally reported by the authors, for similar balanced data sets) are shown.

The higher sensitivity of KD4i is due in part to the power of the ILP approach, but also to the parameters used in the final set of rules. Indeed, all the tools tested here use different sets of parameters to characterize the NFS-Indels. As shown on Table 7, the disorder probability and the sequence conservation are important parameters for all three methods. However, the KD4i rule set includes other parameters that are used in only one or the other of the existing methods, such as the solvent accessible surface area (also used in DDIG-in) or location in a Pfam functional domain (also used in SIFT-Indel). In addition, we have highlighted the importance of some novel parameters, notably the local perturbation induced by the variant, in terms of amino acid volume for example.

KD4i is a rule-based system and the output is a prediction in binary form (deleterious/neutral). One major advantage of our approach is that the rules used to predict deleterious NFS-Indels are available in a human-interpretable format. In addition to providing useful information about the pathogenicity of specific NFS-Indels, it is possible to consider the prediction performance of a given rule independently of the rest. In order to identify the best performing rules (i.e. those resulting in predictions with the highest confidence), we ranked them according to two characteristics: coverage and precision.

Coverage is defined as the percentage of deleterious NFS-Indels in the training set that can be explained by a given rule. Given that we have 757 deleterious (positive) NFS-Indels in our data set and we have defined 241 rules, the expected average coverage is 3.2%. Values below this cutoff thus reflect rules that are more specific for a particular type of NFS-Indel, while larger values reflect more general rules. The average coverage of our rules is 2.8%, reflecting a slight tendency towards more specific rules. The maximum coverage is 7.4% and the minimum 0.8%.

Precision (also known as positive predictive value) is the proportion of positive test results that are true positives and reflects the probability that a mutation predicted to be deleterious is truly deleterious. For two rules with the same precision, we consider the best rule to be the one that has higher coverage. Since we set the noise option in the ILP algorithm to 0 (i.e. the rules cannot cover any negative examples), the precision of the rules in the training set is 100%. Figure 9 shows the precision of the rules calculated for the complete data set and for the test set only. If the test set is representative of the whole data set, the precision values should be similar. As expected, the precision of the rules observed in the test set is generally slightly lower than for the whole set, with less rules achieving 100% precision. The precision should be increased when the size of whole data set is increased and the precision in the test set should tend towards that observed for the whole data set, as was demonstrated in 45 .

The complete list of ranked rules is provided in Additional file 5, together with their coverage and precision statistics, and the top ranking rules are discussed in the following section.

Another way to estimate the most reliable predictions is to identify the NFS-Indels that are predicted to be deleterious by more than one rule. Table 8 shows the average precision as a function of the number of rules that cover a given prediction. We observe that the precision generally increases as the number of rules increases.

The KD4i method provides explicit rules that shed light on the reasons for the pathogenicity of specific NFS-Indels. In this section, we describe the three top ranking rules in the final rule set identified above (corresponding to fold 1).

1. Rule 20 (coverage 7.26%, precision 100%):

deleterious (A) if secondary_ structure (A, strand), block (A, true), local_perturbation_region_polarity_total (A, equal), relative_indel_position (A, middle).

This rule can be interpreted as: a NFS-Indel (A) is deleterious if it is located in a beta strand and in a conserved block, the polarity of the NFS-Indel is equal to that of the local region and the NFS-Indel is found in the central region of the protein. The rule thus indicates the specific conditions that determine NFS-Indel pathogenicity in beta strands. Indeed, we have observed that deleterious NFS-Indels occur more often than neutral NFS-Indels in secondary structure elements (Figure 7) and in particular in beta strands, probably due to their highly organized structure. However, KD4i is able to explain in more detail some of the reasons underlying this difference. In fact, almost half of the deleterious NFS-Indels in strands are found in the central region of the protein and in a conserved block. The prediction is completed by the specification of the local perturbation of the polarity in the region.

2. Rule 14 (coverage 7.00%, precision 100%):

deleterious (A) if pfam (A, true), domain (A, true), conserved_residue (A, false), local_perturbation_region_polarity_total (A, equal), indel_hydrophobicity_total (A, very_hydrophobic).

This rule describes NFS-Indels that are located within a Pfam or Uniprot domain, but do not affect a conserved residue. In this case, a very hydrophobic NFS-Indel can still have a deleterious effect on the protein structure or function.

3. Rule 19 (coverage 6.80%, precision 100%):

deleterious (A) if block (A, true), probability_of_disorder (A, structured), local_perturbation_region_polarity_total (A, equal), indel_hydrophobicity_total (A, very_hydrophobic), local_perturbation_in_site_volume_average (A, two_more).

The rule describes a NFS-Indel in a conserved block that is situated in a structured region of the protein. Here, the insertion/deletion of very hydrophobic and very large residues probably disrupts the organization of the local region.

The complete set of rules is provided in the Additional file 5.

In order to assess the reliability of our predictions, we analyzed population data from the 1000 Genomes Project 46 , including genomes of 1,092 healthy individuals from 14 populations. The majority of the variants observed in the 1000 Genomes are therefore expected to be neutral. However, several studies 47 48 have shown that healthy individuals may carry deleterious variants without any obvious phenotypic effects. As an example, Watson’s genome 49 has a well-known Alzheimer’s variant without apparent clinical effect. Nevertheless, these variants are expected to be observed with low frequencies at the population level. Conversely, variants found with higher frequencies are less likely to be deleterious. Therefore, we grouped the NFS-Indels in the 1000 Genomes by allele frequency (at 0.1 intervals) and compared these frequencies with the predictions produced by KD4i (Figure 10). As might be expected, the percentage of variants predicted to be deleterious decreases as the allele frequency increases (r = −0.82), thus confirming the pertinence of our predictions.

To illustrate the potential of the KD4i approach to infer new knowledge about individual variants, we have chosen a variant from the test set (fold 1), namely a deletion of an asparagine residue at position 526 (N526del) of the KIF5A protein. KIF5A (UniProt ID: Q12840) belongs to the microtubule-associated protein family of kinesins, that serve as molecular motors to distribute intracellular cargo along microtubules. KIF5A is expressed exclusively in neurons, and has been recently linked to hereditary spastic paraplegias (HSPs), a genetically heterogeneous group of neurodegenerative disorders characterized by progressive lower-limb spasticity and weakness. The HSP pathology is characterized by axonal degeneration of motor and sensory neurons 50 .

The variant (KIF5A_N526del) is predicted to be deleterious by 7 rules. We can compute the total precision, by taking the true/false positives for each rule and applying:

Precision total = ∑ True positives True positives + False positives

Since the precision is 100% and we have a large number of rules, we have high confidence in the prediction.

In order to extract more detailed information, we constructed a histogram of the usage of each parameter in this set of 7 rules. As shown in Figure 11, conservation-related (conserved residue and MACSIMS domain and block) and functional (Pfam site) parameters are largely represented. In addition, since the amino acid deleted is polar, hydrophilic and small, we hypothesize that the variant does not disrupt the structural folding process. We can conclude at this point that the deletion is probably a loss-of-function variant.

In the literature, the mutated protein, KIF5A_N526del is described as stable, but nonfunctional 50 . The mutation was subsequently shown to decouple microtubule binding of the motor, leading to inactivation of ATPase. Interestingly, drosophila larvae harboring either homozygous or hemizygous mutations in homologous genes of KIF5A exhibit aberrant neuronal intracellular anterograde trafficking of membranous organelles and accumulation in so-called “organelle jams”, that leads to neurodegeneration 51 .