The study was conducted on 142 unrelated patients with breast and/or ovarian cancer either with early onset or with a breast/ovarian cancer family history. Patients had been previously tested negative for a BRCA1/2 mutation, selected either for a predisposition probability higher than 70% according to the Claus model 2 or for enrichment in ovarian cancer cases: 87 patients (61%) had a personal or family history of both breast and ovarian cancer, 10 patients (7%) had a personal or family history of ovarian cancer only and 45 patients (32%) had a personal or family history of breast cancer only (Table 1). All patients attended a visit with a geneticist and a genetic counsellor in a family cancer clinic, mostly at the Institut Curie, Paris, France. Patients gave their informed consent for genetic testing. The study was approved by the local Ethics Committee in Institut Curie.

*Family history was defined in first- or second-degree relatives in the same lineage.

Genomic DNA was extracted from 2 mL whole-blood samples collected on EDTA with the Quickgene 610-L automated system (Fujifilm) according to the manufacturer’s instructions. RAD51 paralog mutation screening was performed on coding exons and exon-intron junctions by multiplex PCR and Enhanced Mismatch Mutation Analysis (EMMA) 21 except for 2 RAD51B exons which were analysed by simplex PCR and direct sequencing (Additional file 1: Table S1 and Additional file 2: Table S2). PCR products showing abnormal EMMA profiles were analysed by sequencing on an ABI PRISM 3130XL Genetic analyzer (Applied Biosystems).

RNA was extracted from lymphoblastoid cell lines using TRIzol reagent according to the manufacturer’s instructions (Invitrogen). 2 μg of total RNA from each sample was used for reverse transcription in a 40 μL reaction using the GeneAmp RNA PCR Core kit according to the manufacturer’s instructions (Applied Biosystems). cDNA was amplified with forward and reverse primers gcattcagcaccttcagctt and ctttcggtcccaatgaaaga for RAD51C exon 5 skipping, tgacctgtctcttcgtactcg and for RAD51C exon 8 skipping.

For RAD51B immunostaining, 4-μm-thick paraffin sections were cut and mounted on glass slides (Superfrost+, Menzel Glazer). Preparations were dried for one hour at 58°C, then overnight at 37°C. Sections were deparaffined with toluene and rehydrated with ethanol. Preparations were pretreated with citrate buffer (0.01 M citric acid pH 6.0), and a heat-based antigen retrieval method was used prior to incubations. Endogenous peroxidase was blocked using 3% hydrogen peroxidase solution for 5 minutes. The primary anti-RAD51B antibody used (clone NBP1-66539, dilution 1/200) was from Novus Biologicals. Sections were incubated for 15 minutes at 22°C with the primary antibody followed by staining with anti-rabbit HRP antibody (Leica Biosystems) for 10 minutes. Sections were then revealed in a diaminobenzidine solution for 15 minutes and stained with hematoxylin for 7 minutes.

Three bioinformatics tools were used for missense variants pathogenic prediction: Align-GVGD 22 23 , SIFT 24 25 and Polyphen-2 26 27 . Multiple sequence alignment (MSA) for Align-GVGD and SIFT analysis was an alignment of protein sequences of 11 species: Human (Homo sapiens), Chimpanzee (Pan troglodytes), Macaque (Macaca mulatta), Mouse (Mus musculus), Rabbit (Oryctolagus cuniculus), Dog (Canis familiaris), Cat (Felis catus), Bovine (Bos taurus), Opossum (Monodelphis domestica), Platypus (Ornithorynchus anatinus), Chicken (Gallus gallus) and Frog (Xenopus tropicalis). Missense variants were interpreted as likely deleterious if they were classified as deleterious or probably damaging by the three tools.

Frequencies of mutations and likely deleterious variants were compared between the cases and two control samples from online databases: European-American controls from Exome Variant Server 28 and European controls from 1000 Genomes project 29 . In a first step the two control sample variant frequencies were compared using Fisher’s exact test in order to check there was no significant difference. In a second step the control samples were pooled and the overall control variant frequency was compared with the case sample one, using Fisher’s exact test. All the tests were two-sided, with a p-value of 0.05 considered significant. Computations were performed using the XLSTAT-2013 software.

We identified a RAD51B mutation and a likely deleterious variant in two patients: the RAD51B c.452 + 3A > G mutation was detected in a breast and ovarian cancer family case and the RAD51B p.Arg159Cys variant was detected in a family with 3 breast cancer cases.

RAD51B has been previously evaluated as a candidate gene for breast cancer predisposition but no mutation was detected in a study of 188 multiple breast cancer family cases (Johnson et al. 20 ). The low frequency of RAD51B mutations may account for the differences observed between our results and those reported by Johnson et al., as previously described for RAD51C, RAD51D and XRCC2 16 17 18 . More generally concerning RAD51B involvement in cancer, previous studies have identified chromosomal rearrangements disrupting RAD51B in benign tumours, particularly uterine leiomyomas 31 32 . Overall, haploinsufficiency of RAD51B was shown to induce genomic instability in human cells, suggesting its involvement in cancer predisposition 33 . In addition, our findings must be interpreted in the context of two genome-wide association studies (GWAS) that identified the minor allele of single nucleotide polymorphisms (SNPs) in RAD51B acting as low risk factors for breast cancer: rs999737 34 and rs1314913 35 , located in RAD51B introns 10 and 7, respectively. Overall, these findings might indicate that RAD51B acts as a susceptibility factor or as a major gene depending on the context. Indeed, it cannot be excluded that the minor allele of these SNPs indirectly reflects a major influence of RAD51B 36 , as a recent study showed that high risk rare mutations can account for some synthetic associations identified by GWAS 37 .

RAD51B c.452 + 3A > G is a novel mutation. Several arguments strongly support its causality: this variation is absent in the thousands of controls tested in online databases (Exome Variant Server 28 , dbSNP 38 , 1000 Genomes 29 ); in silico prediction concluded this variation was likely to result in an out-of-frame exon skipping leading to a truncated or unstable protein; RAD51B immunohistochemistry in breast carcinoma cells of the patient bearing this variation showed a loss of expression of RAD51B.

The RAD51B p.Arg159Cys variant is reported in Exome Variant Server: this variant was detected in 2 out of 4,299 controls in European-American populations. We consider this variant to be a likely deleterious variant because it occurs in a functional domain and results in replacement of a highly conserved amino acid with subsequent high Grantham score and Align-Grantham Variation Grantham Deviation (Align-GVGD) maximum score (C65), and very low frequencies are reported in Exome Variant Server. Its occurrence in controls could be explained by an incomplete penetrance.

Overall, for the RAD51B gene, one truncating mutation and one likely deleterious variant were detected in 2 out of 142 patients selected for enrichment in breast/ovarian cancer cases. In online databases, one truncating mutation and two likely deleterious variants were detected in 4 out of 4,678 controls (Additional file 4: Table S4). Frequency of RAD51B variants was significantly higher in cases (p = 0.012), which suggests RAD51B variants are associated with breast/ovarian cancer risks.

The XRCC3 p.Arg150Cys variant was detected in a family with 1 ovarian cancer and 5 breast cancer cases. Like the RAD51B missense variant reported in this study, the XRCC3 p.Arg150Cys variant is reported with a very low frequency in Exome Variant Server (1 out of 4,276 controls in European-American populations). We consider this variant to be a likely deleterious variant because it occurs in a functional domain and results in replacement of a highly conserved amino acid with subsequent high Grantham score and Align-GVGD maximum score (C65).

This study is the first report of XRCC3 mutation screening in breast and ovarian cancer predisposition. Numerous association studies have evaluated XRCC3 SNPs as candidate risk factors for breast cancer, but the results of these studies remain controversial. A recent meta-analysis suggested that the minor allele of XRCC3 p.Thr241Met SNP was a low risk factor and XRCC3 IVS5-14A > G SNP a low protective factor for breast cancer 39 .

Several studies have reported RAD51C causal mutations in breast and ovarian cancer predisposition 16 40 41 . Two novel RAD51C splicing mutations are reported in this study: RAD51C c.1026 + 5_1026 + 7del mutation is truncating, resulting in an out-of-frame exon 8 skipping and RAD51C c.706-2A > G mutation leads to the loss of 44 amino acids in a functional domain of the protein by an in-frame exon 5 skipping. These two RAD51C mutations were detected in families with both breast and ovarian cancer cases, which is consistent with previous studies. As this set of patients was enriched with ovarian cancer cases and due to the low frequency of RAD51C mutations, other studies must be conducted in larger series to evaluate whether RAD51C confers predisposition to ovarian cancer alone or to both breast and ovarian cancer, which remains controversial 42 .