Collection of individual patient data on Case Report Forms (CRFs) in clinical research has traditionally been done by investigators in their offices summarizing medical charts on paper forms (pCRFs), a tedious method that could result in data errors and wrong conclusions 1 2 . Electronic data capture has in recent years been increasingly used in both industry and academic research settings 3 4 . The feasibility of electronic CRFs (eCRFs) has been documented by numerous studies analyzing data collected on websites, laptops or digital pens 5 6 7 8 9 10 . Since the mid-1980s, eCRFs have increased data quality and completeness by using alarms, automatic completions and reminders 11 12 , reducing losses and transport logistics, especially for multicenter trials 13 . Moreover, use of eCRFs permits speedier database processing and shorter study periods, resulting in lower costs 7 11 14 15 16 17 18 19 . Previous studies of eCRFs have primarily focused on the investigators’ point of view, while few have documented the perspectives of the other stakeholders 5 6 20 21 .

Despite their demonstrated usefulness, eCRFs have not become dominant 3 . Welker has identified some of the barriers to their dissemination: the lack of available on-site technology, insufficient assistance by Information Technologies staff or software providers, investigators’ lack of motivation and interference of eCRFs with their clinical tasks, complexity of installation and maintenance of the software and high investment cost 22 . The labor cost of data entry is transferred from clerks to investigators who may see few tangible benefits apart from a better quality of data and speedier study completion. While some investigators have integrated the electronic interface into their medical practice to the point of making it an asset 23 , the majority views the implementation of an eCRF in a paper-based working environment as a source of redundancy 19 24 .

Our objective was to formally describe the efficiency (measured by satisfaction, cost and duration of the study) of electronic and paper CRFs in the context of biomedical research conducted in hospitals.

The primary endpoint was the satisfaction of stakeholders in a clinical study: investigators, clinical research associates (CRAs) and data managers (DMs). Secondary endpoints were costs and duration of the studies. Our hypotheses were that eCRFs save cost and time 18 22 , and would be preferred by those 3 stakeholders 5 .

We included 27 clinical studies, 16 (59%) using pCRFs and 11 (41%) eCRFs.

The main characteristics of the studies are summarized in Table 1. Electronic CRFs studies were mostly large multicenter, national and phase 3 clinical trials while pCRFs studies were trials with few patients and centers. The majority of pCRFs were drug trials (15/16; p = 0.036), and eCRFs were more often used in trials with a significantly higher number of patients (355 vs. 60 patients in pCRFs; p = 0.014) and fewer data (17 vs. 39 pages with pCRFs; p = 0.027). The number of patients was the only explanatory variable for CRF choice (Table 2).

*Significant difference between eCRFs and pCRFs (p < 0.05).

^αFrom level A = minimal, e.g. trial involving only additional blood sample collection, to D = major risk, e.g. trial of innovative therapies, phase I or II trials.

^βNumber of variables in database = number of patients x number of variables in CRF.

The modeled probability is the choice of an eCRF for the study as opposed to a pCRF.

In this first description of the use of eCRFs and pCRFs across 27 clinical studies, we found that most stakeholders were satisfied with eCRFs and that the use of eCRFs was associated with shorter study duration and lower cost per patient. Average duration of recruitment did not differ between pCRF and eCRF studies, despite a greater number of trials investigating rare and pediatric conditions in the pCRF group.

Data managers reported that eCRFs saved time and improved data quality, however we cannot exclude that the perception of fewer queries simply results from a smaller number of variables monitored; clinical research associates valued the automated quality checks and easier storage of electronic data. Both DMs and CRAs preferred eCRF for multicenter trials. Improvements are needed to facilitate the integration of eCRFs into clinical practice, including widespread adoption of portable devices such as digital pens 6 or graphic tablets, which one investigator described as “the solution for the future, and already used very successfully in anesthesia ”.

It appeared that stakeholders’ characteristics do not predict preferences, except for young and computer literate investigators who tended to be more demanding with eCRFs, probably because they had greater expectations of eCRF and yet were more aware of its limitations.

In addition to exploring stakeholder perspectives, we found that eCRFs were mainly used in large, national and multicenter trials, whereas pCRFs were used in small high risk drug trials, possibly because of the greater reliability of written documents.

The few studies investigating stakeholders’ experiences of CRFs have mostly focused on investigators and found a high level of support for eCRFs 24 . We included two additional key stakeholders and revealed more mixed results in the level of satisfaction and preferences regarding eCRFs. CRAs agreed with the Litchfield hypothesis 5 that monitoring would be more efficient with an eCRF. Moreover, as expected 17 26 , acceptance of eCRFs by investigators remained one of their biggest challenges. López-Carrero reported very positive reviews of eCRFs by investigators, with three-quarters finding data entry easier and more than a half saying that it decreased workload. We did not find investigators to be so optimistic about eCRF capacities, particularly with respect to data entry and workload. The fact that the López-Carrero 21 investigators were asked about a pilot project with a “specifically designed” eCRF may explain our less enthusiastic responses.

While literature reviews have found favorable results for eCRFs in terms of study duration and costs, we found that the lower cost per patient was explained by the large patient number in eCRF trials. This may be attributable to our sample which did not include pCRF studies with large patient numbers, or eCRF studies with small patient numbers and/or to the fact that most of the other cost studies were based on models 18 or used ad-hoc prototypes 27 28 .

The retrospective non randomized design of our study was dictated by two major issues: time constraints and acceptability. Randomization would not have been acceptable to investigators who want to be able to make their own choices and it also would not have been able to show that eCRF and pCRF have their specific indications and are not therefore perfect substitutes. There is a kind of ‘indication bias’ as shown both in Table 1 and in the stakeholders’ responses. Indeed we have shown that for some studies pCRF is cheaper and appears more trustworthy. In addition, unobserved factors must also influence the choice between eCRF and pCRF as can be assumed by, for example, the fact that the large difference in the number of variables between the 2 collection methods was not correlated to any of the characteristics of the studies that we investigated.

Whilst the CRA and Investigator questionnaires response rates were low, due to many stakeholders having changed institutions and no longer being locatable, they emanated from 9/10 AP-HP units and 20 different hospitals spread over the French territory. Our small sample of respondents is mirrored in other studies, López-Carrero et al. 21 had a similar problem with only 27 investigators answering from 33 centers, and Lium et al. 24 interviewed 18 physicians, but from only 2 centers.

Most clinical trials sponsored by the AP-HP currently use CleanWEB for eCRFs. As a result, the stakeholders’ answers were related to that software. Our results may not adequately reflect the current situation of eCRF since half of the studies using eCRFs had started between 2004 and 2006 and are not representative of the current capacities of electronic data capture. Recent adaptations of CleanWEB include certification with Clinical Data Interchange Standards Consortium CDISC^a, the possibility to connect to the server on-line or enter data off-line and direct data export to SAS, which may improve future satisfaction with eCRFs.

We were unable to suggest a financial advantage in using eCRFs in trials with fewer than 50 patients, given that the smallest eCRF trial had 47 patients. Nonetheless, CleanWEB price discounts for monocentric eCRFs could make it cost-efficient, even for small trials. Likewise, we were not able to draw any conclusions regarding pCRF trials with more than 700 patients. We did not take into account the annual maintenance package of 125,580€, billed to the institution for all studies using CleanWEB. We used bootstrap replications for the cost analysis because of the small number of studies and because calculation of means after log transformation resulted in a comparison of geometric mean costs instead of arithmetic means. The bootstrap method requires the true distribution of the data to be adequately represented by its empirical distribution 25 . Finally, as we did not select the studies based on their characteristics, we had much heterogeneity in terms of risk level and trial phase.

An important aspect in comparing pCRFs and eCRFs that we have only touched upon in our questionnaires is the management of the delivery of drugs and placebo in pharmaceutical trials. Computerized treatment management programs that may be included in eCRF software enable investigators to streamline complex processes, including: managing the supply of medication to the centers even for long-term treatments, managing intricate protocols and overseeing product expirations. This system manages stock control and prevents waste, thus preventing breaks in inclusions due to supply ruptures and avoiding mistakes in treatment delivery to the patient.

This study examined eCRFs and pCRFs from the viewpoint of investigators and other important stakeholders. It found that eCRFs and pCRFs are used in studies with different patient numbers, center numbers and risk. The first ones are more advantageous in large, low–risk studies and gain support from a majority of stakeholders. Our findings also suggest that eCRF and pCRF may not be substitutes but complement each other with their own specific indications. The choice between paper and electronic CRF is a significant step in the design and execution of clinical studies; it should be discussed with the involved stakeholders and based on efficiency.

^aCDISC: standards of acquisition, exchange, submission and archive of clinical research data that enable information system interoperability to improve medical research.

eCRF: Electronic case report form; pCRF: Paper case report form; CRAs: Clinical research associates; DM: Data managers; AP-HP: Assistance Publique-Hôpitaux de Paris; CDISC: Clinical data interchange standards consortium.

The authors declare that they have no competing interests.

ALJ, CA and IDZ conceived and designed the experiment. ALJ performed the experiment. ALJ and CQ realized the acquisition and management of data, and analyzed the data. ALJ, CQ and IDZ wrote the paper. All authors read, commented and approved the final manuscript.