Skip to main content

Open data 5 years on: a case series of 12 freedom of information requests for regulatory data to the European Medicines Agency

Abstract

Background

Clinical trial (and other) data from the European Medicines Agency (EMA) offers the best available opportunity to address the extensive reporting bias in pharmaceutical trial literature. Data are requested via freedom of information requests, but 5 years on, little is known about how the system is working.

Methods

Case series of 12 requests for regulatory data (clinical study reports and other regulatory data) relating to 29 different compounds. We logged start and end dates for correspondence with and data releases from the EMA, the need for additional correspondence and appeal of initial negative decisions, and inspected data releases for redaction. We measured: time from initial request to first substantive response from the EMA, to final decision from the EMA (in case of appeal), to initial receipt of documents, and to completion of request; number of data transmission batches generated; number of pages received for each request; average number of pages per batch over time (for releases in multiple batches); judgment as to whether the request was satisfied.

Results

We found great variability in time to receive an initial decision from the EMA (1 to 13 weeks). Additional correspondence with the EMA was necessary in 10 of 12 requests. Four of 12 were initially refused but 3 of 4 were allowed on appeal after 3 to 33 additional weeks. One request was denied despite appeal. Time to final decision was 1 to 43 weeks. We received data for 11 of 12 requests in 98 batches. While two requests remain outstanding as at June 2015 the remaining nine requests took a median 43 weeks to completion (range: 17 to 186 weeks). Despite redaction in 10 of 11 releases (mainly of researcher and participant identifying information), 8 requested were wholly satisfied.

Conclusions

The EMA is the only regulator in the world that is routinely releasing original clinical trial data, but release can take considerable time to occur and often only after a lengthy correspondence. Given its importance for research and significance for transparency we suggest ways in which the process could be made more efficient.

Peer Review reports

Background

For those engaged in research synthesis, regulatory data not traditionally publicly available are a crucial weapon to avoid or minimize the impact of reporting bias. These data, typified by clinical study reports ranging from hundreds to thousands of pages, provide a wealth of auditable details on clinical trials well beyond the most extensive journal publications [1].

While calls to address the many problems of reporting biases go back decades [2], the past 5 years have witnessed notable progress in the range of campaigns [3, 4], policies [5, 6], statements and structures [711] established to expand public access to clinical trial data.

Among all the data transparency initiatives, one process remains unique: that of the European Medicines Agency (EMA). Since November 2010, the EMA as data holder has been providing requestors with clinical trial data relating to compounds on which a decision has been made under the centralized registration procedure [5].

The EMA has two distinct policies affecting public access to clinical trial data in its holdings, one on request [12, 13] and the second for data in marketing authorization applications (MAAs) regarding MAAs submitted on or after 1 January 2015 (with web data release to start in 2016) [14].

Since 2010, the EMA has released over two million pages of regulatory data on request [15, 16], mainly to industry [16]. The number of requests has grown over time, from 20 requests per month during the first 2 years to nearly double that in the 6-month period after, but pages released in this time period decreased from around 70,000 pages per month to 44,000 [16].

In September 2013, the EMA expanded its ability to deal with the growing number of requests, dedicating a full single 12-person team to a new “Access to Documents Service” (ATD). When the service began in 2010, five EMA staff who had other job responsibilities unrelated to disclosure handled all requests. Following the reorganization, the new ATD team had 12 full-time employees (personal communication, Anne-Sophie Henry-Eude, 17 November 2014). In 2014, the EMA also released a guide to help requestors [17].

However, as repeated requestors and recipients of data from the EMA, our experience has been one of increasing delay and complexity.

We report on the details and outcome of 12 requests for regulatory data made by us to the EMA between 2011 and June 2015.

Methods

Types of requests

We examined all requests we submitted to the EMA between November 2010 (when the new EMA policy went into force) and 30 May 2015, regardless of their fate (acceptance, partial rejection, or complete rejection). We included all requests for trial data (the text of the trial itself with its protocol and amendments, analysis plans and tables of results – the clinical study report – and other documents used by regulators related to trials such as parts of submissions for the MAA based on trials). We included animal study data. We treated each request as a separate case, thereby constructing our own case series of requests.

Following the submission and receipt of a request, the Agency may additionally seek clarification, such as which specific data are being requested, and in what priority. If the EMA decides to grant the request, it will consult with the marketing authorization holder of the relevant product prior to release of data. If the EMA rejects a request, it allows the requestor 15 working days to appeal this decision. The EMA may grant certain portions of a request, but reject others (again, allowing appeal).

We extracted information across two domains: correspondence and data releases.

Extraction of correspondence

We logged all electronic correspondence and linked data releases to us from the EMA. We recorded the type of documents requested, the initial date of each request and the date of initial EMA approval or rejection of each request.

We extracted and tabulated data on a simple spreadsheet on the object and date of the initial request and answer, in what professional capacity we made the request (academic researcher or journalist), the presence, count and timelines of appeals and their resolution, and whether additional correspondence took place including clarifications, request modifications and prioritization of releases from a list of EMA holdings.

Extraction of data releases

We recorded the number of pages released and total number of batches (release of data on a given date) in which documents were released. Larger requests are typically split into multiple batches with no single batch fulfilling the scope of the overarching request.

We extracted the start and finish date of each data release. For requests ongoing as of 1 June 2015, we used the most recent data release date prior to this date. We also recorded presence and type of redactions and omission of entire sections of clinical study reports.

Analysis

We calculated the length of time from initial request to: initial decision by the EMA, final decision (in case of appeals), initial receipt of documents, and completion of request (or most recent data release, in case of ongoing requests).

We calculated the number of data transmission batches our requests generated, the total number of pages for each request and the average number of pages per batch over time, for requests that were split into multiple batches.

Finally, we made a subjective judgment as to whether our request was satisfied (i.e., what we received matched what we requested), recorded as either “Yes,” “Mostly,” or “No.”

We did not seek ethical approval of our research as it is not research on human subjects.

Results

We made 15 freedom of information requests to the EMA between January 2011 (the date of our earliest request) and 1 June 2015 (our cutoff date). Three requests were excluded from our analysis (two for correspondence unrelated to trial data and one because of submission only 4 days prior to our cutoff date). One request was for documents relating to an EMA review of safety data from a single company on a range of its products [18]. The remaining 11 requests related to 29 compounds: 18 antibiotics, 2 antivirals, 1 monoclonal antibody, 4 vaccines, 1 volume expander, 1 antipsychotic, 1 statin, and 1 antidiabetic (Table 1). Nine of the 12 requests were made in our capacity as academic researchers; 3 were made in our capacity as journalists.

Table 1 Table of 12 included requests in chronological order

Table 2 shows the great time variability to receive an initial decision from the EMA (1 to 13 weeks). Ten of 12 requests entailed additional correspondence with the EMA beyond the initial request, for example to clarify the scope of the EMA’s holdings for a particular therapeutic.

Table 2 Timeline of correspondence milestones

After clarification, 4 of the 12 requests were initially rejected. Reasons included then-active legal proceedings and confidentiality regarding an ongoing regulatory procedure. Examples of request, rejection, and appeal letters can be seen at http://www.bmj.com/tamiflu/ema/. We appealed in all cases, and received a positive decision for three of four appeals in 3 to 33 additional weeks. The time to receive a final decision, therefore, took between 1 and 43 weeks.

In total, the EMA sent us data in response to 11 of 12 requests. The EMA began sending data for half of requests within 9 weeks. However, for the other half, it took between 15 and 58 weeks before any data were received. Two requests were outstanding at our cutoff date. Of the remaining nine requests, it took a median 43 weeks to between initial request to final receipt of data (range: 17 to 186 weeks) (Table 2).

There was great variability in the length of released documents. Clinical study reports ranged from 7315 to 25,453 pages. Releases delivered in batches were divided into no more than 5 batches before the year 2013, but afterwards were divided into between 10 and 22 batches. We observed no relationship between the overall number of pages released and number of batches but requests since 2013 were generally delivered in a greater number of smaller batches. There was no relationship between the number of batches and number of clinical study reports. Our first request was released in 4 batches and contained a total of 16 clinical study reports (average 6363 pages per batch), whereas our eighth request for 3 clinical study reports was delivered across 12 batches (average 1733 pages per batch) (Table 3; Fig. 1).

Table 3 Released documents
Fig. 1
figure 1

Timeline of 12 requests along four milestone dates (initial request, initial decision, initial release of data, and final release of data). Data points indicate, from left to right: initial request, initial decision, initial release of data, and final release of data

Only our first request was released without any redaction. All others had variable amounts and types of redactions. Names and other potentially identifying details of researchers and participants (e.g., participant ID number) was the most common—but inconsistently applied—type of redaction. Other details redacted in some cases included procedures or methods (e.g., histology testing details), formulation lot numbers, and the ID numbers of other studies. Some redactions were so extensive that we were unable to guess what information was redacted. Despite the redactions, we judged that in 8 of 12 cases the releases matched our request, 2 cases of partial match, and 2 cases of no match.

The dataset underlying this analysis is available, in Microsoft Excel format, as an online supplement (see Additional File 1).

Discussion

With little exception, the US Food and Drug Administration treats clinical study reports and other parts of the dossier submitted by sponsors as commercial confidential information and, therefore, not releasable under the US Freedom of Information Act. In contrast, the EMA interprets all documents, including clinical study reports, to be subject to its “reactive” freedom of information policy and is the only regulator in the world routinely releasing such data. However, the agency is dealing with a huge and growing number of requests [15, 16].

We made our first request to the EMA in January 2011, 2 months after the promulgation of its then new policy. We requested clinical study reports for inclusion in our Cochrane review of neuraminidase inhibitors [19]. Four months later, we reached this goal, receiving 16 unredacted clinical study reports for oseltamivir. These documents offered a wealth of detail greatly surpassing any journal-based trial publication. We became quick advocates of using regulatory data for research synthesis because of its powerful potential to address reporting bias in journal publications [20].

However, as time passed we also witnessed considerable heterogeneity in times and complexity of the request process while batches tended to become smaller and fulfillment of requests took longer. Whereas in 2011, we received 16 clinical study reports (25,453 pages) across 4 batches in 4 months, a recent request for clinical study reports for another product has so far been delivered across 22 batches in almost a year, with an average of 289 pages per batch. The fewer pages per batch imposes an administrative burden on requestors as clinical study reports may be split across multiple files and multiple batches but must be manually combined, to create a complete report, necessary to carry out an assessment. Second, the tight appeal time (15 working days) imposes a fair and timely evaluation of whether redactions are reasonable for each batch of documents. We found this task hard to keep up with. We suggest keeping a careful and up-to-date request log to avoid losing data or duplicating the request, both of which happened to us.

Long data access request and receipt timelines may provide an obstacle for researchers accustomed to readily accessible journal publications and encourage neglect of what is as near a “gold standard” of evidence completeness as is currently possible. Some researchers may choose to forgo a chance to address reporting bias in their work and in doing so perpetuate the known problems in contemporary scientific publications.

A second concern is that the length of the EMA’s process may lead some to argue against the EMA’s approach itself and in favor of other data access initiatives such as ClinicalStudyDataRequest.com, a joint initiative of 12 pharmaceutical companies. While we support all efforts in this area, the EMA’s approach remains unique, and is based on the view that it is the regulator’s duty to make available data underlying decision-making for all drugs in its purview. We concur, and believe this approach to be self-evident for any scientific body operating within a political democracy as the EMA does not screen requests by requiring a reason for accessing the data (but a rationale for access when appealing a negative decision is prudent).

We remain firm advocates of broader use of clinical study reports but users should also be aware of EMA-unrelated complexities in the most basic first step: identifying a trial. In some cases, a trial can have up to four identifiers: two different alphanumeric identifiers given by the manufacturer, a registry number and sometimes an acronym. Because the EMA, the manufacturer and registry operators do not necessarily provide tables of holdings that cross-reference these IDs, this can be a further source of confusion and resource wastage.

Over time, CSRs available under the EMA’s “reactive” policy may become less important as they relate to progressively older pharmaceuticals and the EMA’s “prospective” policy of publishing clinical trial data applies to a greater number of drugs in current use. However, the EMA’s reactive policy will remain of great interest for the foreseeable future, especially should the prospective policy falter.

Simple measures may at least partially address some of the problems we identified. First, a permanent forum between the EMA and requestors may help each side maximize the efficiency of the data request/release process through mutual education and exchange of views.

Second, a publicly available list of holdings by compound name would focus requests and avoid unnecessary correspondence in identifying desired materials. On several occasions the EMA offered us a list of its holdings relating to our requests. These proved invaluable in prioritizing and focusing our requests. Although these lists were custom-created for our requests, publicly available lists across EMA holdings would help clarify requests up front and moderate expectations. One tangible way to support the EMA’s efforts would be to call for volunteer researchers to help build a list of EMA holdings, perhaps in specific topic areas.

Third, release letters could be sent in the body of an email instead of the current practice of email attachments and web-based downloads. This would ensure their searchability and ease of reading. Sections that are generic to all letters should be reduced as much as possible and made visually distinct from the portions that require the requestor’s attention.

Conclusions

The EMA is the only regulator in the world that is routinely releasing part of its holdings, but our experience shows that document release can take considerable time to occur and often only after a lengthy correspondence. Despite the problems, the EMA’s unique efforts should not be undermined. Independent researchers and users should instead support the policy as at present it is the most promising way to address reporting bias in literature. In this report, we have identified areas for improving the system. This case series details our experience with the EMA, but we would like to hear from researchers who approached industry.

Abbreviations

ATD:

Access to Documents Service

CHMP:

Committee for Medicinal Products for Human Use

CSR:

Clinical Study Report

CTD:

Common Technical Document

EMA:

European Medicines Agency

ICH:

International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use

References

  1. Doshi P, Jefferson T. Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports. BMJ Open. 2013;3(2):e002496.

    Article  PubMed Central  PubMed  Google Scholar 

  2. Chalmers I. Underreporting research is scientific misconduct. JAMA. 1990;263(10):1405–8.

    Article  CAS  PubMed  Google Scholar 

  3. AllTrials. All trials registered. All results reported [Internet]. AllTrials. Available from: http://www.alltrials.net/. Accessed 18 August 2015.

  4. Spelsberg A. Petition · Berliner Erklärung 2012: An die Bürgerinnen und Bürger in Europa: Die Verheimlichung von klinischen Studiendaten stoppen · Change.org [Internet]. 2012. Available from: https://www.change.org/p/berliner-erkl%C3%A4rung-2012-an-die-b%C3%BCrgerinnen-und-b%C3%BCrger-in-europa-die-verheimlichung-von-klinischen-studiendaten-stoppen. Accessed 18 August 2015.

  5. European Medicines Agency. European Medicines Agency policy on access to documents (related to medicinal products for human and veterinary use) POLICY/0043 [Internet]. 2010. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2010/11/WC500099473.pdf. Accessed 18 August 2015.

    Google Scholar 

  6. PhRMA, EFPIA. Principles for responsible clinical trial data sharing [Internet]. 2013. Available from: http://phrma.org/sites/default/files/pdf/PhRMAPrinciplesForResponsibleClinicalTrialDataSharing.pdf. Accessed 1 August 2013.

    Google Scholar 

  7. Institute of Medicine. Sharing clinical trial data: maximizing benefits, minimizing risk. Washington, DC: The National Academies Press; 2015.

    Google Scholar 

  8. Barron H, Rosenblatt M. Access to patient-level trial data. N Engl J Med. 2014;370(5):485–6.

    Article  CAS  PubMed  Google Scholar 

  9. Nisen P, Rockhold F. Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med. 2013;369(5):475–8.

    Article  PubMed  Google Scholar 

  10. Moorthy VS, Karam G, Vannice KS, Kieny M-P. Rationale for WHO’s new position calling for prompt reporting and public disclosure of interventional clinical trial results. PLoS Med. 2015;12(4):e1001819.

    Article  PubMed Central  PubMed  Google Scholar 

  11. World Health Organization. WHO Statement on Public Disclosure of Clinical Trial Results [Internet]. WHO. 2015. Available from: http://www.who.int/ictrp/results/reporting/en/. Accessed 18 August 2015.

  12. European Medicines Agency. Send a question to the European Medicines Agency [Internet]. 2015. Available from: http://www.ema.europa.eu/ema/index.jsp?curl=pages/about_us/landing/ask_ema_landing_page.jsp&mid=WC0b01ac05806499f0. Accessed 18 August 2015.

    Google Scholar 

  13. European Medicines Agency. Release of data from clinical trials [Internet]. 2012. Available from: http://www.ema.europa.eu/ema/index.jsp?curl=pages/special_topics/general/general_content_000555.jsp&mid=WC0b01ac0580607bfa. Accessed 27 November 2012.

    Google Scholar 

  14. European Medicines Agency. European Medicines Agency policy on publication of clinical data for medicinal products for human use [Internet]. 2014. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2013/06/WC500144730.pdf. Accessed 24 June 2013.

    Google Scholar 

  15. Doshi P, Jefferson T. The first 2 years of the European Medicines Agency’s policy on access to documents: secret no longer. JAMA Intern Med. 2013;173(5):380–2.

    Article  PubMed  Google Scholar 

  16. Doshi P. Transparency interrupted: the curtailment of the European Medicines Agency’s policy on access to documents. JAMA Intern Med [Internet]. 2013. Available from: http://dx.doi.org/10.1001/jamainternmed.2013.9989. Accessed 20 August 2013.

    Google Scholar 

  17. European Medicines Agency. Guide on access to unpublished documents [Internet]. 2014. Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2013/06/WC500144730.pdf. Accessed 17 August 2015.

    Google Scholar 

  18. European Medicines Agency. European Medicines Agency finalises review of medicines concerned by Roche pharmacovigilance inspection [Internet]. 2013. Available from: http://www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/news/2013/11/news_detail_001962.jsp&mid=WC0b01ac058004d5c1. Accessed 18 August 2015.

    Google Scholar 

  19. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2012;1:CD008965.

    PubMed  Google Scholar 

  20. Doshi P, Jones M, Jefferson T. Rethinking credible evidence synthesis. BMJ. 2012;344:d7898.

    Article  PubMed  Google Scholar 

  21. European Medicines Agency. European Medicines Agency starts infringement procedure to investigate Roche’s alleged non-compliance with pharmacovigilance obligations [Internet]. 2012. Available from: http://www.ema.europa.eu/ema/index.jsp?curl=pages/news_and_events/news/2012/10/news_detail_001641.jsp&mid=WC0b01ac058004d5c1. Accessed 7 November 2012.

    Google Scholar 

Download references

Acknowledgements

PD gratefully acknowledges the American Association of Colleges of Pharmacy for its funding support (US$10,000) for a separate study to analyze written medical information regarding the possible harms of statins. AACP had no involvement in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of this manuscript. This study had no funding.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Peter Doshi.

Additional information

Competing interests

PD and TJ were co-recipients of a UK National Institute for Health Research grant (HTA – 10/80/01 Update and amalgamation of two Cochrane reviews: neuraminidase inhibitors for preventing and treating influenza in healthy adults and children: http://www.nets.nihr.ac.uk/projects/hta/108001). This review relied, at one point, on clinical study reports received from the EMA. In addition, PD received €1500 from the European Respiratory Society in support of his travel to the society’s September 2012 annual congress in Vienna, where he gave an invited talk on oseltamivir. PD is also an associate editor at The BMJ. TJ receives royalties from his books published by Blackwell and Il Pensiero Scientifico Editore, Rome. TJ is occasionally interviewed by market research companies for anonymous interviews about phase 1 or 2 pharmaceutical products. In 2011–2013, TJ acted as an expert witness in a litigation case related to oseltamivir phosphate (Tamiflu, Roche), and in a labor case on influenza vaccines in healthcare workers in Canada. In 1997–1999 TJ acted as a consultant for Roche, in 2001–2002 for GSK, and in 2003 for Sanofi-Synthélabo for pleconaril (an antirhinoviral agent, which did not get approval from the Food and Drug Administration). TJ was a consultant for IMS Health in 2013, and in 2014 was retained as a scientific adviser to a legal team acting on the drug oseltamivir (Tamiflu, Roche). In 2014–2015 TJ was a member of two advisory boards for Boerhinger and is in receipt of a Cochrane Methods Innovations Fund grant to develop guidance on the use of regulatory data in Cochrane reviews. TJ has a potential financial conflict of interest in the investigation of the drug oseltamivir. TJ is acting as an expert witness in a legal case involving the drug oseltamivir (Roche). TJ is a member of an Independent Data Monitoring Committee for a Sanofi Pasteur clinical trial.

Authors’ contributions

PD and TJ contributed equally to all aspects of this work. Both authors read and approved the final manuscript.

Additional file

Additional file 1:

Study data extraction sheet. (xlsx 31 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Doshi, P., Jefferson, T. Open data 5 years on: a case series of 12 freedom of information requests for regulatory data to the European Medicines Agency. Trials 17, 78 (2016). https://doi.org/10.1186/s13063-016-1194-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13063-016-1194-7

Keywords