Skip to main content

Completeness and accuracy of national cancer and death registration for outcome ascertainment in trials—an ovarian cancer exemplar



There is a trend to increasing use of routinely collected health data to ascertain outcome measures in trials. We report on the completeness and accuracy of national ovarian cancer and death registration in the United Kingdom Collaborative Trial of Ovarian Cancer Screening (UKCTOCS).


Of the 202,638 participants, 202,632 were successfully linked and followed through national cancer and death registries of Northern Ireland, Wales and England. Women with registrations of any of 19 pre-defined ICD-10 codes suggestive of tubo-ovarian cancer or notification of ovarian/tubal/peritoneal cancer from hospital episode statistics or trial sites were identified. Copies of hospital and primary care notes were retrieved and reviewed by an independent outcomes review committee. National registration of site and cause of death as ovarian/tubal/peritoneal cancer (C56/C57/C48) obtained up to 3 months after trial censorship was compared to that assigned by outcomes review (reference standard).


Outcome review was undertaken in 3110 women on whom notification was received between 2001 and 2014. Ovarian cancer was confirmed in 1324 of whom 1125 had a relevant cancer registration. Sensitivity and specificity of ovarian/tubal/peritoneal cancer registration were 85.0% (1125/1324; 95% CI 83.7–86.2%) and 94.0% (1679/1786; 95% CI 93.2–94.8%), respectively. Of 2041 death registrations reviewed, 681 were confirmed to have a tubo-ovarian cancer of whom 605 had relevant death registration. Sensitivity and specificity were 88.8% (605/681; 95% CI 86.4–91.2%) and 96.7% (1482/1533, 95% CI 95.8–97.6%), respectively. When multiple electronic health record sources were considered, sensitivity for cancer site increased to 91.1% (1206/1324, 95% CI 89.4–92.5%) and for cause of death 94.0% (640/681, 95% CI 91.9–95.5%).

Of 1232 with cancer registration, 8.7% (107/1232) were wrongly designated as ovarian/tubal/peritoneal cancers by the registry and 4.0% (47/1172) of confirmed tubo-ovarian cancers were mis-registered. In 656 with death registrations, 7.8% (51/656) were wrongly assigned as due to ovarian/tubal/peritoneal cancers while 6.2% (40/645) of confirmed tubo-ovarian cancer deaths were mis-registered.


Follow-up of trial participants for tubo-ovarian cancer using national registry data will result in incomplete ascertainment, particularly of the site due in part to the latency of registration. This can be reduced by using other routinely collected data such as hospital episode statistics. Central adjudication by experts though resource intensive adds value by improving the accuracy of diagnoses.

Trial registration

ISRCTN: ISRCTN22488978. Registered on 6 April 2000

Peer Review reports


Currently, most trials use multiple sources for outcome ascertainment such as case record forms, patient questionnaires and electronic national health datasets. The latter includes cancer and death registrations, hospital administrative records (e.g. hospital episode statistics in the UK), national audit data and specialised health datasets. This is often followed by central adjudication to ensure consistency and minimise bias [1]. Central review requires medical records to be retrieved, checked for completeness, redacted with regard to any reference to randomisation group and patient identifiers and the collated information reviewed by a specialist committee. Given the expensive and time-consuming nature of this process, there is a clear need to evaluate alternative strategies.

There has been an explosion in the use of electronic national datasets for clinical research. As the data quality of such resources improves and access processes for research become more streamlined, this is likely to be a less resource-intensive and less costly option for outcome ascertainment in trials. The use of cancer and death registration alone is especially relevant to large randomised controlled trials of cancer screening where cancer site and disease-specific cause of death (CoD) are primary outcome measures. Data from the US National Lung Screening Trial (NLST) suggests that using death certificate CoD alone (18%; 95% CI 4.2–25.0) would not have impacted on the published (20%; 95% CI 6.7–26.7) lung cancer mortality reduction where the process included central adjudication [2]. This was also noted in the Finnish prostate cancer screening trial [3]. However, other trials have raised concerns about the completeness and accuracy of such data [4]. In the Health Insurance Plan of New York breast screening trial [4], there was no screening benefit based on death certificates alone whereas analysis performed using adjudicated data showed a significant effect. This raises the need for further investigation before universal adoption.

In the United Kingdom Collaborative Trial of Ovarian Cancer Screening (UKCTOCS) [5, 6], follow-up relies on multiple data sources including national electronic health-related datasets with expert adjudication (by outcomes review committee) being used for confirmation of ovarian cancer diagnosis and CoD. This was felt necessary as ovarian cancer often presents late with widely disseminated disease, heightening the possibility of disease misclassification. In addition, decreased investigation in older women, accident and emergency presentation and mortality within the first month of presentation could contribute to ovarian cancer being reported as a malignant neoplasm of unknown origin (ICD-10 code C80) [7]. The trial data provides an opportunity to determine the completeness and accuracy of national ovarian cancer and death registration in England, Wales and Northern Ireland using outcomes review as the reference standard.


Between April 2001 and September 2005, 202,638 postmenopausal women aged 50–74 from the general population were recruited. They were randomised to annual screening by transvaginal ultrasound (50,639) or a multimodal strategy using serum CA125 interpreted by the Risk of Ovarian Cancer Algorithm (ROCA), followed by ultrasound as a second-line test (50,640) or control (no intervention; 101,359) [5, 6]. The full trial protocol is accessible at Trial registration is ISRCTN22488978. This analysis is based on data collected prospectively up to the end of the initial follow-up phase of UKCTOCS (2001–2014).

Multiple sources were used for the follow-up of trial participants (Table 1) [6]. Women who provided written consent for follow-up through national registries were flagged using their NHS number, date of birth and address for cancer and/or death registrations with NHS Digital (England and Wales) and the Northern Ireland Cancer Registry and Business Services Organisation Health and Social Care Northern Ireland (BSO Northern Ireland), respectively. Censorship data for the initial analysis of trial data was 31 December 2014 [6]. The last notification prior to data freeze for trial outcome analysis was received for both cancer and deaths for England and Wales on 25 March 2015 (NHS Digital), deaths for Northern Ireland on 9 April 2015 (BSO, Northern Ireland) and cancers for Northern Ireland on 15 April 2015 (Northern Ireland Cancer Registry). For women resident in England, data was also available for 2001–2012 from Hospital Episodes Statistics (HES) and up to 2014 from the National Cancer Intelligence Network. Additionally, all women were sent two follow-up postal questionnaires, 3–5 years after randomisation and in 2014. The UKCTOCS coordinating centre also received direct notification from the 13 trial centres and trial participants or their relatives.

Table 1 Data sources used for cancer and death notification in UKCTOCS

Outcomes review committee

An outcomes review committee (ORC) was set up to confirm ovarian cancer diagnosis and death. The committee consisted of two pathologists with specialist interest in gynaecological cancer and two gynaecological oncologists. The ORC members were independent of trial conduct and blinded to the randomisation group.

Identification of cases and collation of evidence

Cases were identified for OR based on a cancer/death registration with any one of 19 preselected International Classifications of Disease (10th Revision, ICD-10) codes (WHO 2003) [7] (Supplementary Table 1, see Additional file 1) or a notification of possible ovarian cancer through the other sources listed above. The preselected ICD-10 codes were defined at trial initiation as those potentially associated with an underlying ovarian cancer [6]. Cases with an ICD-10 C80 (n = 936) death registration and a cancer registration of a non-ovarian malignancy were reviewed by a designated gynaecological oncologist and excluded from the below process.

Dedicated coordinating centre personnel initiated the collection of pertinent documents from National Health Service (NHS) Trusts, general practices, local health authorities, private hospitals and hospices. These included copies of medical records, histology, cytology reports, operative notes, diagnostic imaging reports, hospital letters and discharge summaries, multidisciplinary meeting summaries, chemotherapy/radiotherapy notes and autopsy reports. Procurement of the histology report was obligatory in cases where pathological examination had been undertaken. The documents were organised in chronological order and any direct or indirect mention of the randomisation group redacted, prior to submission to ORC. The minimum dataset required for review comprised three separate documents which could include, in addition to those listed above, death certificates and cancer registrations.

Outcomes review

Figure 1 shows a schematic for outcomes review. For all cases, a cancer review form (Supplementary Figure 1, see Additional file 2), and where applicable a death review form (Supplementary Figure 2, see Additional file 3), was completed. If the reviewer deemed the documentation adequate, then a cancer diagnosis or CoD was assigned. An algorithm with detailed rules for site allocation (Supplementary Figure 3, see Additional file 4) ensured robust, reproducible and transparent assignment in various clinical scenarios. In all cases where there was a discrepancy regarding ovarian cancer classification between the reviewer and cancer or death registry, the case was forwarded to a second ORC member for an independent review. If both reviewers were in agreement, the assigned diagnosis/CoD was accepted. If there was disagreement, a third ORC member independently reviewed the case and a consensus decision was made based on the majority view. If a reviewer was unable to make a definitive diagnosis, the reviewer could request either further information or review of pathology slides, or a case discussion at an ORC meeting.

Fig. 1
figure 1

Outcomes review process

Outcomes review initially used the WHO 2003 [7] rules for classifying cases into tubo-ovarian or primary peritoneal. More recently, all the peritoneal cancers were reviewed and reclassified as tubo-ovarian cancers based on WHO revised 2014 criteria [8]. This states that peritoneal cancers can only be diagnosed when there is no demonstrable gross or microscopic ovarian or tubal involvement [9].


All trial participants were included in this analysis (Fig. 2). Sensitivities and specificities of cancer/death registration of ovarian cancer obtained up to 3 months postcensorship date were calculated, using central adjudication as the reference standard. We also calculated the sensitivity if all available national electronic health-related dataset sources (Table 1) were used. The definitions for sensitivity and specificity were as defined by the STARD 2015 guidelines (

Fig. 2
figure 2

Evaluation of national registrations of tubo-ovarian (includes ovarian, fallopian tube and peritoneal cancer) cancer/death compared to the reference standard (identification through multiple sources, hospital notes retrieval and independent outcomes review)

In addition, in all cases with a cancer registration and/or death certification, overall agreement of disease site and CoD between cancer and death registries and OR was assessed using the Cohen’s kappa (κ) statistic [10], with confidence intervals calculated using 1000 bootstrap samples. Zero weighting was applied to all off-diagonal cell entries.

We also considered the false-negative rate (FNR = 1-sensitivity) of cancer registration and death certification over time, with ORC-confirmed diagnosis and CoD as the gold standard and summarising with a test for proportional trend (Stata package ptrend). This test takes a standard χ2 test of cancer registration (or death certification) versus year, on k = (row-1) × (column-1) degrees of freedom (df), and partitions the χ2 value into a linear component (trend test) on 1 df and a ‘departure’ from linearity component on k-1 df. We excluded the year 2014 from the cancer registration analysis, as during exploratory analysis there was a large number of missing registrations.


The current analysis was funded by a peer-reviewed NIHR HTA grant (16/46/01) and The Eve Appeal. The funders were not involved in the analyses nor in the writing of this report.


Baseline characteristics of the trial population have been previously described [11]. Of the 202,638 postmenopausal women, 202,632 (99.9%) women were electronically flagged for cancer and death registrations in the relevant national registries of England, Wales and Northern Ireland (Fig. 2).

Performance characteristics of cancer and death registration for tubo-ovarian cancer

Our algorithm identified 3110 women requiring outcomes review for tubo-ovarian cancer diagnosis. The ORC confirmed 1324 as having tubo-ovarian cancer. Of them, 1125 women had an ovarian, tubal or primary peritoneal cancer registration. Overall sensitivity and specificity of ovarian, tubal or primary peritoneal cancer registration within 3 months of censorship date were 85.0% (1125/1324; 95% CI 83.7–86.2%) and 94.0% (1679/1786; 95% CI 93.2–94.8%), respectively (Fig. 2). In an additional 81 of the 1324 women, we received an ovarian, tubal or primary peritoneal cancer notification from other national datasets (death registration 51, HES 27, NCIN 3). If all available national electronic health-related datasets (Table 1) were considered, sensitivity would be 91.1% (1206/1324; 95% CI 89.4–92.5%). Figure 3a and Supplementary Table 2 (see Additional file 2) show the proportion of missing cancer registrations over time. Excluding the last year (2014), there was no statistical evidence of a trend in the proportion of missing cancer registrations (p = 0.261) or any other form of temporal variability (p = 0.549).

Fig. 3
figure 3

Proportion of missing registrations (includes ovarian (ICD-10 C56), tubal (ICD 10-C57) and peritoneal (ICD-10 C48) registrations) of tubo-ovarian cancers by year of diagnosis

Our algorithm identified 2214 women who died and required outcomes review for CoD. Only one of these women did not have a death registration as she had died outside the UK. The ORC confirmed 681 as having died due to tubo-ovarian cancer. Of them, 605 women had an ovarian, tubal or primary peritoneal cancer CoD registration (Table 2). Overall sensitivity and specificity of ovarian, tubal or primary peritoneal cancer death registration within 3 months of the censorship date were 88.8% (605/681; 95% CI 86.4–91.2%) and 96.7% (1482/1533; 95% CI 95.8–97.6%), respectively (Fig. 2). In the remaining 76 women, 29 women had an ovarian, tubal or primary peritoneal cancer registration, six had an ovarian cancer HES record. If all available national electronic health-related datasets (Table 1) were considered, sensitivity would be 94.0% (640/681; 95% CI 91.9–95.5%). The false-negative rates by year are presented in Fig. 3b (Supplementary Table 2, see Additional file 5). We found strong evidence of a negative linear trend in the proportion of missing death certificates (p = 0.00002) but no further departure from that linearity (p = 0.474).

Table 2 National cancer and death registrations versus outcomes review confirmation of tubo-ovarian cancer (a) diagnosis and (b) cause of death

Agreement of registry- and outcome review-assigned tubo-ovarian cancer diagnosis

There was a relevant ICD-10 cancer registration in 1474 of the 3110 women who underwent outcomes review for potential tubo-ovarian cancer (Table 2). One hundred seven (8.7%; 107/1232) of 1232 cases registered as ovarian, tubal or primary peritoneal cancer were reclassified on outcome review as malignant neoplasm of unknown origin (2.7%; 33/1232), no cancer (2.1%; 26/1232) and other primary cancer (3.9%; 48/1232). Over one third were endometrial (41.6%; 20/48). About one fourth (27.1%; 13/48) of the other primary cancers were metastatic to the ovary.

In addition, 47 (4.0%; 47/1172) of the 1172 women confirmed as having tubo-ovarian cancer by outcomes review would have been missed (Table 2). Almost three fourths of these women were registered as neoplasms of the ovary of uncertain behaviour (ICD-10 D39.1). The majority of these were reclassified on outcomes review as borderline epithelial or non-epithelial ovarian cancer (ICD-10 C56.0). Majority (80.3%; 10/12) of the missed invasive epithelial cancers were at an advanced stage (Table 3). The overall agreement (85.4%; 1259/1474) between the disease site ascribed by the cancer registries and those designated by the ORC was moderate (Cohen’s kappa (κ) 0.55; 95% CI 0.5–0.6).

Table 3 Characteristics of additional tubo-ovarian cancer (a) cases and (b) deaths identified through adjudication that did not have a national cancer registration

Agreement of registry- and outcome review-assigned ovarian cancer deaths

At censorship, 2041 cases had undergone outcomes review and had a relevant ICD-10 death registration. Of the 656 ovarian cancer death registrations, 92.2% (605/656) were confirmed on outcomes review. The CoD in 51 (7.8%; 51/656) of 656 women registered as having died of ovarian, tubal or primary peritoneal cancer were reclassified on outcome review as non-ovarian. This includes malignant neoplasm of unknown origin (51.0%; 26/51), other malignancy (37.3%; 19/51) or non-cancer causes (11.8%; 6/51). Over half of the other cancer deaths were due to endometrial cancer (52.6%; 10/19) (Table 2). In addition, 40 (6.2%; 40/645) of 645 women confirmed by outcomes review as having died of tubo-ovarian cancer were registered as having died of non-tubo-ovarian cancer causes (Table 3). The agreement (86.3%; 1763/2041) between cancer registry and OR was substantial (kappa 0.78; 95% CI 0.76–0.80).


Main findings

Our findings demonstrate that in trials using follow-up through national cancer registration alone in England, Wales and Northern Ireland, 15% of tubo-ovarian cancer cases may be missed. Similarly, the use of national death registration alone would result in 11% of tubo-ovarian cancer deaths being missed. The proportions of missing cancer registrations showed no decline over time unlike the deficit in disease-specific death registrations. These rates could be halved if additional national electronic datasets such as hospital episode statistics are used. When extrapolating these results, two facts must be taken into consideration. We included cancer and death registrations of primary peritoneal cancer in ovarian cancer incidence and mortality statistics. This is not the norm adopted by the UK Office of National Statistics or US SEER Registries. We obtained final follow-up data from the registries 3 months after the censorship date in order to fulfil reporting guidelines for the trial. Longer intervals from censorship to obtaining registry data are likely to reduce the rate in the last year.

Adjudication by an independent review process can improve the accuracy of cancer and death registrations. In those where a cancer registration was available, 9% of registry reported tubo-ovarian cancers would have inaccurate cancer site assignment. Additionally, 4% of the final outcomes confirmed cancers would have been missed as they were registered as non-ovarian cancers. Reliance on death registration alone would have resulted in a similar proportion (7.8%) of deaths being wrongly assigned to tubo-ovarian cancers and 6.2% of ovarian cancer deaths being missed.

Strengths and limitations

A key strength of our analysis is that we compared cancer diagnoses and causes of death reported by the national registries of England, Wales and Northern Ireland with those assigned through independent central adjudication in a population cohort of over 3000 women. Of the 202,638 women, 202,632 (99.9%) women were electronically flagged using their NHS number to the relevant national cancer and death registries. The multiple additional data sources (Table 1) we used to identify potential ovarian cancer cases further ensured that we had a complete population dataset of cases.

We did not restrict the study to women with the tubo-ovarian cancer-specific ICD-10 codes (C56, C57.0, C48.1 and C48.2) but used 19 possible ICD-10 codes, in particular malignant neoplasm primary site unknown (ICD-10 C80) that might have potentially included tubo-ovarian cases.

The peritoneal cancers reported by cancer registries were included as tubo-ovarian in this analysis. While this is the norm among clinicians and researchers, these cancers are not included in national tubo-ovarian statistics. We did this to ensure that registry completeness rates were not underestimated due to the 2014 WHO revision of the primary site definition [8]. As the latter is adopted by pathology departments the world over, it is likely that cancers previously denoted as primary peritoneal will be registered as tubo-ovarian.

A limitation is that the cases were diagnosed over a prolonged period between 2001 and 2014. To address this, we explored time trends. While there were no time trends in missing cancer registration data, the completeness of death registration data improved over the years.


The use of UK cancer registry data alone would have allowed us to detect 85% of the tubo-ovarian cancer cases. This is an improvement from the 78% sensitivity that we reported in our previous trial where women were diagnosed with these cancers between 1986 and 1993 [12]. It is likely that the rates have improved further in the more recent years. The use of multiple national electronic data sets especially hospital administration records can augment these rates. A recent report on tubo-ovarian cancers diagnosed between 2004 and 2012 in Switzerland also found that the hospital registry data provided complementary information to that from the cancer registries [13]. Sourcing data directly from live healthcare systems allows trials to overcome the latency in the national cancer registration systems. A median latency of 18 (range 4–60) months to completion of incidence ascertainment was reported in a recent survey of the European Network of Cancer Registries [14].

The overall agreement regarding ovarian cancer site between OR and cancer registries was moderate. A key area of discrepancy was related to tumours defined by the ICD-10 code D39.1, neoplasms of the ovary of unknown or uncertain behaviour. On outcome review, the majority were reclassified as malignant neoplasm of the ovary (ICD-10 codes C56), either borderline epithelial ovarian or granulosa cell tumours. Most of these discrepancies likely reflect historical coding practices and classification systems used by regional registries prior to centralization of services in 2013. Furthermore, coding for ovarian cancers since 2011 uses the ICD-O-3 system [15], where the topographical code (site of cancer) is clearly separated from the morphological code which includes the behaviour (malignant/benign). In the current system, as borderline ovarian tumours are classified as C56 (topographical code) with a morphology code of 1, there is less likelihood of the cases not being registered as ovarian cancer. However, our data for 2012 to 2014 are not sufficiently large to confirm this.

The 9% of tubo-ovarian cancers reported by the registries that were classed as ‘not ovarian’ cancers on review were equally distributed between three main groups: (1) miscoding of a benign mass as a cancer, (2) other primary cancer, and (3) malignant neoplasm, primary site unknown (C80). The other primary group included just under a third of cancers that metastasized to the ovary, which might have led to the confusion. The ovaries are a frequent site for metastases, with 5–30% of ‘ovarian masses’ being secondary metastasis reported from non-ovarian cancers [16].

The cause of death on death certificates is not always accurate [17] due to a variety of reasons (inexperience of certifying physician, lack of sufficient time/information, coding errors). However, in our trial, there was substantial agreement regarding the primary site between the death registry-reported ovarian cancer deaths and those assigned by the review committee. This has also been reported in screening trials of other cancers (e.g. prostate) which used adjudication panels [18, 19]. Nonetheless, 6.2% additional tubo-ovarian cancer deaths were identified through the adjudication process. A high proportion were registered as malignancies with an unknown primary site (C80) and all were at an advanced stage. In advanced cancer, where multiple sites are involved, it may be difficult to assign the primary site [20]. The lack of consistency in the approach adopted by pathologists to the assignment of the primary site during our trial led to a proposal for unified criteria for tubo-ovarian site assignment [21] which has since been adopted in international pathology reporting [22] and clinical guidelines [23].

The adjudication process used in the UKCTOCS trial is not dissimilar to that used in the Prostate Lung Colorectal and Ovarian Cancer screening trial [24]. In contrast to the cluster randomised trial of PSA testing for prostate cancer [25] where data was abstracted into vignettes for expert review, we provided copies of original clinical notes to the ORC members. To minimise ascertainment bias, any reference to the trial/randomisation arms was redacted in these documents. Review only occurred if the minimal required number of documents was available. ORC members who were highly experienced gynaecological oncology surgeons or pathologists used an algorithm developed specifically during the trial to assign a cancer site. To ensure accuracy and reproducibility over the years and across assessors, we audited the reviews and introduced a classification algorithm (Supplementary Figure 3, see Additional file 4). The resource implications of central adjudication can be significant, as it requires many hours of highly trained senior staff time. Often as in our trial, it is challenging to estimate costs as reviewers usually volunteer their time and work outside normal office hours. To save the effort and cost of adjudication, we used individual as opposed to group adjudication, a work flow that involved more than one assessment only when there was discrepancy between the primary reviewer and registered ICD-10 code, with group review limited to those where there was discrepancy between two reviewers.

Adjudication also provides information on other key variables such as staging and morphology which in the past have been poorly documented by registries [26] but now improving [27]. In our trial, we observed a stage shift in the multimodal screening arm [6]. This would not have been quantifiable using data from the registries alone without the detailed review of all available clinical documents as undertaken in UKCTOCS.

Kahan et al. [28] using a simulation model concluded that outcome misclassification can lead to biased treatment effect estimates and reduced power, potentially resulting in an erroneous conclusion regarding efficacy. The implementation of strategies to reduce misclassification is therefore of critical importance in clinical trials. In UKCTOCS, the main impact of adjudication was in improving the accuracy of tubo-ovarian cancer. On death outcomes, there was no significant difference on the mortality impact of screening between the analyses which used ORC-adjudicated disease-specific deaths and the sensitivity analysis limited to deaths with the disease-specific ICD-10 codes (ICD-10 C56, C57 and C48) for tubo-ovarian cancer [10].


Our data suggests that follow-up of trial participants for tubo-ovarian cancer using national registry data will result in incomplete ascertainment particularly of the site due in part to the latency of registration. This can be reduced by using hospital administrative data, which has shorter latency. Central adjudication by experts though resource intensive adds value by improving the accuracy of diagnoses. Revised classification systems for ovarian cancer [8] will improve accurate identification and aid reproducibility of site assignment, but this will require pathologists and clinicians to apply this new system uniformly across the globe.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.



United Kingdom Collaborative Trial of Ovarian Cancer Screening


Outcomes review committee


Cause of death


Risk of Ovarian Cancer Algorithm


Office of National Statistics

BSO Northern Ireland :

Business Services Organisation, Health and Social Care Northern Ireland


International Classifications of Disease (Revision 10)


National Health Service


General practitioners


Invasive epithelial ovarian cancer


  1. Dechartres A, Boutron I, Roy C, Ravaud P. Inadequate planning and reporting of adjudication committees in clinical trials: recommendation proposal. J Clin Epidemiol. 2009;62:695–702.

    Article  Google Scholar 

  2. Marcus PM, Doria-Rose VP, Gareen IF, Brewer B, Clingan K, Keating K, et al. Did death certificates and a death review process agree on lung cancer cause of death in the National Lung Screening Trial? Clin Trials. 2016;13:434–8.

    Article  Google Scholar 

  3. Mäkinen T, Karhunen P, Aro J, Lahtela J, Määttänen L, Auvinen A. Assessment of causes of death in a prostate cancer screening trial. Int J Cancer. 2008;122:413–7.

    Article  Google Scholar 

  4. Doria-Rose VP, Marcus PM, Miller AB, Bergstralh EJ, Mandel JS, Tockman MS, et al. Does the source of death information affect cancer screening efficacy results? A study of the use of mortality review versus death certificates in four randomized trials. Clin Trials. 2010;7:69–77.

    Article  Google Scholar 

  5. Menon U, Gentry-Maharaj A, Ryan A, Sharma A, Burnell M, Hallett R, et al. Recruitment to multicentre trials--lessons from UKCTOCS: descriptive study. BMJ. 2008;337:a2079.

    Article  Google Scholar 

  6. Jacobs I, Menon U, Ryan A, Gentry-Maharaj A, Burnell M, Kalsi JK, et al. Ovarian cancer screening and mortality in the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS): a randomised controlled trial. Lancet. 2016;387:945–56.

    Article  Google Scholar 

  7. World Health Organisation: International statistical classification of disease and related health problems codes, (tenth revision) 2nd edition. (2003).

    Google Scholar 

  8. World Health Organisation: International statistical classification of disease and related health problems codes, (tenth revision) (2014).

    Google Scholar 

  9. Prat J, FIGO Committee on Gynecologic Oncology. Staging classification for cancer of the ovary, fallopian tube, and peritoneum. Int J Gynaecol Obstet. 2014;124(1):1–5.

    Article  Google Scholar 

  10. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:37–46.

    Article  Google Scholar 

  11. Menon U, Gentry-Maharaj A, Hallett R, Ryan A, Burnell M, Sharma A, et al. Sensitivity and specificity of multimodal and ultrasound screening for ovarian cancer, and stage distribution of detected cancers: results of the prevalence screen of the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS). Lancet Oncol. 2009;10:327–40.

    Article  Google Scholar 

  12. MacDonald N, Sibley K, Rosenthal A, Menon U, Jayarajah A, Oram D, et al. A comparison of national cancer registry and direct follow-up in the ascertainment of ovarian cancer. Br J Cancer. 1999;80:1826–7.

    Article  CAS  Google Scholar 

  13. Wieser S, Schmidt M, Kind AB, Heinzelmann-Schwarz VA. Ovarian cancer in Switzerland: incidence and treatment according to hospital registry data. Swiss Med Wkly. 2018;148:w14647. eCollection 2018 Jul 16.

    Article  PubMed  Google Scholar 

  14. Zanetti R, Schmidtmann I, Sacchetto L, Binder-Foucard F, Bordoni A, Coza D, et al. Completeness and timeliness: cancer registries could/should improve their performance. Eur J Cancer. 2015;51:1091–8.

    Article  CAS  Google Scholar 

  15. Internationational Classification of Diseases for Oncology Accessed 1 Aug 2018.

  16. Lee S-J, Bae J-H, Lee A-W, Tong S-Y, Park Y-G, Park J-S. Clinical characteristics of metastatic tumors to the ovaries. J Korean Med Sci. 2009;24:114–9.

    Article  Google Scholar 

  17. Mieno MN, Tanaka N, Arai T, Kawahara T, Kuchiba A, Ishikawa S, et al. Accuracy of death certificates and assessment of factors for misclassification of underlying cause of death. J Epidemiol. 2016;26:191–8.

    Article  Google Scholar 

  18. Godtman R, Holmberg E, Stranne J, Hugosson J. High accuracy of Swedish death certificates in men participating in screening for prostate cancer: a comparative study of official death certificates with a cause of death committee using a standardized algorithm. Scand J Urol Nephrol. 2011;45:226–32.

    Article  Google Scholar 

  19. Turner EL, Metcalfe C, Donovan JL, Noble S, Sterne JA, Lane JA, et al. Contemporary accuracy of death certificates for coding prostate cancer as a cause of death: Is reliance on death certification good enough? A comparison with blinded review by an independent cause of death evaluation committee. Br J Cancer. 2016;115:90–4.

    Article  Google Scholar 

  20. Singh N, Gilks CB, Wilkinson N, McCluggage WG. Assessment of a new system for primary site assignment in high-grade serous carcinoma of the fallopian tube, ovary, and peritoneum. Histopathology. 2015;67:331–7.

    Article  Google Scholar 

  21. Singh N, Gilks CB, Hirschowitz L, Kehoe S, McNeish IA, Miller D, et al. Primary site assignment in tubo-ovarian high-grade serous carcinoma: consensus statement on unifying practice worldwide. Gynecol Oncol. 2016;141:195–8.

    Article  Google Scholar 

  22. McCluggage WG, Judge MJ, Clarke BA, et al. Data set for reporting of ovary, fallopian tube and primary peritoneal carcinoma: recommendations from the International Collaboration on Cancer Reporting (ICCR). Mod Pathol. 2015;28:1101–22.

    Article  Google Scholar 

  23. Colombo N, Sessa C, du Bois A, Ledermann J, McCluggage WG, McNeish I, ESMO-ESGO Ovarian Cancer Consensus Conference Working Group, et al. ESMO-ESGO consensus conference recommendations on ovarian cancer: pathology and molecular biology, early and advanced stages, borderline tumours and recurrent disease. Ann Oncol. 2019;30:672–705.

    Article  CAS  Google Scholar 

  24. Miller AB, Yurgalevitch S, Weissfeld JL, Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial Project Team. Death review process in the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial. Control Clin Trials. 2000;21(6 Suppl):400S–6S.

    Article  CAS  Google Scholar 

  25. Williams NJ, Hill EM, Ng SY, Martin RM, Metcalfe C, Donovan JL, CAP Cause of Death Committee, et al. Standardisation of information submitted to an end-point committee for cause of death assignment in a cancer screening trial. BMC Med Res Methodol. 2015;15:6.

    Article  Google Scholar 

  26. Walters S, Maringe C, Butler J, Brierley JD, Rachet B, Coleman MP. Comparability of stage data in cancer registries in six countries: lessons from the International Cancer Benchmarking Partnership. Int J Cancer. 2013;132:676–85.

    Article  CAS  Google Scholar 

  27. McPhail S, Johnson S, Greenberg D, Peake M, Rous B. Stage at diagnosis and early mortality from cancer in England. Br J Cancer. 2015;112(Suppl 1):S108–15.

    Article  Google Scholar 

  28. Kahan BC, Feagan B, Jairath V. A comparison of approaches for adjudicating outcomes in clinical trials. Trials. 2017;18:266.

    Article  PubMed  PubMed Central  Google Scholar 

Download references


We thank the volunteers without whom the trial would not have been possible and everyone involved in conduct and oversight. We are also very grateful to the current members of the UKCTOCS Trial Steering Committee—Prof Henry Kitchener (Chair), Prof Julietta Patnick, Prof Jack Cuzick and Ms Annwen Jones.


The current analysis is supported by the National Institute for Health Research (NIHR) HTA grant (16/46/01) and The Eve Appeal. UKCTOCS was funded by the Medical Research Council (G9901012 and G0801228), Cancer Research UK (C1479/A2884) and the Department of Health, with additional support from The Eve Appeal. Researchers at UCL are supported by the NIHR University College London Hospitals (UCLH) Biomedical Research Centre and MRC CTU at UCL core funding (MR_UU_12023).

Disclaimer: Views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

Author information

Authors and Affiliations



This analysis was conceived by UM, JKK and AR. SM, SA, MH, DMC and CK collated clinical notes for review by the outcomes review committee. NS, MW, EB and RW performed the outcome review. JKK, UM and AGM performed the literature search. AR prepared the dataset and MB performed the statistical analysis. JKK, AR, MB, AGM and UM interpreted the data. JKK, AR, AGM and UM prepared the figures and tables. AR, JKK, AGM, MB and UM contributed to the writing of the manuscript. AGM, SA, MH, DMC, JKK, SM, CK and UM were involved with the trial implementation and data collection. All authors (JKK, AR, DMC, NS, MB, EB, SA, MH, SM, CK, AGM, RW, MW, LF, SC, SS, AM, MP, IJ, UM) critically reviewed and approved the report before submission. Authors LF, SC, SS, AM and MP contributed to the trial design initiated by IJ and UM.

Corresponding author

Correspondence to Usha Menon.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the UK North West Multicentre Research Ethics Committees (North West MREC 00/8/34) on 21 June 2000 with site-specific approval from the local regional ethics committees and the Caldicott guardians (data controllers) of the primary care trusts. All trial participants provided written informed consent. The study was conducted in accordance with the Declaration of Helsinki and Good Clinical Practice guidelines.

Consent for publication

Not applicable

Competing interests

UM has stock ownership in Abcodia awarded to her by UCL. IJ is a co-inventor of the Risk of Ovarian Cancer Algorithm (ROCA). The ROCA has been licenced to Abcodia Ltd. by the owners of the ROCA, Massachusetts General Hospital (MGH) and Queen Mary University of London (QMUL). IJ has a financial interest in Abcodia Ltd. as a shareholder and director and is entitled to a royalty payment via MGH and QMUL from any commercial use of the ROCA. He is a trustee (2012–2014) and Emeritus Trustee (2015 to present) for The Eve Appeal. SJS co-developed the risk of ovarian cancer algorithm with all rights assigned to MGH and QMUL. MGH has co-licenced software to Abcodia implementing the algorithm. SJS receives personal fees from the LUNGevity Foundation and SISCAPA Assay Technologies as a member of their Scientific Advisory Boards and from Abcodia as a consultant. SA is funded by a research grant from Abcodia. All other authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Supplementary Table 1.

International Classifications of Disease, (Revision 10) used for data linkage with national cancer registries in England, Wales and Northern Ireland.

Additional file 2: Supplementary Figure 1.

Outcomes Cancer Review Form.

Additional file 3: Supplementary Figure 2.

Outcomes Death Review Form.

Additional file 4: Supplementary Figure 3.

Algorithm for assignment of tubo-ovarian cancer site based on available data.

Additional file 5: Supplementary Table 2.

Year of diagnosis / death of tubo-ovarian cancers with missing registration.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kalsi, J.K., Ryan, A., Gentry-Maharaj, A. et al. Completeness and accuracy of national cancer and death registration for outcome ascertainment in trials—an ovarian cancer exemplar. Trials 22, 88 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: