- Open Access
SMS text pre-notification and delivery of reminder e-mails to increase response rates to postal questionnaires in the SUSPEND trial: a factorial design, randomised controlled trial
Trials volume 16, Article number: 295 (2015)
Patient-reported outcomes are vital in informing randomised controlled trials (RCTs) and health-care interventions and policies from the patient’s perspective. However, participant non-response may introduce bias and can affect the generalisability of the trial.
This study evaluates two interventions aimed at increasing response rates to postal questionnaires within a large, UK-wide RCT: pre-notification via short messenger service (SMS) text prior to sending the initial mailing of trial questionnaires versus no pre-notification; for non-responders to the initial mailing of the questionnaires, an e-mail reminder (containing a hyperlink to complete the questionnaire online) versus a postal reminder.
This study is a 2×2 partial factorial design RCT nested within an RCT of medical expulsive therapy for ureteric stone disease. Participants who supplied a mobile telephone number were randomly assigned to receive an SMS text pre-notification of questionnaire delivery or no pre-notification. Those who supplied an e-mail address were randomly assigned to receive a questionnaire reminder by e-mail or post. Participants could be randomly assigned to the pre-notification comparison or the reminder comparison or both. The primary outcome measure was response rate at each questionnaire time point.
Four hundred eighteen participants were randomly assigned to the SMS pre-notification comparison (80 % were male, and the mean age was 41 years with a standard deviation (SD) of 11.1). The intervention had no effect on response rate at either questionnaire time point. In subgroup analyses, SMS pre-notification increased response rates in women but only at the first questionnaire time point. One hundred nineteen participants were randomly assigned to the reminder comparison (80 % were male, and the mean age was 42 years with an SD of 12.1). There was no difference in response rate in those who received an e-mail reminder compared with those who received a postal reminder.
SMS text pre-notification of questionnaire delivery and email delivery of questionnaire reminders did not improve response rates. There was some evidence to suggest that SMS text pre-notification may be effective in women, and further studies to investigate this may be warranted. E-mail reminders for participants to return their postal questionnaire could be advantageous given that response rates were similar following either type of reminder and the low cost of delivering an e-mail compared with a postal reminder.
This is a substudy of the SUSPEND trial (ISCTRN69423238) (18 Nov. 2010).
Patient perspectives should provide evidence to inform the design, delivery, and evaluation of health care . The knowledge that can be gained from informing health outcomes with evidence from the patient’s perspective means that patient-reported outcomes are key in many randomised controlled trials (RCTs) of health-care interventions. However, these are often collected by postal questionnaires, and participant non-response can reduce effective sample size, potentially introduce bias, and impact on the generalisability of the trial results . There is, therefore, a need for researchers to encourage participants to respond to questionnaires and deliver sound, unbiased trials in a timely and resource-efficient manner to provide evidence that informs health-care policies and guidelines and, ultimately, better public health.
Many ways have been suggested to improve response rates to questionnaires, and these are discussed in two Cochrane reviews [3, 4]. Both reviews include RCTs of strategies to improve questionnaire response rates; however, the more recent review  is limited to those conducted within parent RCTs. The overwhelming evidence in both of these reviews, and in another review within population-based cohort studies , supports the use of monetary incentives in improving response rates. However, the use of monetary incentives carries practical and ethical considerations, especially within studies that are publically funded.
In 2012, surveys by the Office of National Statistics reported that 91 % of the UK population owned a mobile phone and that 16- to 24-year-olds were most likely to possess one . E-mail use is also widespread: 75 % of the UK population in 2013 used the internet to send and receive e-mails . The use of e-mail was most common amongst 25- to 34-year-olds (of whom 89 % accessed the internet for e-mail), and men were more likely to use e-mail than women (78 % versus 72 %). These data are of particular interest given that younger people are less likely to respond to questionnaires than older people [8, 9] but are also more likely to use mobile phones and e-mail. However, there is a lack of evidence about the use of mobile telephone and e-mail technology to influence response rates in clinical trials. The purpose of the study described here was to investigate whether technology favoured by younger age groups could be used to increase response rates to questionnaires in the SUSPEND trial, a large RCT of medical expulsive therapy for ureteric stone disease . The hypothesis for the SUSPEND response rate study (SUSRes) is that the use of SMS text pre-notification of questionnaire delivery and e-mail delivery of questionnaire reminders (with a link to complete the questionnaire in a secure web site) will improve questionnaire response rates.
SUSRes was a randomised, controlled, 2×2 partial factorial design nested within the SUSPEND RCT , testing two methods—the use of SMS text pre-notification of questionnaire delivery and e-mail delivery of questionnaire reminders (with a link to complete the questionnaire in a secure web site)—to improve questionnaire response rates. Within SUSPEND, patient-reported outcomes were collected by postal questionnaire at 4 and 12 weeks, and SUSRes was conducted in 24 SUSPEND sites throughout the UK from June 2012 to March 2014. SUSRes was not planned at the outset of the SUSPEND RCT; rather, it was developed in response to lower-than-expected response rates to postal questionnaires within SUSPEND.
Ethics and other approvals
The study was reviewed and approved by the East of Scotland Research Ethics Service as a substantial amendment to the SUSPEND trial (Research Ethics Committee) reference 10/S0501/31) and received approval from local research and development departments at the SUSPEND sites (Aberdeen Royal Infirmary; Addenbrooke’s Hospital, Cambridge; Bristol Royal Infirmary; Broadgreen Hospital, Liverpool; Cheltenham General Hospital; Derriford Hospital, Plymouth; Freeman Hospital, Newcastle upon Tyne; Guy’s Hospital, London; Manchester Royal Infirmary; Morriston Hospital, Swansea; Norfolk and Norwich University Hospital; Pinderfields Hospital, Wakefield; Queen Elizabeth Hospital, Birmingham; Raigmore Hospital, Inverness; Royal Hallamshire Hospital, Sheffield; Southampton General Hospital; Southmead Hospital, Bristol; St George’s Hospital, London; St James’s University Hospital, Leeds; Sunderland Royal Hospital; The James Cook University Hospital, Middlesbrough; Torbay Hospital, Torquay; University Hospital of South Manchester; Western General Hospital, Edinburgh) before commencing. Participants provided written informed consent for SUSPEND which encompassed the SUSRes study.
Participants who were newly randomly assigned to the SUSPEND trial, had not reached the 4-week time point, and were willing to supply a mobile phone number or an e-mail address (or both) were considered for the study. Participants could be included in one or both comparisons. All participants who met the inclusion criteria and where a mobile phone number was recorded were included in the SMS text comparison. Participants for whom an e-mail address was provided were included in the e-mail comparison. All participants provided written informed consent for the SUSPEND trial and therefore this study.
Interventions and controls
There were two comparisons (each with a separate randomisation) within the SUSRes study.
SMS text pre-notification comparison
All participants randomly assigned to the intervention arm were sent an SMS text message pre-notification of the delivery of the initial 4- and 12-week questionnaires. The message was generated automatically from the SUSPEND trial database and sent via an external supplier the same day the questionnaire was dispatched from the trial office. The SMS read: “Many thanks for participating in the SUSPEND trial. You will shortly receive your xx-week questionnaire. We hope you can take a few minutes to complete this and return it to us”. Participants randomly assigned to the control arm were not sent any pre-notification of the delivery of the 4- and 12-week questionnaires.
E-mail reminder comparison
All participants who were randomly assigned to the intervention arm and who did not respond to the initial 4- or 12-week questionnaire received an e-mail which included a link to complete the questionnaire online or were invited to return the paper copy if they wished. Participants who were randomly assigned to the control arm and who did not respond to the initial 4- or 12-week questionnaire received their reminder by post with a further copy of the questionnaire.
Both types of reminder would have been generated on the same day, two weeks after the initial questionnaire was sent out by post from the trial office, by the SUSPEND trial database. The e-mails were automatically sent from the trial database and the postal reminders were printed and posted by trial office staff on the same day.
The wording of the reminder e-mail reflected the wording and layout of the letter that accompanied the postal questionnaires to avoid any bias by modifying this. The online questionnaire contained the same questions as the postal questionnaire.
Allocation and randomisation
Participants were randomly allocated to the intervention or control groups of the SUSRes study on a 1:1 basis by using a computer-generated system that was concealed and remote from the users. The randomisation algorithm was permuted blocks (block size of 4) stratified by age (≤ 40 years or > 40 years) and sex as these variables are known to affect response rate [8, 9]. The allocated group applied to both SUSPEND questionnaire time points (4 and 12 weeks). Owing to the nature of the intervention, it was not possible to blind the participants or trial office staff to allocation; however, the researchers remained blind.
The primary outcome for both comparisons was defined as questionnaire response rate at each time point. There were no secondary outcomes.
The sample size for the SUSRes study was dictated by the number of participants still to be recruited into SUSPEND at the time the study started (710); therefore, no formal sample size calculation was undertaken.
All analyses were performed in Microsoft Excel (2010; Microsoft Corporation, Redmond, WA, USA) and IBM SPSS statistics version 22.0 (IBM Corporation, Somers, NY, USA). Two comparisons were made: SMS text message pre-notification versus no pre-notification of questionnaire delivery and questionnaire e-mail reminder versus postal reminder.
Participant baseline characteristics (age and sex) between the intervention and control groups within each intervention comparison were compared by using an independent t test or chi-squared test as appropriate.
The primary outcomes (response rate at each time point) were analysed on the basis of the intention-to-treat principle , and all participants were analysed as randomly assigned. To assess the impact of each intervention, adjusted odds ratios (ORs) (adjusted for the stratification variables age and sex) with 95 % confidence intervals (CIs) were computed by using logistic regression. Allocation to the other comparison was also added as a co-variate in the adjusted analysis. Possible interaction between the two interventions was explored with logistic regression analysis in those participants who had been randomly assigned in both comparisons.
The effects of age and sex were considered in pre-specified subgroup analyses by using the same analysis techniques described above and adjusted for age or sex as appropriate and allocation to the other comparison.
The flow of participants through the SUSRes study is shown in Figs. 1 and 2 according to the recommendations of the CONSORT statement . In total, 710 participants who entered the SUSPEND RCT were assessed for eligibility to enter the SUSRes study. Of those, 418 (59 %) were eligible to be randomly assigned for the pre-notification comparison and 119 (17 %) were eligible to be randomly assigned to the reminder comparison. Within the pre-notification comparison, 80 % were male and the mean age was 41 years (standard deviation (SD) of 11.1). Within the reminder comparison, 80 % were male and the mean age was 42 years (SD of 12.1). Within each comparison, the randomised groups were well balanced (Table 1) and there were no statistically significant differences between the intervention and control groups.
At 4 weeks, the questionnaire response rate was slightly higher (57 %) following an SMS text pre-notification than in the control group (no SMS text: 52 %), but the difference was not statistically significant (adjusted OR 1.24, 95 % CI 0.84–1.82; Table 2). There was no effect of an e-mail reminder compared with a postal reminder on questionnaire response rates (68 % versus 66 %, respectively) at 4 weeks (adjusted OR 1.11, 95 % CI 0.51–2.40; Table 2). There was no effect of either intervention at 12 weeks (Table 2).
Subgroup analysis of response by sex and age suggested that women were more likely to respond to the 4-week questionnaire following an SMS text pre-notification (adjusted OR 2.58, 95 % CI 1.05–6.33; Table 3). There was no effect of an e-mail reminder in sex or age subgroup analysis (data not shown).
The factorial trial design allows for an interaction between the two interventions to be evaluated. There was no evidence to suggest an interaction, although the SUSRes study was not powered to detect this.
The aim of the SUSRes study was to provide evidence in relation to the use of SMS text pre-notification of questionnaire delivery and e-mail reminders following questionnaire non-return on response rates within the SUSPEND trial. The SUSPEND trial lent itself to the SUSRes study as the response rate to the trial questionnaires, which collected patient-reported outcomes, was low at the time of the inception of the SUSRes study. Being relatively young  and predominantly male , which are also the groups more likely to own a mobile phone  and use e-mail , the trial population was regarded as being one less likely to respond to questionnaires.
SMS text pre-notification failed to have any statistically significant effect on the response rates in the SUSRes trial. This supports previously published data [13, 14], although these studies used different delivery methods for the intervention and different time points of intervention delivery. The observed effect of an SMS text pre-notification in women at 4 weeks may be a chance finding. A bespoke trial database, designed and supported by a dedicated programming team with experience and familiarity with the necessary technology, meant that SMS text pre-notification was relatively straightforward and inexpensive to implement within SUSPEND; however, this may not be as simple to implement in other RCTs. When such technology is available, the small potential benefit of using SMS text pre-notification in women may outweigh the cost of such a strategy: this is worthy of further investigation.
We hypothesised that the use of e-mail reminders, which included a link to complete the questionnaire online, would increase questionnaire response rates. However, this was not the case and the use of e-mail reminders did not have any effect on response rates at 4 and 12 weeks in the SUSRes study; this held true when a post-hoc per-protocol analysis was performed on the participants who required a reminder (e-mail versus postal reminder response rates were 47 % versus 54 % at 4 weeks and 52 % versus 49 % at 12 weeks). This is in contrast to the literature which suggests that, in surveys, e-mail delivery of questionnaires reduces response rate . Given that there was no negative effect of providing questionnaire reminders by e-mail compared with post and that the cost of an e-mail reminder is insignificant (if a trial has a database or information technology system that can manage this) compared with the cost of a postal reminder, this may be a strategy that trialists wish to consider implementing.
Factorial design trials have been used previously in RCTs to investigate methods to improve response rates [16–19]. The SUSRes study differed from a conventional 2×2 factorial design  in that, instead of being randomly assigned once to one of four study groups (that is, no pre-notification/postal reminder, no pre-notification/e-mail reminder, pre-notification/postal reminder, or pre-notification/e-mail reminder), participants were subjected to two separate randomisations: no pre-notification versus SMS text pre-notification and postal reminder versus e-mail reminder. The study was conducted in this manner (a partial factorial design) as it was anticipated that participants may not provide both a mobile phone number and an e-mail address and therefore performing two separate randomisations would most efficiently use the available patient population. These fears were well founded as only 56 of the 710 SUSPEND participants (8 %) provided both a mobile phone number and an e-mail address, which would have markedly reduced the sample size.
The main weaknesses of the SUSRes study relate to the number of participants. Firstly, the number who were potentially eligible when the SUSRes study started was limited by the number of participants who had yet to be recruited into the SUSPEND trial (n = 710). Because this number was fixed, we did not undertake a formal sample size calculation, and this is a further limitation of our study. Secondly, there was a low recruitment rate to the SUSRes study. Only 59 % (418/710) of those recruited into the SUSPEND trial were eligible for the pre-notification comparison study, and only 17 % (119/710) for the reminder comparison study. There are a number of possible explanations for the low recruitment rate. The patient population in the SUSPEND trial has ureteric stone disease. These patients were recruited to SUSPEND in an acute setting and were often discharged quickly. It is therefore possible that collection of essential trial data (e.g., baseline clinical characteristics) took priority over the collection of information such as e-mail addresses and mobile phone numbers, which may have appeared superfluous to the main trial but were essential for inclusion in the SUSRes study. Anecdotally, some research staff saw themselves as gate keepers of participants’ personal information and were not comfortable collecting these data. Seeking this information directly from the participants may be a solution, particularly if they are also asked about preferred methods of contact. The risk of a type II error (i.e., the risk of failing to detect a difference between two groups when one truly exists) is increased when studies are underpowered, which the SUSRes study is likely to be.
Despite the low recruitment rate to both SUSRes studies, the gender and age (≤ 40 years and > 40 years) of those included in the postal versus e-mail reminder comparison and the gender of those included in the SMS text pre-notification were similar to those of patients who were not included. Potentially reflecting levels of mobile phone ownership, 70 % of those younger than 40 were included in the SMS text pre-notification comparison compared with 53 % of those who were older than 40 years.
Whereas we know how many SUSRes study text messages and e-mail reminders were undelivered and how many postal questionnaires were returned unopened, we do not know how many others were received but not opened or read by the participant. Mobile phone numbers can be shared between people and text messages may not have been received if the mobile phone was switched off or out of service. E-mail messages may have been routinely filtered into junk or spam folders. Postal reminders may have been undelivered (but not returned to the trial office) or discarded.
SMS text pre-notification of questionnaire delivery and e-mail delivery of questionnaire reminders did not increase questionnaire response rate in the SUSPEND trial population. However, there was some evidence to suggest that SMS text pre-notification may be effective in women, and further studies to investigate this may be warranted. E-mail reminders for participants to return their postal questionnaire could be advantageous given that response rates were similar following either type of reminder, together with the low cost of delivering an e-mail compared with a postal reminder.
Consolidated Standards Of Reporting Trials
Randomised controlled trial
Short message service
Spontaneous Ureteric Stone Passage ENabled by Drugs
SUSPEND response rate study
Facey K, Boivin A, Gracia J, Ploug Hansen H, Lo Scalzo A, Mossman J, et al. Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care. 2010;26:334–40.
Armstrong B, White E, Saracci R. Principles of exposure measurement in epidemiology. In: Kelsey J, Marmot M, Stolley P, Vessey M, editors. Monographs in Epidemiology and Biostatistics. 1st ed. New York: Oxford University Press Inc.; 1995. p. 294.
Edwards P, Roberts I, Clarke M, DiGuiseppi C, Wentz R, Kwan I, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009;(3):MR000008.
Brueton V, Tierney J, Stenning S, Nazareth I, Meredith S, Harding S, et al. Systematic review of strategies to reduce attrition in randomised trials. Cochrane Database Syst Rev. 2013;12, MR000032.
Booker C, Harding S, Benzeval M. A systematic review of the effect of retention methods in population-based cohort studies. BMC Public Health. 2011;11:249.
Office of National Statistics. Users of mobile phones or smartphones, in the last 3 months, by age group, 2012. http://www.ons.gov.uk/ons/about-ons/business-transparency/freedom-of-information/what-can-i-request/previous-foi-requests/people-and-places/mobile-and-smartphone-usage/mobile-phone-users-by-age-group--gb--2012.xls. Accessed 5 May 2014.
Office of National Statistics. Internet Access - Households and Individuals, 2013. http://www.ons.gov.uk/ons/rel/rdit2/internet-access---households-and-individuals/2013/index.html. Accessed 5 May 2014.
Eaker S, Bergstrom R, Bergstrom A, Adami HO, Nyren O. Response rate to mailed epidemiologic questionnaires: a population-based randomized trial of variations in design and mailing routines. Am J Epidemiol. 1998;147:74–82.
Warner ET, Glasgow RE, Emmons KM, Bennett GG, Askew S, Rosner B, et al. Recruitment and retention of participants in a pragmatic randomized intervention trial at three community health clinics: results and lessons learned. BMC Public Health. 2013;13:192.
McClinton S, Starr K, Thomas R, McLennan G, McPherson G, McDonald A, et al. Use of drug therapy in the management of symptomatic ureteric stones in hospitalized adults (SUSPEND), a multicentre, placebo-controlled, randomized trial of a calcium-channel blocker (nifedipine) and an α-blocker (tamsulosin): study protocol for a randomized controlled trial. Trials. 2014;15:238.
Montori VM, Guyatt GH. Intention-to-treat principle. Can Med Assoc J. 2001;165:1339–41.
Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.
Ashby R, Turner G, Cross B, Mitchell N, Torgerson D. A randomized trial of electronic reminders showed a reduction in the time to respond to postal questionnaires. J Clin Epidemiol. 2011;64:208–12.
Man MS, Tilbrook HE, Jayakody S, Hewitt CE, Cox H, Cross B, et al. Electronic reminders did not improve postal questionnaire response rates or response times: a randomized controlled trial. J Clin Epidemiol. 2011;64:1001–4.
Shih T, Fan X. Comparing response rates from web and mail surveys: a meta analysis. Field Methods. 2009;20:249.
Puffer S, Porthouse J, Birks Y, Morton V, Togerson D. Increasing response rates to postal questionnaires: a randomised trial of variations in design. J Health Serv Res Pol. 2004;9:213.
Sharp L, Cochran C, Cotton SC, Gray NM, Gallagher ME, Tombola GR. Enclosing a pen with a postal questionnaire can significantly increase the response rate. J Clin Epidemiol. 2006;59:747–54.
Beebe TJ, Rey E, Ziegenfuss JY, Jenkins S, Lackore K, Talley NJ, et al. Shortening a survey and using alternative forms of prenotification: impact on response rate and quality. BMC Med Res Methodol. 2010;10:50.
van der Mark LB, van Wonderen KE, Mohrs J, Bindels PJ, Puhan MA, Ter Riet G. The effect of two lottery-style incentives on response rates to postal questionnaires in a prospective cohort study in preschool children at high risk of asthma: a randomized trial. BMC Med Res Methodol. 2012;12:186.
Montgomery AA, Peters TJ, Little P. Design, analysis and presentation of factorial randomised controlled trials. BMC Med Res Methodol. 2003;3:26.
SUSPEND was funded by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme (project 80/71/01) and will be published in full in Health Technology Assessment. The Health Services Research Unit of the University of Aberdeen is funded in part by the Chief Scientist Office of the Scottish Government Health and Social Care Directorates. The views and opinions expressed herein are those of the authors and do not necessarily reflect those of the Chief Scientist Office, HTA programme, NIHR, National Health Service, or Department of Health. The authors thank the SUSPEND staff and participants, without whom this study would not have been possible.
The authors declare that they have no competing interests.
KS was responsible for the concept, design, and execution of the study, statistical analysis, and writing the final manuscript. GMcP and SCC were responsible of the design and execution of the study and reviewed and approved the final manuscript. MF was responsible for the design of the information technology systems and reviewed and approved the final manuscript. All authors read and approved the final manuscript.
About this article
Cite this article
Starr, K., McPherson, G., Forrest, M. et al. SMS text pre-notification and delivery of reminder e-mails to increase response rates to postal questionnaires in the SUSPEND trial: a factorial design, randomised controlled trial. Trials 16, 295 (2015). https://doi.org/10.1186/s13063-015-0808-9
- Randomised controlled trial
- Short messenger service (SMS)
- Electronic mail (e-mail)