Skip to main content
Download PDF

The PRINTEMPS study: protocol of a cluster-randomized controlled trial of the local promotion of a smartphone application and associated website for the prevention of suicidal behaviors in the adult general population in France

Trials volume 21, Article number: 553 (2020) Cite this article

Abstract

Background

Suicide constitutes a cause of death which could be prevented by e-health programs accessible to the general population. Effective promotion has the potential to maximize the uptake of such programs. However, few e-health programs have been combined with promotion campaigns. The primary objective of this trial is to assess the effectiveness of a tailored promotion, at a local level, of a mobile application and website offering evidence-based content for suicide prevention (the StopBlues program), and to compare the effectiveness of two types of local promotion in terms of their impact on suicidal acts. Secondary objectives focus on the effectiveness of the promotion in terms of the intensity of utilization of the StopBlues program, help-seeking behaviors and the level of psychological impairment of program users.

Methods/design

This is a three-arm, parallel-group, cluster-randomized controlled trial, with before-and-after observation. Thirty-four clusters, corresponding to geographical areas sharing a common local authority in France, will be included. They will be randomly assigned to one of the following arms with a ratio of 1:1:1: a control group; a basic promotion group in which promotion of the StopBlues program will be done by local authorities; and an intensified promotion group in which basic promotion will be supplemented by an additional one in a general practitioner’s waiting room. The primary outcome measure will be the number of suicidal acts within each cluster over a 12-month period following the launch of the intervention. Baseline data will be collected for each cluster over the 12-month period prior to the trial. Secondary outcomes will include length of use of the StopBlues program, measures of help-seeking behaviors and level of psychological distress among users of the program, as well as the cost-effectiveness and budgetary impact of its promotion. A more sustained promotion by local authorities will also be implemented after 12 months in the control group and assessed using the same outcome measures.

Discussion

This research should contribute to the sparse evidence base regarding the promotion of e-health programs and will support the wider delivery of the intervention evaluated if proven effective.

Trial registration

ClinicalTrials.gov, ID: NCT03565562. Registered on 11 June 2018

Peer Review reports

Background

Suicide is a human tragedy, resulting from significant psychological distress, and is among the leading causes of death worldwide [1]. France is one of the most affected Western European countries, with around 11,400 deaths by suicide (13.8 per 100,000 inhabitants) and nearly 200,000 suicide attempts annually [2,3,4]. Moreover, surveys in the general population have estimated that one out of 20 French people has attempted suicide at least once over their life course [5]. The societal costs of suicide are also particularly high. In countries where it has been documented, the magnitude of the economic burden of suicide has been found to range from several million to several billion dollars per year depending on the size of the population [6,7,8,9].

Yet, suicide represents a cause of avoidable death which could be significantly reduced by effective prevention programs in the general population targeting the main challenges in the field of suicide prevention. These challenges include impaired upstream identification and care for people at risk, whereas early detection and management of mental disorders and suicide ideation has been identified as one of the current key priorities for efficient suicide prevention by the World Health Organization (WHO) [10]. These observations underscore the need to develop tools to help self-recognition of suicidal risk and associated mental disorders in the general population, as well as to encourage help-seeking behaviors in this population. Given the high, and constantly increasing, rate of Internet access [11] and smartphone ownership [12], e-health programs could usefully complement current suicide-prevention measures in addressing the remaining challenges. It is estimated that an increasing share of the population resorts to the Internet in search of answers to their health concerns [13, 14], and this is particularly the case for mental health [15, 16]. It has also been shown that health-information-seeking on the Internet is higher among those with stigmatized health conditions, including depression and anxiety, and that users with these conditions increase their utilization of healthcare after accessing such information [17, 18].

The StopBlues program, including a smartphone application and an associated website, was developed in this context. This program is available free-of-charge for the general population living in France. Its content, accessible only to users aged over 18 years (as declared at the first connection), is evidence-based and was built based on previous literature reviews [19, 20] and on advice from expert groups, including people who have experienced psychological distress (participatory design process). All in all, the StopBlues program provides a comprehensive prevention approach including awareness strategies (provision of general information on mental health, stigma, benefits of help-seeking, etc.), screening strategies (mood-tracking and self-assessment questionnaires), strategies for accessing support (locator of nearby helps and emergency “get help now” button) and mental health strategies (safety plan and toolbox with relaxation and mindfulness exercises) as recommended by previous research [19].

However, e-health programs for suicide prevention should be associated with effective promotion strategies to ensure that people who could benefit from these programs are aware of them, as well as to help them identify reliable programs among those available online and encourage their use [19, 21]. In parallel with this targeted effect, such promotion campaigns may have ecological effects similar to those previously observed for other awareness campaigns [22,23,24], by decreasing the stigmatization associated with mental health issues in civil society, and changing the attitudes and norms associated with psychological distress in the adult general population. This would in turn facilitate help-seeking behaviors, leading to a decrease in the occurrence of psychological distress and suicidal ideations, and ultimately preventing suicidal acts. Suicide-prevention measures deployed in the community have been shown to be particularly effective in several studies [25,26,27,28], and previous research has involved cities or counties in the development of such measures [29, 30]. The additional involvement of health professionals, in particular general practitioners (GPs), in promotion campaigns for suicide prevention carried out at the local level could also be beneficial. Indeed, even if people with mental disorders prefer not to seek formal professional help [22], GPs are the health specialists whom people suffering from psychological distress or at risk of suicide most easily turn to, even if not always addressing these topics directly. Among people who committed suicide, 20% did so within 1 day of a visit to their GP [31]. Involving local authorities as well as GPs in promoting suicide-prevention actions through dedicated interventions could, therefore, have complementary benefits, provided that they address the potential lack of sufficient skills, as well as of technical, financial and human resources at their level. However, to the best of our knowledge, no study has involved a large number of local actors in the promotion of an evidence-based e-health tool for suicide prevention, although the development of promotion measures for e-health programs has been advocated by previous work [19, 21].

Objectives of the trial

Primary objective

In this context, the primary objective of our trial (“Programme de Recherche INTerventionnelle et Evaluative Mené pour la Prévention du Suicide”, PRINTEMPS) is to assess the effectiveness of an intervention which includes a tailored, local-level promotion of the StopBlues program, and to compare the effectiveness of two types of local promotion (with or without a passive involvement of GPs through their waiting rooms) on deaths by suicide and suicide attempts over a 12-month period.

Secondary objectives

The secondary objectives will be measured both at the cluster level (geographical area sharing a common authority) and among users of the StopBlues program.

Secondary objectives considered at the cluster level are as follows:

  1. 1.

    To assess the cost-effectiveness and to run a budgetary impact analysis of the intervention

Secondary objectives measured among users of the StopBlues program are as follows:

  1. 2.

    To assess the effectiveness of the intervention in terms of the intensity of the utilization of the program

  2. 3.

    To assess the effectiveness of the intervention for help-seeking behaviors and participation in supportive activities (any activity that a user finds beneficial for their mental health)

  3. 4.

    To assess the effectiveness of the intervention for the evolution of the level of psychological impairment

It should be noted that an additional form of promotion (more sustained promotion by local authorities, described below) will be implemented after 12 months in the control group and evaluated using the same primary outcome measure. In parallel, the long-term effectiveness of the two main types of local promotion (with or without a passive involvement of GPs through their waiting room) will also be assessed.

Methods/design

Study design

This study will be implemented as a three-arm, parallel-group, cluster-randomized controlled superiority trial, with before-and-after observation. Cluster-randomization was deemed mandatory because of potential contamination issues. Indeed, the very nature of the intervention assessed (i.e., a promotion program) cannot be applied at the individual level. This is typically a cluster-cluster intervention, as defined by Eldridge et al. [32]. Clusters were defined as geographical areas sharing a common local authority. These local authorities can either be cities or a grouping of cities (“communautés de communes” or “communautés d’agglomérations”) based on the local health governance of each area. If several adjacent local authorities volunteer individually to participate in the trial, they will be considered as a unique local authority.

Clusters will be randomly assigned to one of the following three arms with a ratio of 1:1:1:

  • Arm 1: no promotion of the StopBlues program (control group)

  • Arm 2: promotion of the StopBlues program by local authorities only (basic promotion group)

  • Arm 3: promotion of the StopBlues program by local authorities with an additional passive involvement of GPs through their waiting rooms (intensified promotion group)

The initial trial duration will be of 12 months. Baseline data on the number of suicidal acts, collected 12 months prior to the intervention, will also be considered in each cluster. After a 12-month post-randomization period, clusters from the control group will be allowed for a further 12-month period to launch their promotional campaign supported by the research team through regular contacts and additional technical and financial resources in comparison to arms 2 and 3 (more sustained promotion by local authorities). This will facilitate the recruitment of clusters as well as their adherence to the intervention during the first 12-month period [33]. It will also enable us to determine whether promotion led by local authorities could be improved by supporting them in the provision of additional promotional resources.

A complete description of the different steps of the trial is presented in Fig. 1 using the Timeline cluster tool of Caille et al. [34].

Fig. 1
figure 1

Timeline cluster tool of the intervention

Full size image

We adhere to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines in reporting the trial in this protocol (see Supplementary material, Fig. 1 and Table 2). Items from the WHO Trial Registration Data Set (TRDS) [35, 36] are provided in Table 1.

Table 1 Items from the World Health Organization (WHO) trial registration data set
Full size table

Recruitment and eligibility

Thirty-four clusters will be recruited in France, through presentations of the research project in local authorities’ annual gatherings and through direct contact from the research team. Included local authorities, located in France (either mainland France or overseas territories), will be all those that volunteer to participate in the intervention, and which provide a signed agreement for participation in the research. Their list will be available upon request to the research team.

The StopBlues program will be freely available in the general population and, therefore, accessible by the populations of all local authorities involved in the trial.

Sample size and power

The number of clusters was fixed by the identification of 34 clusters that agreed to be involved in the trial. The proportion of inhabitants who died from suicide or who made suicide attempts over a 12-month period was estimated to be 0.215% (personal data). We used an expected rate of 0.190% in the experimental group, which represents a reduction of 11.6%, slightly higher than the 10% global target supported by WHO for interventions on suicide [10]. In a two-arm, individually randomized trial, this would lead to a required number of participants of 1,229,350 to achieve a power of 80% with a Type I error rate of 2.5%. Type I error is set at 2.5% because two experimental groups are compared to the control group. Clustering also has to be taken into account. As the number of voluntary clusters was fixed, we calculated what the intraclass correlation coefficient (ICC) would be in order to achieve a power of 80% with our pre-specified number of clusters. Although we acknowledge that such an approach is not classical, it enables us to take into account feasibility reasons. Our mean cluster size equals 138,172 inhabitants and, therefore, randomizing 11 clusters in each arm allows for achieving 80% power if the ICC is around 10−5. Although this is a very low ICC, one has to keep in mind that clusters are geographical areas and ICCs are known to be lower in large clusters as compared to small ones. Moreover, ICC is known to depend on prevalence [37], with small ICC values for prevalence close to 0 or 1, and we expect very low rates (about 0.2%). As a consequence, we considered that a 10−5 ICC value would be appropriate in our trial.

Randomization

Clusters will be randomized all at once and their allocation will be carried out using stratified randomization by minimization. The minimization will use two important factors that can affect the statistical modeling and population homogeneity in the three arms. Indeed, minimization enables a better balance of the prognostic parameters between the different strategy groups than classical randomization, especially when the number of randomization units is small. The considered factors will be the type of local authority forming the cluster (independent cities only, cities which have been grouped for the present study because they were adjacent, or groups of cities, i.e. “communautés de communes” or “communautés d’agglomérations,” which are administratively grouped) and the size of the population, in order to have close distribution of local authority types and median size of population in the three arms.

The minimization parameters will be used to adjust statistical analyses of the primary and secondary outcomes. Randomization will be performed by an independent statistician who will not be involved in the further steps of the research.

Blinding

The very nature of the intervention does not allow any form of blinding. However, participants living in the selected clusters will not be informed that their local authority is included in a randomized trial. Only those who use the StopBlues program will be asked to provide consent to have their data collected through the application or associated website (but this does not correspond to a consent to be included in the randomized trial). Moreover, the primary outcome will be assessed using administrative databases with a pre-specified algorithm. Therefore, although there is no blinding in the present trial, it is not at risk of performance bias or detection bias.

Intervention to be evaluated

The intervention evaluated corresponds to the tailored local promotion of the StopBlues program. Its description below complies with the Template for Intervention Description and Replication (TIDieR) Checklist for intervention description [38].

The StopBlues program will be promoted by the French local authorities participating in the trial. A basic communication toolkit, including a promotion guide, posters, leaflets and key communications messages, will be provided to each local authority at the beginning of the intervention. In addition, a contact person from the municipal staff already involved in health prevention will be appointed in each local authority. They will be in charge of coordinating the promotion of the StopBlues program at the local level and will participate in a day-long training session prior to the beginning of the promotion campaign. Local authorities will then be in charge of promoting the program by choosing their usual municipal communication medium (local newspapers, bulletins and billboards, posters in local shops and bus stops, etc.), by involving local actors of their choice and by developing additional promotion tools if they wish. They will, therefore, be able to develop a tailored intervention adapted to their local context.

Additional promotion of the StopBlues program (intensified promotion) will be carried out in several local authorities by the additional involvement of local GPs. As GPs’ workload is significant in France, due to the fact that they have been put at the heart of care coordination [39], a passive involvement of GPs in the promotion of the StopBlues program was designed. They will receive communication tools on the StopBlues program, such as leaflets or posters, to put in their waiting rooms in order to inform patients and their relatives about the existence of the program.

Finally, the local authorities of the control group will be offered a more sustained promotion at 12 months, which is not part of the main intervention but will also be evaluated during the trial. This promotion will consist of the same promotion as in the basic promotion group to which will be added regular support from the research team with stronger contacts and technical and financial resources for local authorities to implement the promotion and print the tools (flyers, posters, etc.). The inclusion of GPs in this promotion process will be left to local authorities’ discretion.

Outcomes

Primary outcome measure

The primary outcome measure will be the number of suicidal acts (both death by suicides and suicide attempts) during the 12 months after the start of the intervention. The number of deaths by suicide will be obtained from the Epidemiology Center for Medical Causes of Death (“Centre d’épidémiologie sur les causes médicales de décès”, CépiDc) while the number of suicide attempts will be approximated by the number of suicide attempts in contact with the healthcare system (other than primary care only). To estimate the number of suicide attempts seen in acute care, we will use the exhaustive French National Hospital Discharge Database (“Programme de médicalisation des systèmes d’information en médecine, chirurgie, obstétrique,” PMSI-MCO) [40]. To estimate the number of suicide attempts seen in emergency care, we will use the Oscour database which records visits to emergency departments [41]. Participants from clusters included in the study will be identified through their zip code of residence.

Secondary outcome measures

Cost-effectiveness and budgetary impact analysis

The cost-effectiveness and budget impact of the intervention will be assessed using a societal perspective. The budget impact analysis will be carried out at 12 and 24 months while the cost-effectiveness analysis will be evaluated over the entire user’s lifetime.

Costs linked to the implementation of the intervention will be measured at 12 months and at the end of the intervention. They will include fixed costs resulting from the creation of the promotion toolkit for the StopBlues program. They will also include semi-fixed costs, linked to potential specific communication strategies developed by the different local authorities for the implementation of a tailored intervention, and to the human resource costs of involving municipal staff. Costs of suicidal acts will be estimated by considering direct costs for the healthcare system as well as indirect costs for individuals attempting suicide and their relatives.

Moreover, we will use the 12-item Short Form health survey (SF-12) [42] among users of the StopBlues program to obtain a generic measure of perceived health-related quality of life which can then be used to estimate a preference-based measure of health for economic evaluations [43]. This has been previously validated in French [44].

Intensity of the utilization of the StopBlues program by its users

The intensity of the utilization of the StopBlues program by its users will be assessed through a complementary set of measures. They include the number of downloads and connections to the StopBlues program, the time-lapse between the registration and the last connection to the program; the proportion of users who were made aware of the program through the different communication channels (such as communication by their local authority, a relative, GP’s waiting room, etc.) and the proportion of users who subscribed for a relative and not for themselves. The last two measures will be assessed at the first connection to the StopBlues program (T0) by an ad hoc questionnaire. Finally, we will measure the proportion of users who completed a safety plan at 12 and 24 months (Table 2).

Table 2 Measures, data collection methods and timeframe
Full size table

Help-seeking behaviors and participation in supportive activities by users of the StopBlues program

Help-seeking behaviors (towards health professionals or lay individuals) and participation in supportive activities by users of the StopBlues program will be assessed using an ad hoc questionnaire adapted from the General Help-seeking Questionnaire (GHSQ) [45] at users’ first connection (T0) and then regularly until the end of the intervention (Table 2).

Level of psychological impairment of users of the StopBlues program

Several outcome measures will be considered to assess the effect of the intervention on the level of psychological impairment of users of the StopBlues program. They include psychological distress, level of depression, anxiety and suicidal risk (including a past history of suicide attempts). They will be measured using the 12-item General Health Questionnaire (GHQ-12) for psychological distress [46], the Patient Health Questionnaire (PHQ-9) for the level of depression [47], the seven-item Generalized Anxiety Disorder-7 (GAD-7) for the level of anxiety [48] and the Mini International Neuropsychiatric Interview (MINI) suicidality module for the level of suicidal risk [49]. All questionnaires have previously been used in the French language and are estimated to have good measurement properties [50,51,52,53,54,55,56,57,58,59]. The level of psychological distress will be measured at the first connection of the user to the StopBlues program (T0) and then regularly until the end of the intervention (see Table 2). Depression and anxiety levels will be measured for users having a GHQ-12 score suggesting potential psychological distress. Users will also have to indicate their overall mood level through a mood-tracking system including three simple Likert scales ranging from 0 (worst possible) to 100% (perfect). These will focus respectively on users’ global health, mood and overall level of energy. These scales will not be considered in the evaluation of the level of psychological impairment of users of the StopBlues program but will condition the measurement of suicidal risk. The suicidal risk will indeed only be measured if the score of the PHQ-9 questionnaire is superior or equal to 10, or if the score of the GAD-7 questionnaire is superior to 7, or if the score on one of the scales of the mood-tracking system is inferior or equal to 40% or inferior or equal to 50% with a 20% decrease in comparison to the previous score. The suicidal risk will be measured at T0’ (see Table 2) and then regularly until the end of the intervention. Where the user has registered with the StopBlues program for a relative and not for themselves, they will be asked to fill in the Montgomery-Asberg Depression Scale (MADRS) [60] at the first connection (T0).

Long-term effectiveness of the intervention

The long-term effectiveness of the intervention will be measured by the numbers of deaths by suicide and suicide attempts (in contact with the healthcare system other than primary care only) considered separately during the 24 months following the start of the intervention.

Effectiveness of the more sustained promotion by local authorities

The effectiveness of the more sustained promotion by local authorities will be measured by the numbers of deaths by suicide and suicide attempts considered separately in the 12 months following the start of this form of promotion in the control group.

Table 2 provides a summary of all outcome measures collected by level of availability as well as their timeframe.

If users of the StopBlues program stop connecting to it, they will receive push-up notifications on their smartphones or reminder emails regarding the completion of questionnaires, to improve adherence.

Statistical considerations

Characteristics of local authorities and of users of the StopBlues program will be described by the mean and standard deviation (SD) for continuous variables or by number (%) for categorical variables.

To assess the effectiveness of the intervention on the primary outcome (number of suicidal acts at the local authority level), we will compare the number of overall suicidal acts at 12 months and of suicides and suicide attempts separately between the trial arms adjusted on the number observed during the 12 months before the intervention.

We will use a Poisson regression analysis with clusters as analysis units and cluster size as offsets and adjust for baseline data and local authorities’ characteristics (population size, ecological index of deprivation, type of local authorities forming the cluster, etc.). A similar method will be used for all secondary outcomes measured at the level of local authorities.

For secondary outcomes measured on the users of the StopBlues program, mixed-model repeated-measures analyses will be used. This method will enable the inclusion of all users regardless of their level of adherence to the program, which will allow us to carry out the analysis despite some missing values. Adjustments will be made on the initial level of psychological distress of each user and on demographics. By using mixed-model repeated-measures analyses [61], we will also be able to account for potential clustering effects through the incorporation of adequate random effects for local authorities. Transformation of outcome variables will be done as necessary to meet distributional assumptions and to accommodate potential outlying observations. Additional subgroup analyses will be carried out to assess separately the effect of the intervention on users with a history of suicide attempts and those without.

Ethical considerations

This trial was granted ethical approval by the relevant ethics committee (see the “Declarations” section below). It should be noted that participants are not informed that they are included in a randomized trial. Randomization is carried out at the cluster level (geographical areas). There are no opt-out options for people living in geographical areas allocated to one of the two experimental groups, and there is no direct relationship with participants to collect data for the primary outcome. Absence of information and consent to be included in the trial is, therefore, in agreement with the Ottawa Statement on ethical issues for cluster-randomized trials [62]. This was also positively assessed by the relevant ethics committee. In addition, the risk to users of the StopBlues program was evaluated as minimal due to a number of specific features of the program (emergency “get help now” button, safety plan, locator of nearby helps, push-up notifications reminding users to get help, information on prevention regularly updated, etc.). In particular, if someone expressed suicidal ideation while using the program, they would straight away be directed to a call either to general health emergency services (“get help now” button) or to their personal emergency contacts (professionals or relatives) if they filled in the safety plan. All this explains that no data monitoring committee was created for this study and that no detailed stopping guidelines for the trial were enacted. Nevertheless, if an increase in the number of suicidal acts was to be found in the participating local authorities during the first round of analysis of data, the decision to stop the trial could be taken in consultation with funders and health authorities. They will also be involved in the decision to continue the intervention after the trial based on its results.

All users of the StopBlues program will provide written informed consent before their data is collected for research purpose. They will indeed have to sign a consent form for participation in the intervention and will be able to withdraw from the research whenever they want. Their data will be stored on a secure server and any identifying information necessary for the access to the program (email address) will be encrypted through a unique user identifier. Data files used for linkage will be stored separately from raw research data.

It should be noted that both local authorities included in the trial and users of the StopBlues program will not be prohibited from participating in other interventions and/or receiving concomitant care.

Dissemination plan

Findings from the research will be published in aggregated form at the local authority level (no information on individual users of the StopBlues program will ever be provided). Given the broad scope of research objectives for this trial, the dissemination plan for this study includes several publications reporting on preliminary and secondary outcomes. Authorship will be based on actual contributions to the research work. Scientific publications will be complemented by communications aimed at policy-makers in order to ensure the continuation of the intervention, after termination of its funding in the framework of the research project, if results prove satisfactory.

The trial outcomes will be reported in line with the Consolidated Standards of Reporting Trials (CONSORT) guidelines for cluster-randomized trials [63, 64]. An anonymized dataset (therefore, not requiring additional consent) will also be made available by the authors within 3 years of the final collection of data. This dataset will be made available to other researchers for the development of further studies, upon submission of a research protocol to our research consortium and after obtaining adequate ethical approvals.

Discussion

Previous research has shown that awareness campaigns in the community can be effective for suicide prevention, by decreasing the stigma associated with mental disorders and help-seeking, in particular when such campaigns are associated with other interventions [22, 65]. However, so far, there is a dearth of research regarding promotion of e-health tools for suicide prevention. The intervention evaluated in the PRINTEMPS study is, therefore, innovative. In addition, formal and rigorous evaluation of this type of intervention in the general population is lacking in the current literature. We will assess the effectiveness of the intervention through the implementation of a three-arm, parallel-group, cluster-randomized controlled trial, which will add to the currently sparse evidence base, both for the promotion of e-health programs and the involvement of the general population.

If proven effective, the intervention of the PRINTEMPS study could be more widely delivered in French-speaking settings, providing a valuable new resource for suicide prevention. In parallel, the extrapolation of the promotion strategy to other contexts could be explored. This includes extrapolation to other national contexts; for example, by adapting it for use in several other European countries, and to other specific populations such as, for example, an e-health suicide-prevention tool for younger individuals or individuals experiencing mental health issues in the workplace. While participating local authorities are selected on a voluntary basis and cannot be expected to be fully representative of all French local authorities, replication and transferability will be facilitated by a process evaluation of the PRINTEMPS study, which will complement the assessment of the effectiveness of the intervention by assessing the fidelity and the quality of the implementation. Qualitative methods will be used to clarify causal mechanisms and identify the contextual factors possibly influencing them [66], and to identify the key functions of the intervention [67,68,69]. Using interviews with the contact person in charge of managing the implementation of the intervention in the different clusters and an ethnographic case study in a reasoned sample of local authorities, this evaluation should highlight the conditions in which the intervention is effective, and will look for its possible unexpected effects.

Trial status

The protocol reported here corresponds to the version of 1 September 2018. The intervention of the PRINTEMPS study was launched in April 2018 and the recruitment of potential users of the StopBlues program is taking place until March 2020.

Availability of data and materials

The datasets generated and/or analyzed during the current study will be made available from the corresponding author (unique handler of the final dataset) on reasonable request within 3 years of the final collection of data.

Abbreviations

CCTIRS:

French Advisory Committee for Data Processing in Health Research

CépiDc:

Epidemiology Center on Medical Causes of Death

CNIL:

French Data Protection Authority

CONSORT:

Consolidated Standards of Reporting Trials

GAD-7:

Generalized Anxiety Disorder-7

GHQ-12:

12-item General Health Questionnaire

GHSQ:

General Help-seeking Questionnaire

GP:

General practitioner

ICC:

Intraclass correlation coefficient

INSERM:

French National Institute for Health and Medical Research

MADRS:

Montgomery-Asberg Depression Scale

MINI:

Mini International Neuropsychiatric Interview

PHQ-9:

Depression module of the Patient Health Questionnaire

PMSI-MCO:

French National Hospital Discharge Database

PRINTEMPS:

Programme de Recherche INTerventionnelle et Evaluative Mené pour la Prévention du Suicide

SD:

Standard deviation

SF-12:

12-item Short Form health survey

SPIRIT:

Standard Protocol Items: Recommendations for Interventional Trials

TIDieR:

Template for Intervention Description and Replication

TRDS:

Trial Registration Data Set

WHO:

World Health Organization

References

  1. World Health Organization. Preventing suicide: a global imperative [Internet]. Geneva: World Health Organization; 2014. Accessed on 1 Nov 2017. Available from: http://apps.who.int/iris/bitstream/10665/131056/1/9789241564779_eng.pdf.

    Google Scholar 

  2. Organization for Economic Co-operation and Development. “Suicide” in health at a glance 2011: OECD indicators. Paris: Organization for Economic Co-operation and Development; 2011. Accessed on 25 Jan 2017. Available from: http://www.oecd.org/els/health-systems/49105858.pdf.

    Google Scholar 

  3. Mouquet M-C, Bellamy V. Suicides et tentatives de suicide en France. Etude et Résultats; 2006.

    Google Scholar 

  4. Observatoire National du Suicide. Suicide. Etat des lieux des connaissances et perspectives de recherche? Paris: Observatoire National du Suicide; 2014. Accessed on 2 Nov 2017. Available from: http://drees.solidarites-sante.gouv.fr/IMG/pdf/rapportons2014-mel.pdf.

    Google Scholar 

  5. Beck F, Guignard R, Du Roscoät E, Saïas T. Tentatives de suicide et pensées suicidaires en France en 2010. Bull Epidémiol Hebd. 2010, 47:488–92.

  6. O’Dea D, Tucker S. The cost of suicide to society. Wellington: Ministry of Health; 2005.

    Google Scholar 

  7. Kennelly B. The economic cost of suicide in Ireland. Crisis. 2007;28:89–94.

    Article  PubMed  Google Scholar 

  8. Le Jeannic A, Vinet M, Quelen C, Lefèvre T, Clément M, Gervaix J, et al. The economic burden of suicide in France in 2009. Submitted.

  9. Shepard DS, Gurewich D, Lwin AK, Reed GA, Silverman MM. Suicide and suicidal attempts in the United States: costs and policy implications. Suicide Life Threat Behav. 2016;46:352–62.

    Article  PubMed  Google Scholar 

  10. World Health Organization. Mental health action plan 2013–2020. Geneva: World Health Organization; 2013.

    Google Scholar 

  11. World internet users statistics usage and world population stats [Internet]. Accessed on 27 Feb 2014]. Available from: http://www.internetworldstats.com/stats.htm.

  12. IC4D 2012. Information and communications for development: maximizing mobile [Internet] 2012. Available from: http://www.worldbank.org/ict/IC4D2012.

    Google Scholar 

  13. Atkinson NL, Saperstein SL, Pleis J. Using the internet for health-related activities: findings from a national probability sample. J Med Internet Res. 2009;11:e4.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Dickerson S, Reinhart AM, Feeley TH, Bidani R, Rich E, Garg VK, et al. Patient Internet use for health information at three urban primary care clinics. J Am Med Inform Assoc. 2004;11:499–504.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Fox S. The social life of health information. Washington DC: Pew Internet and American Life Project; 2009.

    Google Scholar 

  16. Powell J, Clarke A. Internet information-seeking in mental health: population survey. Br J Psychiatry J Ment Sci. 2006;189:273–7.

    Article  Google Scholar 

  17. Berger M, Wagner TH, Baker LC. Internet use and stigmatized illness. Soc Sci Med. 2005;61:1821–7.

    Article  PubMed  Google Scholar 

  18. Taylor-Rodgers E, Batterham PJ. Evaluation of an online psychoeducation intervention to promote mental health help seeking attitudes and intentions among young adults: randomised controlled trial. J Affect Disord. 2014;168:65–71.

    Article  PubMed  Google Scholar 

  19. Larsen ME, Nicholas J, Christensen H. A systematic assessment of smartphone tools for suicide prevention. PLoS One. 2016;11:e0152285.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  20. De Jaegere E, Portzky G, van den Berg M, Wallyn S. Ethical guidelines for technology-based suicide prevention programmes WP 5. Euregenas: Unit for Suicide Research, Ghent University; 2013.

  21. MasterMind Consortium. Policy recommendations based on MasterMind results. Available from: http://mastermind-project.eu/wp-content/uploads/2017/08/MMind_Policy-brief.pdf.

  22. Coppens E, Van Audenhove C, Scheerder G, Arensman E, Coffey C, Costa S, et al. Public attitudes toward depression and help-seeking in four European countries baseline survey prior to the OSPI-Europe intervention. J Affect Disord. 2013;150:320–9.

    Article  PubMed  Google Scholar 

  23. Gronholm PC, Henderson C, Deb T, Thornicroft G. Interventions to reduce discrimination and stigma: the state of the art. Soc Psychiatry Psychiatr Epidemiol. 2017;52:249–58.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Thornicroft C, Wyllie A, Thornicroft G, Mehta N. Impact of the “Like Minds, Like Mine” anti-stigma and discrimination campaign in New Zealand on anticipated and experienced discrimination. Aust N Z J Psychiatry. 2014;48:360–70.

    Article  PubMed  Google Scholar 

  25. Allen J, Mohatt G, Fok CCT, Henry D, People Awakening Team. Suicide prevention as a community development process: understanding circumpolar youth suicide prevention through community level outcomes. Int J Circumpolar Health. 2009;68:274–91.

    Article  PubMed  Google Scholar 

  26. Bean G, Baber KM. Connect: an effective community-based youth suicide prevention program. Suicide Life Threat Behav. 2011;41:87–97.

    Article  PubMed  Google Scholar 

  27. Walker RL, Ashby J, Hoskins OD, Greene FN. Peer-support suicide prevention in a non-metropolitan U.S. community. Adolescence. 2009;44:335–46.

    Article  PubMed  Google Scholar 

  28. Scott A, Guo B. For which strategies of suicide prevention is there evidence of effectiveness? [Internet]. Copenhagen: WHO Regional Office for Europe; 2012. Accessed on 4 Nov 2017. Available from: http://www.euro.who.int/__data/assets/pdf_file/0003/168843/HEN-Suicide-Prevention-synthesis-report.pdf.

    Google Scholar 

  29. Ono Y, Sakai A, Otsuka K, Uda H, Oyama H, Ishizuka N, et al. Effectiveness of a multimodal community intervention program to prevent suicide and suicide attempts: a quasi-experimental study. PLoS One. 2013;8:e74902.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Székely A, Konkolÿ Thege B, Mergl R, Birkás E, Rózsa S, Purebl G, et al. How to decrease suicide rates in both genders? An effectiveness study of a community-based intervention (EAAD). PLoS One. 2013;8:e75081.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  31. Pearson A, Saini P, Da Cruz D, Miles C, While D, Swinson N, et al. Primary care contact prior to suicide in individuals with mental illness. Br J Gen Pract J R Coll Gen Pract. 2009;59:825–32.

    Article  Google Scholar 

  32. Eldridge SM, Ashby D, Feder GS. Informed patient consent to participation in cluster randomized trials: an empirical exploration of trials in primary care. Clin Trials Lond Engl. 2005;2:91–8.

    Article  Google Scholar 

  33. Frounfelker RL, Ben-Zeev D, Kaiser SM, O’Neill S, Reedy W, Drake RE. Partnering with mental health providers: a guide for services researchers. J Ment Health. 2012;21:469–77.

    Article  PubMed  Google Scholar 

  34. Caille A, Kerry S, Tavernier E, Leyrat C, Eldridge S, Giraudeau B. Timeline cluster: a graphical tool to identify risk of bias in cluster randomised trials. BMJ. 2016;354:i4291.

    Article  PubMed  Google Scholar 

  35. Chan A-W, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 Statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158:200–7.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Chan A-W, Tetzlaff JM, Gøtzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ. 2013;346:e7586.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Gulliford MC, Adams G, Ukoumunne OC, Latinovic R, Chinn S, Campbell MJ. Intraclass correlation coefficient and outcome prevalence are associated in clustered binary data. J Clin Epidemiol. 2005;58:246–51.

    Article  CAS  PubMed  Google Scholar 

  38. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: Template for Intervention Description and Replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.

  39. Chevreul K, Berg Brigham K, Durand-Zaleski I, Hernandez-Quevedo C. France. Health system review. Health Syst Transit. 2015;17(3):1–218.

  40. Agence technique de l’information sur l’hospitalisation. PMSI-MCO. Présentation [Internet]. Accessed on 11 Sep 2019. Available from: https://www.atih.sante.fr/mco/presentation.

  41. Santé Publique France. Réseau OSCOUR® – Organisation de la surveillance coordonnée des urgences [Internet]. Accessed on 11 Sep 2019. Available from: https://www.surveillance-syndromique-sursaud-R/reseau-oscour-R-organisation-de-la-surveillance-coordonnee-des-urgences.

  42. Ware J, Kolinski M, Keller S. How to score the SF-12 physical and mental health summaries: a user’s manual. The Health Institute, New England Medical Centre: Boston; 1995.

    Google Scholar 

  43. Brazier JE, Roberts J. The estimation of a preference-based measure of health from the SF-12. Med Care. 2004;42:851–9.

    Article  PubMed  Google Scholar 

  44. Gandek B, Ware JE, Aaronson NK, Apolone G, Bjorner JB, Brazier JE, et al. Cross-validation of item selection and scoring for the SF-12 Health Survey in nine countries: results from the IQOLA Project. International Quality of Life Assessment. J Clin Epidemiol. 1998;51:1171–8.

    Article  CAS  PubMed  Google Scholar 

  45. Wilson C, Deane F, Ciarrochi J, Rickwood D. Measuring help-seeking intentions: properties of the General Help Seeking Questionnaire. Can J Couns. 2005;39:15–28.

    Google Scholar 

  46. Goldberg D, Williams P. A user’s guide to the General Health Questionnaire. NFER-Nelson, Windsor, Berks, 1988.

  47. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16:606–13.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166:1092–7.

    Article  PubMed  Google Scholar 

  49. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59(Suppl 20):22–33 quiz 34–57.

    PubMed  Google Scholar 

  50. Afzali M, Birmes P, Vautier S. Evidence-based Ordering of MINI suicidality module’s response profiles: application to SMPG data. [Internet]. 2014. Accessed on 14 Nov 2017. Available from: https://hal.archives-ouvertes.fr/hal-01083235/document.

    Google Scholar 

  51. Cuéllar-Flores I, Sánchez-López MP, Limiñana-Gras RM, Colodro-Conde L. The GHQ-12 for the assessment of psychological distress of family caregivers. Behav Med. 2014;40:65–70.

    Article  PubMed  Google Scholar 

  52. Salama-Younes M, Montazeri A, Ismaïl A, Roncin C. Factor structure and internal consistency of the 12-item General Health Questionnaire (GHQ-12) and the Subjective Vitality Scale (VS), and the relationship between them: a study from France. Health Qual Life Outcomes. 2009;7:22.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Norton J, de Roquefeuil G, David M, Boulenger J-P, Ritchie K, Mann A. Prevalence of psychiatric disorders in French general practice using the Patient Health Questionnaire: comparison with GP case-recognition and psychotropic medication prescription. L’Encéphale. 2009;35:560–9.

    Article  CAS  PubMed  Google Scholar 

  54. Arthurs E, Steele RJ, Hudson M, Baron M, Thombs BD, (CSRG) Canadian Scleroderma Research Group. Are scores on English and French versions of the PHQ-9 comparable? An assessment of differential item functioning. PLoS One. 2012;7:e52028.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  55. Micoulaud-Franchi J-A, Lagarde S, Barkate G, Dufournet B, Besancon C, Trébuchon-Da Fonseca A, et al. Rapid detection of generalized anxiety disorder and major depression in epilepsy: validation of the GAD-7 as a complementary tool to the NDDI-E in a French sample. Epilepsy Behav. 2016;57:211–6.

    Article  PubMed  Google Scholar 

  56. Donker T, van Straten A, Marks I, Cuijpers P. Quick and easy self-rating of Generalized Anxiety Disorder: Validity of the Dutch web-based GAD-7, GAD-2 and GAD-SI. Psychiatry Res. 2011;188:58–64.

    Article  PubMed  Google Scholar 

  57. Fann JR, Berry DL, Wolpin S, Austin-Seymour M, Bush N, Halpenny B, et al. Depression screening using the Patient Health Questionnaire-9 administered on a touch screen computer. Psychooncology. 2009;18:14–22.

    Article  PubMed  PubMed Central  Google Scholar 

  58. van Ballegooijen W, Riper H, van Straten A, Kramer J, Conijn B, Cuijpers P. The effects of an internet based self-help course for reducing panic symptoms – Don’t Panic Online: study protocol for a randomised controlled trial. Trials. 2011;12:75.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Furnham A, Cheng H. GHQ score changes from teenage to young adulthood. J Psychiatr Res. 2019;113:46–50 Accessed on 18 Mar 2019; Available from: http://www.sciencedirect.com/science/article/pii/S0022395618312366.

    Article  PubMed  Google Scholar 

  60. Montgomery SA, Asberg M. A new depression scale designed to be sensitive to change. Br J Psychiatry J Ment Sci. 1979;134:382–9.

    Article  CAS  Google Scholar 

  61. Verbeke G. Linear mixed models for longitudinal data. New York: Springer; 2000.

    Google Scholar 

  62. The Ottawa statement on the ethical design and conduct of cluster randomized trials [Internet]. Accessed on 27 Nov 2019. Available from: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001346.

  63. Campbell MK, Piaggio G, Elbourne DR, Altman DG, CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661.

    Article  PubMed  Google Scholar 

  64. Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Hegerl U, Maxwell M, Harris F, Koburger N, Mergl R, Székely A, et al. Prevention of suicidal behaviour: results of a controlled community-based intervention study in four European countries. PLoS One. 2019;14:e0224602.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  66. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

    Article  PubMed  PubMed Central  Google Scholar 

  67. Byng R, Norman I, Redfern S, Jones R. Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. BMC Health Serv Res. 2008;8:274.

    Article  PubMed  PubMed Central  Google Scholar 

  68. Villeval M, Bidault E, Shoveller J, Alias F, Basson J-C, Frasse C, et al. Enabling the transferability of complex interventions: exploring the combination of an intervention’s key functions and implementation. Int J Public Health. 2016;61:1031–8.

    Article  PubMed  Google Scholar 

  69. Hamilton AB, Cohen AN, Glover DL, Whelan F, Chemerinski E, McNagny KP, et al. Implementation of evidence-based employment services in specialty mental health. Health Serv Res. 2013;48:2224–44.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

The authors are indebted to Agnès Caille, for her help on the Timeline cluster tool, to Agnès Dumas for summarizing the process evaluation in the discussion and to Jean-Baptiste Hazo for his support throughout the PRINTEMPS study.

Funding

This research has benefited from the joint assistance of the French National Health Insurance Fund for Employees (CNAMTS), the French Directorate General of Health (DGS), the Arc Foundation for Cancer Research, the French National Cancer Institute (INCa), the French National Institute for Prevention and Education in Health (INPES), the French National Institute of Health and Medical Research (INSERM), the French Inter-Departmental Agency for the Fight against Drugs and Addictive Behaviors (Mildeca) and the French Social Security Scheme for Liberal Professionals (RSI) as part of the “Primary Prevention” call for proposals issued by IReSP and INCa in 2013. This study received funding from the French Public Health Agency (Santé Publique France) in the frame of this call (grant agreement 026/14). All involved agencies had no role in the conception and design of the study and will have no role in its management, in the acquisition, analysis and interpretation of data, or in the submission of findings for publication; the same applies to the study sponsor. None will have ultimate authority over any of these activities.

Author information

Authors and Affiliations

  1. Université de Paris, Unité UMR 1123 ECEVE, INSERM, Paris, France

    Coralie Gandré, Anaïs Le Jeannic, Marie-Amélie Vinet, Kathleen Turmaine, Jean-Luc Roelandt, Corinne Alberti & Karine Chevreul

  2. Health Economics Clinical Research Platform (URC Eco), AP-HP, 1 Place du Parvis Notre-Dame, 75004, Paris, France

    Anaïs Le Jeannic, Marie-Amélie Vinet & Karine Chevreul

  3. Department of Psychiatric Emergency and Acute Care, Lapeyronie Hospital, CHU Montpellier, Montpellier, France

    Philippe Courtet

  4. Neuropsychiatry, Epidemiological and Clinical Research, INSERM, University of Montpellier, Montpellier, France

    Philippe Courtet

  5. World Health Organization Collaborating Centre for Research and Training in Mental Health, Établissement Public de Santé Mentale Lille Metropole, Lille, Hellemmes, France

    Jean-Luc Roelandt

  6. Department of Adult Psychiatry, CHU Lille, Lille, France

    Guillaume Vaiva

  7. Centre National de Ressources et Résilience pour le Psychotraumatisme (Cn2r Lille Paris), Lille, France

    Guillaume Vaiva

  8. INSERM CIC 1415, CHRU de Tours, Tours, France

    Bruno Giraudeau

  9. Tours University, Nantes University, INSERM SPHERE, U1246, Tours, France

    Bruno Giraudeau

  10. Hôpital Robert Debré, CIC-EC, Unité INSERM CIC 1426, Assistance Publique-Hôpitaux de Paris, Paris, France

    Corinne Alberti

Authors
  1. Coralie Gandré
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anaïs Le Jeannic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marie-Amélie Vinet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kathleen Turmaine
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Philippe Courtet
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jean-Luc Roelandt
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Guillaume Vaiva
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Bruno Giraudeau
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Corinne Alberti
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Karine Chevreul
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

CG and ALJ participated in the conception and design of the study and prepared the first draft of the manuscript. MAV and KT participated in the conception and design of the study and in the drafting of the manuscript. PC, JLR, GV, CA participated in the conception and design of the study. BG participated in the design of the study and in the drafting of the manuscript. KC conceived and designed the study, and participated in the drafting of the manuscript. All authors give final approval of the manuscript submitted.

Corresponding author

Correspondence to Karine Chevreul.

Ethics declarations

Ethics approval and consent to participate

This trial was granted ethical approval by the Ethics Committee of the French national institute for health and medical research (INSERM) (Approval n° 15–240 of 7 July 2015), by the French Advisory Committee for Data Processing in Health Research (CCTIRS) (Approval n° 15–793 of 30 September 2015) and by the French Data Protection Authority (CNIL) (Decision DR-2016-421 of 3 November 2016). The trial has been registered at ClinicalTrials.gov (ID: NCT03565562) on 11 June 2018. Any modification to the protocol will be submitted to the approval of the relevant Ethics Committee before implementation. Informed consent will be obtained from all clusters and users of the StopBlues programs.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) Checklist.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gandré, C., Le Jeannic, A., Vinet, MA. et al. The PRINTEMPS study: protocol of a cluster-randomized controlled trial of the local promotion of a smartphone application and associated website for the prevention of suicidal behaviors in the adult general population in France. Trials 21, 553 (2020). https://doi.org/10.1186/s13063-020-04464-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13063-020-04464-2

Keywords

Trials

ISSN: 1745-6215