Improving analysis of cognitive outcomes in cardiovascular trials using different statistical approaches

Abstract

Background

The Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA) questionnaires are commonly used to measure global cognition in clinical trials. Because these scales are discrete and bounded with ceiling and floor effects and highly skewed, their analysis as continuous outcomes presents challenges. Normality assumptions of linear regression models are usually violated, which may result in failure to detect associations with variables of interest.

Methods

Alternative approaches to analyzing the results of these cognitive batteries include transformations (standardization, square root, or log transformation) of the scores in the multivariate linear regression (MLR) model, the use of nonlinear beta-binomial regression (which is not dependent on the assumption of normality), or Tobit regression, which adds a latent variable to account for bounded data. We aim to empirically compare the model performance of all proposed approaches using four large randomized controlled trials (ORIGIN, TRANSCEND, COMPASS, and NAVIGATE-ESUS), and using as metrics the Akaike information criterion (AIC). We also compared the treatment effects for the methods that have the same unit of measure (i.e., untransformed MLR, beta-binomial, and Tobit).

Results

The beta-binomial consistently demonstrated superior model performance, with the lowest AIC values among nearly all the approaches considered, followed by the MLR with square root and log transformations across all four studies. Notably, in ORIGIN, a substantial AIC reduction was observed when comparing the untransformed MLR to the beta-binomial, whereas other studies had relatively small AIC reductions. The beta-binomial model also resulted in a significant treatment effect in ORIGIN, while the untransformed MLR and Tobit regression showed no significance. The other three studies had similar and insignificant treatment effects among the three approaches.

Conclusion

When analyzing discrete and bounded outcomes, such as cognitive scores, as continuous variables, a beta-binomial regression model improves model performance, avoids spurious significance, and allows for a direct interpretation of the actual cognitive measure.

Trials registration

ORIGIN (NCT00069784). Registered on October 1, 2003; TRANSCEND (NCT00153101). Registered on September 9, 2005; COMPASS (NCT01776424). Registered on January 24, 2013; NAVIGATE-ESUS (NCT02313909). Registered on December 8, 2014.

Background

Over the last two decades, there have been extensive efforts to test interventions to slow cognitive decline and prevent dementia. Results have been at best inconsistent, and researchers have questioned whether limitations of commonly used methodological approaches might be obscuring real effects. For instance, commonly used standard cognitive tests such as the Mini-Mental State Examination (MMSE) [1, 2] and the Montreal Cognitive Assessment (MoCA) [3] have strong ceiling and floor effects (i.e., highly skewed). The floor effect is often a result of the cognitive measures’ poor sensitivity in detecting severe cognitive impairment [4], leading to a clustering of scores at the low end. This precludes their analysis as continuous variables with the consequence that true differences between randomly allocated groups in a trial may not be detected. Indeed, the MMSE and MoCA are often analyzed with generalized linear regression models such as analysis of covariance (ANCOVA) that assumes normality of outcomes for pre- and post-scores or change scores that do not fully account for their psychometric properties. As it is rare for the results of cognitive tests to be normally distributed, the use of linear regression models (which assume normality) may be inappropriate and hamper the ability to detect a true effect.

When distributional assumptions are violated, alternative approaches for the analyses of data include the transformation of the cognitive scores to create a normal distribution (after which linear regression can be used), the use of different regression models that are not dependent on normality, or the use of other assessments where the scores are normally distributed. For example, in international studies, researchers normalize a skewed distribution using country-standardized z-scores [5], or transformed scores such as $\sqrt{\text{Max}+1}-\sqrt{(\text{Max}+1)-\text{score}}$ and $\text{log}\left(\text{Max}+1\right)-\text{log}\left((\text{Max}+1)-\text{score}\right)$ [6]. Regressions that are not dependent on normality (due to bounded data) are beta-binomial regression [7,8,9] and Tobit regression [10,11,12]. In the beta-binomial regression, the outcome measure derived from the sum of all discrete questions is modeled as a discrete binomial and a beta distribution to account for overdispersed data. Tobit regression treats the outcome responses as if they were normally distributed but censors them if they are outside a given range. Although these three approaches have been used in observational studies [8, 9], they are rarely used in clinical trials.

This study aims to empirically evaluate model performance of different approaches analyzing continuous cognitive scores versus linear models of the untransformed data using four large randomized controlled trials. We also compared treatment effects across methods that use the same unit of measurement (i.e., untransformed MLR, beta-binomial, and Tobit). The lessons learnt from these post hoc analyses may inform analyses of data from other cognitive measurements characterized by discrete and bounded outcomes in clinical trials with non-normally distributed scores.

This paper is organized as follows. Data section presents a description of the four trials. Methods section starts by introducing generalized linear regressions to transformed outcomes and non-linear regressions and follows by describing the analysis used to compare the different approaches against the standard or untransformed generalized linear regression. The analysis includes assessing the distribution of the scores, estimating the effect size (except for the ones with the transformation), computing the confidence intervals, and evaluating model fit measured by the Akaike information criterion (AIC) [13]. Finally, in the Results section, we provide a brief discussion of the results obtained and conclude with some general conclusions and recommendations in the “Discussion” section.

Methods

Overview of the four clinical trials

We used data from 4 international clinical trials. The Outcome Reduction with an Initial Glargine INtervention (ORIGIN) trial investigated the effect of insulin glargine (versus standard care) on cardiovascular events in patients with diabetes and pre-diabetes [14]. The MMSE was administered at baseline, year 2, year 5, and the last follow-up. The effect of the interventions on the MMSE was assessed with repeated-measures ANOVA before and after adjusting for selected baseline covariates [15]. The Telmisartan Randomised AssessmeNt Study in ACE iNtolerant subjects with cardiovascular Disease (TRANSCEND) trial assessed the effect of the angiotensin-receptor blocker telmisartan (versus placebo) in patients intolerant to ACE inhibitors with cardiovascular disease or diabetes with end-organ damage [16]. The MMSE was administered at baseline and last follow-up and then dichotomized and analyzed using logistic regression to assess the treatment effect [17]. The Cardiovascular Outcomes for People Using Anticoagulation Strategies (COMPASS) trial evaluated whether rivaroxaban alone or in combination with aspirin would be more effective than aspirin alone for secondary cardiovascular prevention [18]. The New Approach Rivaroxaban Inhibition of Factor Xa in a Global Trial versus ASA to Prevent Embolism in Embolic Stroke of Undetermined Source (NAVIGATE ESUS) trial tested whether rivaroxaban (versus aspirin) would reduce in result in a lower risk of recurrent stroke than aspirin [19, 20]. Both COMPASS and NAVIGATE-ESUS only collected the MoCA scores at baseline and study end. COMPASS has not yet published cognitive measures by the treatment group. However, NAVIGATE-ESUS assessed changes in MoCA scores between treatment groups using the Mann–Whitney U or Kruskal–Wallis test [20].

Overview of the analysis methods

Transformation to enable the use of multivariable linear regression (MLR) Models

The generalized linear regression model [21] assumes normality of outcome and is summarized below.

Let ${\text{y}}_{i}$ be the continuous outcome for individual $i$, $i=1,\dots , n$. The continuous outcome ${\text{y}}_{i}$ is expressed as

$${\text{y}}_{i}={\beta }_{0}+{\beta }_{1}{x}_{i}+{\beta }_{2}{\text{Baseline}}_{i}+{\beta }_{3}{\text{Age}}_{i}+{\beta }_{4}{\text{Sex}}_{i}+{\beta }_{5}{\text{Edu}}_{i} {+ e}_{i}$$

(1)

• ${\beta }_{0}$ is the intercept;

• ${\beta }_{1}$ is the fixed treatment effect for the experimental intervention relative to the control;

• ${x}_{i}\text{ is the binary treatment indicator with }1\text{ for intervention and }0\text{ for control for the }i\mathrm{th}\,\text{individual}$.

• ${\beta }_{k}$ for $k=2, 3, 4,\text{ and }5$ are the fixed effects for the covariates: baseline score, age, sex, and education level, respectively.

• ${e}_{i}\sim N\left(0,{\sigma }_{i}^{2}\right)$ is the error term for individual $i$.

One transformation, such as standardizing cognitive measures, was initially introduced to address methodological issues where the cognitive performance of individuals without impairment, particularly those with higher education, was challenging. This approach was further extended to standardize scores by country, using country-specific baseline parameters, with the baseline mean and standard deviation calculated separately for every country in a multinational trial. This method has the added benefit of accounting for systematic country differences. Other approaches are based on the fact that given that the distributions of the MMSE or MoCA scores are left skewed and that the highest score of each is 30. In these approaches, transformations [expressed as $h\left({y}_{i}\right)$] were done to satisfy the normality assumption of linear regression models [22] as follows: $h\left(\text{score}\right)=\sqrt{\text{Max}+1}-\sqrt{(\text{Max}+1)-\text{score}}$ and $h\left(\text{score}\right)=\text{log}\left(\text{Max}+1\right)-\text{log}((\text{Max}+1)-\text{score})$.

Regression models not dependent on normality: beta-binomial regression

Given the cognitive scores often are calculated by summing equal 0 or 1 scores from a series of questions (e.g., the MMSE was calculated by summing up 30 yes–no questions), the beta-binomial distribution has a similar data structure that consists of a finite sum of Bernoulli variables with a probability parameter analyzed as a random variable that follows a beta distribution.

Let ${y}_{i}$ be a sum of all binary responses of $n$ questions for individual $i$ conditioned on the random variable with a probability parameter ${p}_{i}$ of a Bernoulli distribution. Then ${p}_{i}$ follows a beta distribution with parameters ($a, b$) and,

$$E\left({p}_{i}\right)=\mu = \frac{a}{a+b}, V\left({p}_{i}\right)=\mu (1-\mu )\rho$$

where $\rho ={\left(1+a+b\right)}^{-1}$ is the intraclass correlation coefficient [23]. Then ${y}_{i}$ follows a beta-binomial distribution with

$$E\left(y_i\right)=n\mu,V\left(y_i\right)=n\mu(1-\mu)\left[1+(n-1)\rho\right].$$

The linear predictor of the beta-binomial hierarchical generalized linear model [7] is

$$\text{logit }({p}_{i})={\beta }_{0}+{\beta }_{1}{x}_{i}+{\beta }_{2}{\text{Baseline}}_{i}+{\beta }_{3}{\text{Age}}_{i}+{\beta }_{4}{\text{Sex}}_{i}+{\beta }_{5}{\text{Edu}}_{i}+{\nu }_{i}$$

where ${\nu }_{i}$ is the random intercept effect attributed to individual $i$ and follows a beta distribution. For the covariate ${x}_{i},$ $E\left({p}_{i}\right)=\text{exp}\left({\beta }_{0}+{\beta }_{1}{x}_{i}\right)/(1+\text{exp}\left({\beta }_{0}+{\beta }_{1}{x}_{i}\right))$ [24]. A beta-binomial regression has the random component following a beta distribution with more flexibility on the shape of distribution instead of the common normal distribution with a symmetrical bell shape to model binomial overdispersed. For example, the beta-binomial takes on a U-shaped if both a and b are less than 1 and approximates the binomial distribution if both a and b are greater than 1 [25].

Regression models not dependent on normality: Tobit regression

Tobit regression is an alternative under normality assumptions and in the presence of moderate ceiling or floor effect, by setting a continuous variable as the response constrained or censored to a closed interval [10, 11].

Let ${y}_{i}^{*}$ be the random latent variable that is not censored for individual $i$. Furthermore, it is assumed that ${y}_{i}^{*}$ can be observed for a given range $[l, u]$ and is censored when ${y}_{i}^{*}$ falls outside of that range. The Tobit model with ${y}_{i}^{*}$ conditional on the base-specific parameters ${b}_{i}$ with the linear regression model as the underlying model for ${y}_{i}^{*}$ is given by

$${y}_{i}^{*}|{b}_{i}={\beta }_{0}+{\beta }_{1}{x}_{i}+{\beta }_{2}{\text{Baseline}}_{i}+{\beta }_{3}{\text{Age}}_{i}+{\beta }_{4}{\text{Sex}}_{i}+{\beta }_{5}{\text{Edu}}_{i} {+{z}_{i}{b}_{i}+ e}_{i}$$

with ${b}_{i}\sim N\left(0,\text{ D}\right)$. Then ${y}_{i}$ is obtained from ${y}_{i}^{*}$ as

$${\text{y}}_{i}=l\text{ for }{y}_{i}^{*}\le l$$

$${\text{y}}_{i}={y}_{i}^{*}\text{ for }l<{y}_{i}^{*}\le u$$

$${\text{y}}_{i}=u\text{ for }{y}_{i}^{*}\ge u$$

Although the approach is justified, the estimated coefficients obtained with this approach could go beyond the closed interval due to the artificial situation based on an assumption of censoring and may result in unrealistic interpretations.

Statistical methods

Since the study duration and the frequency of data collection varied across studies, the model used for different analysis approaches had the last follow-up score as the dependent variable and the treatment effect as the independent variable adjusting for the baseline score, age, and sex.

The distributions of the last follow-up MMSE, as well as the country-standardized MMSE score, the square root MMSE, and the log-transformed MMSE, were assessed by plotting histograms and computed skewness and kurtosis. Skewness is a measure of the asymmetry and kurtosis is a measure of “peakedness” of a distribution. A distribution with either an absolute skew value larger than two or an absolute kurtosis larger than seven could be considered as substantial non-normal.

The treatment effect (treatment versus control) in clinical trials on the cognitive function was then analyzed using the untransformed MLR, MLR with country-standardized score, MLR with square root transformation, MLR with log transformation, beta-binomial regression, and Tobit regression. The covariates for the model included baseline score, age, sex, and education (< 12 years of education, ≥ 12 years of education) for both MMSE and MoCA, even though the MoCA score already accounts for the education level by adding one point for those with < 12 years of education as part of its score.

The model performance for the different analysis approaches of the same dataset was compared using the Akaike information criterion (AIC) [13], a measure of the log-likelihood penalized for the number of variables. To ensure comparability of AIC values across methods, the AIC values for the MLRs were obtained using the maximum likelihood estimation. Smaller AIC values indicate a better model fit. The AICs for the models with the transformed outcomes need to have their likelihood multiplied by the corresponding Jacobian matrix to be comparable with other AICs. For the MLR, the normality of residuals was further assessed by plotting the histograms and Q-Q plots. The treatment effects were only compared among untransformed MLR, beta-binomial regression, and Tobit regression given that the units of cognitive measure of transformed MLR became different from the original measure. The treatment effects were reported as mean differences with corresponding 95% confidence intervals (CI) and CI width. The assumption of linearity was assessed using the residual vs predicted plot. In addition, we conducted mixed models with repeated measures (MMRM) and beta-binomial regression for ORIGIN, which is the only study that collected MMSE data multiple times. Analyses were performed using SAS software, Version 9.4 and R Version 4.2.3.

Results

Participants with baseline and last follow-up MMSE scores were included from ORIGIN (n = 11,691 mean age, 63.5 years, 65% male, 38% higher education, mean follow-up, 5.9 years) and TRANSCEND (n = 5815 mean age, 66.9 years, 57% male, 34% higher education, mean follow-up, 4.6 years) (Table 1). Participants with baseline and last follow-up MoCA scores were included from COMPASS (n = 17,864, mean age 68.2 years, 78% male, 47% higher education, mean follow-up, 1.9 years) and NAGIVATE-ESUS (n = 7016, mean age, 66.9 years, 62% male, 46% higher education, mean follow-up, 1 year). The mean baseline and follow-up MMSE scores for ORIGIN and TRANSCEND were 27.9 (SD = 2.9) vs. 27.3 (SD = 3.5) and 27.5 (SD = 3.1) vs. 27.4 (SD = 3.5), respectively, while for COMPASS and NAVIGATE-ESUS, the mean baseline and follow-up MoCA scores were 24.5 (SD = 3.9) vs. 24.4 (SD = 4.0) and 23.1 (SD = 5.5) vs. 23.5 (SD = 5.6), respectively (Tables 1 and 2). There was no significant difference in scores between baseline and follow-up in any of the four studies.

Table 1 Baseline characteristics for ORIGIN, TRANSCEND, COMPASS, and NAVIGATE-ESUS

Table 2 Last follow-up cognitive measures applying transformations and model performance after fitting a mixed linear model

The follow-up MMSE distributions in the ORIGIN and TRANSCEND studies were similarly bounded, with most participants having a maximum score of 30 (Fig. 1). The distribution of change scores from baseline, since the models were adjusted for baseline, exhibited a lack of normality as well (Supplemental Fig. 1). However, the absolute skewness greater than 2 and the absolute kurtosis greater than 8 (Table 2) indicated that the distributions were substantially non-normal. Even after country-standardization of MMSE scores, the distributions still showed large absolute skewness and kurtosis values. Applying a square root or log transformation substantially reduced the skewness and kurtosis values. In contrast, the follow-up MoCA distributions in the COMPASS and NAVIGATE-ESUS studies were left-skewed but still considered normal, with most participants having scores close to the maximum (Fig. 1). The absolute skewness and kurtosis were high but still within the range expected for a normal distribution. Applying a square root or log transformation further improved the skewness of the distribution (Table 2).

Fig. 1

Histograms for the MMSE of ORIGIN and TRANSCEND studies and the MoCA of COMPASS and NAVIGATE-ESUS studies

In terms of fitting the MLR with transformations, we found that the model performance significantly improved for the MMSE scores, and slightly improved for the MoCA scores when we applied either the square root or log transformation as compared to the untransformed MLR (Table 2). For instance, in ORIGIN with MMSE scores, the AIC decreased from 58,013 for the untransformed MLR to 50,317 with the square root transformation, and to 47,162 with the log transformation. In COMPASS with MoCA scores, the AIC decreased from 84,925 for the untransformed MLR to 82,322 with the square root transformation but increased to 85,861 with the log transformation. The histograms of residuals (Supplemental Figs. 2 and 3) became more normally distributed by reducing the initially high skewness and kurtosis values to a more acceptable range after applying square root or log transformations. This was more evident in the Q-Q plots, where most points aligned closely to the straight line for the cognitive measures with a log transformation (Supplemental Figs. 4 and 5).

SQRT: $\sqrt{\text{Max}+1}-\sqrt{(\text{Max}+1)-\text{score}}$

LOG: $\text{log}\left(\text{Max}+1\right)-\text{log}((\text{Max}+1)-\text{score})$

AIC for Beta-binomial were 46,342, 46,342, 22,387, 81,848, and 34,288 for ORIGIN, TRANSCEND, COMPASS, and NAVIGATE-ESUS, respectively.

AIC for Tobit were 49,095, 23,588, 84,112, and 35,641 for ORIGIN, TRANSCEND, COMPASS, and NAVIGATE-ESUS, respectively.

Next, we compared the MLR with two other regression models not only the model performance but also the treatment effects with the three models had the same unit of measure for the cognitive scores.

In the ORIGIN study, the beta-binomial regression model showed a lysignificant higher MMSE score of 0.38 units (95% CI, 0.11 to 0.66; CI width = 0.55, p = 0.006; AIC = 46,342) in the treatment group as compared to the control group. In contrast, both the untransformed MLR (0.049; 95% CI, − 0.06 to 0.15; CI width = 0.21; p = 0.36; AIC = 58,013) and the Tobit regression model (0.13; 95% CI, − 0.02 to 0.28; CI width = 0.30; p = 0.08; AIC = 49,095) showed no significant treatment effect (Fig. 2). A substantial AIC reduction was found in comparing between the MLR untransformed and the beta-binomial. The intracluster correlation coefficients obtained for the beta-binomial models were 0.090, 0.078, 0.022, and 0.044 for ORIGIN, TRANSCEND, COMPASS, and NAVIGATE-ESUS, respectively. Although treatment effects were insignificant across all models for the other studies, a consistent trend was observed in which all three approaches had similar treatment effect estimates. Moreover, the beta-binomial models consistently exhibited slightly lower AIC values, even when compared to the MLR with transformations.

Fig. 2

Between-group mean difference and model performance given by AIC values for the selected methods

Similar results were found in the ORIGIN study with repeated measures, where the beta-binomial and MMRM showed similar treatment effect differences. However, the beta-binomial resulted in a narrower confidence interval width and a smaller AIC (Fig. 3).

Fig. 3

Between-group mean difference and model performance given by AIC values for the ORIGIN study with repeated measures

The baseline cognitive score, baseline age, sex, and education level were mostly significantly associated with the dependent variable across the MLR untransformed, the beta-binomial, and the Tobit regression. Except in the NAVIGATE-ESUS study, the association between sex and cognitive scores was only found significant in the beta-binomial, and Tobit regression, but became not significant in the untransformed MLR. The degree of the associations between the covariates and the outcomes varied with the different approaches but the direction of the association remained consistent (Supplemental Table 1). The assumption of linearity was met, with points mostly hovering around the horizontal line for the generalized linear regression model, beta-binomial, and Tobit regression (Supplemental Figs. 6–7).

Discussion

We used data from four international clinical trials to empirically evaluate the performance of different modeling approaches to analyze cognitive measures, specifically MMSE and MoCA. The methods used were the standard MLR, country-standardized MLR, squared root MLR, log-transformed MLR, beta-binomial, and Tobit regression. The beta-binomial consistently had the lowest AIC values compared to other approaches for both MMSE and MoCA in all studies, suggesting that this approach provided a better effect size estimate associated with an improved model fit, particularly for bounded data. It is also reassuring that the beta-binomial provides a reasonable and good fit, along with unbiased estimates, when the intracluster correlation coefficient is less than 0.1, according to the simulation [26]. The squared root MLR and the log-transformed MLR show a reasonable improvement on the model performance from the MLR standard, but it was challenging to make sense of the between-group mean difference relative to the actual cognitive scores. The effect sizes were significantly different between the beta-binomial and the standard MLR models only in ORIGIN, which had a substantial AIC reduction. For the Tobit regression, there was a small, improved model fit with smaller AIC as compared with the MLR standard. All the four studies had a small effect size and varying magnitude and significance of the covariate association across different models. Based on these results, we recommend considering the beta-binomial to avoid any spurious significance in comparing treatment effects or assessing risk factor effects in discrete and bounded cognitive measures during clinical trials Several studies [7,8,9] support these findings. One study [7] highlighted issues with applying a linear mixed model to outcomes due to ceiling and floor effects. Another study [8] also concluded that the beta-binomial distribution provided the best fit and found similar results when comparing the performance of four distributions (normal, t-family, binomial, and beta-binomial) in analyzing longitudinal data with the MMSE as the response variable. Lastly, another study [9] demonstrated how to analyze and interpret health-related quality of life data using beta-binomial regression.

Due to data availability in four cardiovascular clinical trials, comparisons of different approaches were primarily conducted on response outcomes at two time points (baseline and follow-up). Additionally, we compared the mixed model with repeated measures and the beta-binomial model for the ORIGIN study, which included four time points. If more time points were available, it would be worthwhile to explore the performance of these analysis approaches with a greater number of time points and different methodologies to study the score variations.

Conclusions

These findings demonstrate that beta-binomial regression analyses of continuous cognitive scores (that are derived from clinical instruments with floor and ceiling effects) will optimally assess the effect of an intervention on cognitive status within a randomized clinical trial. It allows for a direct interpretation of results and provides an unbiased estimate in trials using cognitive scores with non-standard psychometric properties. These findings are relevant and applicable to any analysis concerning outcomes influenced by floor and ceiling effects. We aimed to provide non-simulated empirical evidence for outcomes influenced by floor and ceiling effects. Our future work will explore how to estimate the power when analyzing the outcomes using a beta-binomial regression and incorporating repeated measures.