Multilevel Model

Analysis All Participants

Null (unconditional) model

d.Null <- lme(DV~1,random=~1|Case,data=data,
control=list(opt="optim"),na.action="na.omit")
coefficients(d.Null)

##   (Intercept)
## 1    22.36181
## 2    17.65467
## 3    55.76044

intervals(d.Null)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                lower     est.    upper
## (Intercept) 6.483477 31.92564 57.36781
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                    lower     est.    upper
## sd((Intercept)) 7.623288 21.24647 59.21495
## 
##  Within-group standard error:
##    lower     est.    upper 
## 13.02350 16.40752 20.67083

summary(d.Null)

## Linear mixed-effects model fit by REML
##   Data: data 
##        AIC      BIC    logLik
##   336.3845 341.2973 -165.1923
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:    21.24647 16.40752
## 
## Fixed effects:  DV ~ 1 
##                Value Std.Error DF  t-value p-value
## (Intercept) 31.92564  12.54486 36 2.544917  0.0154
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.8747776 -0.6793306 -0.1439468  0.4655299  2.2939603 
## 
## Number of Observations: 39
## Number of Groups: 3

Model with random intercept and change in level (with an autocorrelation component)

d.Level <- lme(DV~1+Phase,random=~1|Case,data=data,
correlation=corAR1(form=~1|Case),control=list(opt="optim"),na.action="na.omit")
coefficients(d.Level)

##   (Intercept)    Phase
## 1    15.73926 9.602826
## 2    13.56880 9.602826
## 3    49.08711 9.602826

intervals(d.Level)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                  lower      est.    upper
## (Intercept) -0.9615989 26.131723 53.22505
## Phase       -4.5345833  9.602826 23.74024
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                    lower     est.    upper
## sd((Intercept)) 6.932852 21.00574 63.64498
## 
##  Correlation structure:
##           lower      est.     upper
## Phi -0.01880442 0.4481973 0.7546592
## attr(,"label")
## [1] "Correlation structure:"
## 
##  Within-group standard error:
##    lower     est.    upper 
## 11.39588 16.34640 23.44749

summary(d.Level)

## Linear mixed-effects model fit by REML
##   Data: data 
##        AIC      BIC    logLik
##   322.8533 330.9079 -156.4266
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:    21.00574  16.3464
## 
## Correlation Structure: AR(1)
##  Formula: ~1 | Case 
##  Parameter estimate(s):
##       Phi 
## 0.4481973 
## Fixed effects:  DV ~ 1 + Phase 
##                 Value Std.Error DF  t-value p-value
## (Intercept) 26.131723 13.345755 35 1.958055  0.0582
## Phase        9.602826  6.963871 35 1.378950  0.1767
##  Correlation: 
##       (Intr)
## Phase -0.291
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.4735422 -0.4999034 -0.1940257  0.4177298  2.4349632 
## 
## Number of Observations: 39
## Number of Groups: 3

Model with random intercept and change in slope (with an autocorrelation component)

d.Time <- lme(DV~1+Time_CTR,random=~1|Case,data=data,
correlation=corAR1(form=~1|Case),control=list(opt="optim"),na.action="na.omit")
# Rename variables and use only complete cases
coefficients(d.Time)

##   (Intercept) Time_CTR
## 1    16.06773 2.162206
## 2    17.92851 2.162206
## 3    58.65117 2.162206

intervals(d.Time)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                 lower      est.     upper
## (Intercept) 1.0873106 30.882469 60.677627
## Time_CTR    0.5279087  2.162206  3.796503
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                    lower     est.    upper
## sd((Intercept)) 8.723114 24.76794 70.32473
## 
##  Correlation structure:
##           lower      est.     upper
## Phi -0.03559898 0.3624528 0.6612477
## attr(,"label")
## [1] "Correlation structure:"
## 
##  Within-group standard error:
##    lower     est.    upper 
## 10.82437 14.67279 19.88943

summary(d.Time)

## Linear mixed-effects model fit by REML
##   Data: data 
##        AIC      BIC    logLik
##   323.4649 331.5195 -156.7324
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:    24.76794 14.67279
## 
## Correlation Structure: AR(1)
##  Formula: ~1 | Case 
##  Parameter estimate(s):
##       Phi 
## 0.3624528 
## Fixed effects:  DV ~ 1 + Time_CTR 
##                 Value Std.Error DF  t-value p-value
## (Intercept) 30.882469  14.67664 35 2.104192  0.0426
## Time_CTR     2.162206   0.80503 35 2.685871  0.0110
##  Correlation: 
##          (Intr)
## Time_CTR -0.018
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.6384733 -0.4539238 -0.1369975  0.4210153  2.5520479 
## 
## Number of Observations: 39
## Number of Groups: 3

Model with random intercept and change in level/slope in Tx (with autocorrelation)

d.Model <- lme(DV~1+Phase+Time_PhaseCTR,random=~1|Case,data=data,
correlation=corAR1(form=~1|Case),control=list(opt="optim"),na.action="na.omit")

# Rename variables and use only complete cases
coefficients(d.Model)

##   (Intercept)    Phase Time_PhaseCTR
## 1    14.36588 10.17285      1.840509
## 2    13.13904 10.17285      1.840509
## 3    47.92556 10.17285      1.840509

intervals(d.Model)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                   lower      est.     upper
## (Intercept)   -2.427525 25.143492 52.714510
## Phase         -4.810155 10.172848 25.155851
## Time_PhaseCTR -1.996903  1.840509  5.677921
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                    lower     est.    upper
## sd((Intercept)) 6.800279 21.00765 64.89753
## 
##  Correlation structure:
##          lower      est.     upper
## Phi 0.01384826 0.4938386 0.7888575
## attr(,"label")
## [1] "Correlation structure:"
## 
##  Within-group standard error:
##    lower     est.    upper 
## 11.34610 16.90492 25.18718

summary(d.Model)

## Linear mixed-effects model fit by REML
##   Data: data 
##        AIC      BIC   logLik
##   320.8441 330.3452 -154.422
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:    21.00765 16.90492
## 
## Correlation Structure: AR(1)
##  Formula: ~1 | Case 
##  Parameter estimate(s):
##       Phi 
## 0.4938386 
## Fixed effects:  DV ~ 1 + Phase + Time_PhaseCTR 
##                   Value Std.Error DF   t-value p-value
## (Intercept)   25.143492 13.566782 34 1.8533129  0.0725
## Phase         10.172848  7.372638 34 1.3798112  0.1767
## Time_PhaseCTR  1.840509  1.888263 34 0.9747101  0.3366
##  Correlation: 
##               (Intr) Phase 
## Phase         -0.311       
## Time_PhaseCTR -0.104  0.191
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.3735497 -0.3437985 -0.1650619  0.3747701  2.4232263 
## 
## Number of Observations: 39
## Number of Groups: 3

Results

Value	Score	p-Value
Autocorrelation	.494
Level	10.17	.177
Slope	1.84	.337

Although the effect size is rather large, it is not significant. Additionally, the obtained change in slope between phases is not significant. This does not necessarily suggest a functional relation cannot be determined. As defined in Carter (2013): effect size does not imply causation and effect sizes are independent of demonstrations of experimental control. I think this is worth taking into consideration when comparing this calculation to the visual analysis and Tau-U.

Analysis Participants 1 and 2 Only

Null (unconditional) model

d.Null <- lme(DV~1,random=~1|Case,data=data2,
control=list(opt="optim"),na.action="na.omit")
coefficients(d.Null)

##   (Intercept)
## 1    19.48376
## 2    19.43932

intervals(d.Null)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                lower     est.   upper
## (Intercept) 13.32787 19.46154 25.5952
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                       lower      est.       upper
## sd((Intercept)) 3.89948e-12 0.3992937 40886326955
## 
##  Within-group standard error:
##    lower     est.    upper 
## 11.43247 15.08514 19.90486

summary(d.Null)

## Linear mixed-effects model fit by REML
##   Data: data2 
##        AIC      BIC    logLik
##   215.8996 219.5562 -104.9498
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:   0.3992937 15.08514
## 
## Fixed effects:  DV ~ 1 
##                Value Std.Error DF  t-value p-value
## (Intercept) 19.46154  2.971883 24 6.548556       0
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.0897689 -0.6279454 -0.2957571  0.6142629  2.6858373 
## 
## Number of Observations: 26
## Number of Groups: 2

Model with random intercept and change in level (with an autocorrelation component)

d.Level <- lme(DV~1+Phase,random=~1|Case,data=data2,
correlation=corAR1(form=~1|Case),control=list(opt="optim"),na.action="na.omit")
coefficients(d.Level)

##   (Intercept)    Phase
## 1    9.262197 15.60619
## 2    9.261188 15.60619

intervals(d.Level)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                  lower      est.    upper
## (Intercept) 0.01550107  9.261693 18.50788
## Phase       4.17728970 15.606188 27.03509
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                        lower      est.        upper
## sd((Intercept)) 6.769095e-25 0.1030037 1.567383e+22
## 
##  Correlation structure:
##          lower        est.     upper
## Phi -0.3731402 0.009216403 0.3888969
## attr(,"label")
## [1] "Correlation structure:"
## 
##  Within-group standard error:
##    lower     est.    upper 
## 10.01420 13.31154 17.69458

summary(d.Level)

## Linear mixed-effects model fit by REML
##   Data: data2 
##        AIC      BIC    logLik
##   207.3619 213.2522 -98.68097
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:   0.1030037 13.31154
## 
## Correlation Structure: AR(1)
##  Formula: ~1 | Case 
##  Parameter estimate(s):
##         Phi 
## 0.009216403 
## Fixed effects:  DV ~ 1 + Phase 
##                 Value Std.Error DF  t-value p-value
## (Intercept)  9.261693  4.469658 23 2.072126  0.0496
## Phase       15.606188  5.524790 23 2.824757  0.0096
##  Correlation: 
##       (Intr)
## Phase -0.808
## 
## Standardized Within-Group Residuals:
##         Min          Q1         Med          Q3         Max 
## -1.49256840 -0.44420007  0.05546369  0.38550114  2.63918458 
## 
## Number of Observations: 26
## Number of Groups: 2

Model with random intercept and change in slope (with an autocorrelation component)

d.Time <- lme(DV~1+Time_CTR,random=~1|Case,data=data2,
correlation=corAR1(form=~1|Case),control=list(opt="optim"),na.action="na.omit")
# Rename variables and use only complete cases
coefficients(d.Time)

##   (Intercept) Time_CTR
## 1    16.67316 1.863349
## 2    16.67540 1.863349

intervals(d.Time)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                  lower      est.     upper
## (Intercept) 10.4306487 16.674279 22.917910
## Time_CTR     0.4292885  1.863349  3.297409
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                        lower     est.       upper
## sd((Intercept)) 1.318692e-12 0.220448 36852668914
## 
##  Correlation structure:
##          lower       est.     upper
## Phi -0.3071034 0.08683331 0.4553658
## attr(,"label")
## [1] "Correlation structure:"
## 
##  Within-group standard error:
##     lower      est.     upper 
##  9.967724 13.319463 17.798254

summary(d.Time)

## Linear mixed-effects model fit by REML
##   Data: data2 
##       AIC      BIC    logLik
##   211.223 217.1133 -100.6115
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:    0.220448 13.31946
## 
## Correlation Structure: AR(1)
##  Formula: ~1 | Case 
##  Parameter estimate(s):
##        Phi 
## 0.08683331 
## Fixed effects:  DV ~ 1 + Time_CTR 
##                 Value Std.Error DF  t-value p-value
## (Intercept) 16.674279 3.0182039 23 5.524570  0.0000
## Time_CTR     1.863349 0.6932322 23 2.687914  0.0131
##  Correlation: 
##          (Intr)
## Time_CTR -0.345
## 
## Standardized Within-Group Residuals:
##        Min         Q1        Med         Q3        Max 
## -1.7157790 -0.3945040 -0.1035519  0.3032299  2.8332068 
## 
## Number of Observations: 26
## Number of Groups: 2

Model with random intercept and change in level/slope in Tx (with autocorrelation)

d.Model <- lme(DV~1+Phase+Time_PhaseCTR,random=~1|Case,data=data2,
correlation=corAR1(form=~1|Case),control=list(opt="optim"),na.action="na.omit")

# Rename variables and use only complete cases
coefficients(d.Model)

##   (Intercept)    Phase Time_PhaseCTR
## 1    9.387673 15.00769      1.374845
## 2    9.387043 15.00769      1.374845

intervals(d.Model)

## Approximate 95% confidence intervals
## 
##  Fixed effects:
##                    lower      est.    upper
## (Intercept)   -0.2033365  9.387358 18.97805
## Phase          3.1577721 15.007691 26.85761
## Time_PhaseCTR -1.3332896  1.374845  4.08298
## attr(,"label")
## [1] "Fixed effects:"
## 
##  Random Effects:
##   Level: Case 
##                        lower      est.        upper
## sd((Intercept)) 2.061527e-26 0.1018918 5.036046e+23
## 
##  Correlation structure:
##          lower       est.     upper
## Phi -0.3615714 0.05043287 0.4459583
## attr(,"label")
## [1] "Correlation structure:"
## 
##  Within-group standard error:
##     lower      est.     upper 
##  9.925827 13.346116 17.944984

summary(d.Model)

## Linear mixed-effects model fit by REML
##   Data: data2 
##       AIC      BIC    logLik
##   205.921 212.7339 -96.96049
## 
## Random effects:
##  Formula: ~1 | Case
##         (Intercept) Residual
## StdDev:   0.1018918 13.34612
## 
## Correlation Structure: AR(1)
##  Formula: ~1 | Case 
##  Parameter estimate(s):
##        Phi 
## 0.05043287 
## Fixed effects:  DV ~ 1 + Phase + Time_PhaseCTR 
##                   Value Std.Error DF  t-value p-value
## (Intercept)    9.387358  4.624533 22 2.029904  0.0546
## Phase         15.007691  5.713908 22 2.626520  0.0154
## Time_PhaseCTR  1.374845  1.305834 22 1.052848  0.3038
##  Correlation: 
##               (Intr) Phase 
## Phase         -0.803       
## Time_PhaseCTR -0.013 -0.056
## 
## Standardized Within-Group Residuals:
##         Min          Q1         Med          Q3         Max 
## -1.65928837 -0.42294962  0.04590414  0.29349822  2.77080473 
## 
## Number of Observations: 26
## Number of Groups: 2

Results

Value	Score	p-Value
Autocorrelation	.050
Level	15.01	.015
Slope	1.37	.304

With Participant 3 removed, the obtained status tracking data from Participants 1 and 2 generate a very large effect with a p-value less than .05. This provides further evidence that Participants 1 and 2 responded to the intervention and demonstrated a significant change in level between the baseline and experimental phases.