[raphaelvallat]

Summary

This PR adds native support for repeated measures in the Evaluation module. Prior to this PR, the Evaluation module treated all nights as fully independent. When several nights come from the same person, they are not independent, and treating them as separate overstates the sample size: the bias and LoA are barely affected, but their confidence intervals and the p-values of the assumption tests become too small.

This PR adds a subjects argument to EpochByEpochAgreement and SleepStatsAgreement. With one night per subject, the results are unchanged (a warning says that subjects has no effect).

API

• subjects can be a list, array, dict or Series, aligned with the hypnograms or sleep sessions.
• EpochByEpochAgreement.get_sleep_stats() adds the subject IDs as an index level, so SleepStatsAgreement(ebe.get_sleep_stats()) picks them up automatically.
• New n_subjects attribute, also shown in the repr of both classes.
• New SleepStatsAgreement.variance_components property with the between-, within-subject and total SD of the differences for each sleep stat.
• plot_blandaltman(hue="subject_id") colors the nights of each subject alike.

What changes when subjects is passed

EpochByEpochAgreement

Method / output	One night per subject (unchanged)	Several nights per subject
get_agreement(), get_agreement_bystage(), get_confusion_matrix(), plot_hypnograms()	Per-night scores	Unchanged (per-night)
summary() descriptive statistics	Mean, SD, etc. across nights	Nights averaged within each subject first, then mean, SD, etc. across subjects (each subject weighs the same). support is the mean number of epochs per night of each subject
summary(ci_method="boot")	Bootstrap resamples nights	Bootstrap resamples subjects. The BCa warning threshold (< 20) applies to the number of subjects
get_confusion_matrix_proportional()	Mean, SD and CI of the row-normalized matrices across nights	Across subjects (nights averaged within subject first). Extra n_subjects column. Parametric CI uses t(n_subjects - 1)
get_sleep_stats()	Index (scorer, sleep_id)	Index (scorer, subject_id, sleep_id)

SleepStatsAgreement

Quantity	One night per subject (unchanged)	Several nights per subject	Reference
Bias	Mean of the differences	Unchanged
SD of the differences, LoA	Sample SD; LoA = bias ± 1.96 SD	Total SD of the one-way random-effects ANOVA (between-subject + within-subject variance, unequal numbers of nights allowed); LoA = bias ± 1.96 total SD	Bland & Altman 2007
Parametric CI of the bias	SD / √n, t(n - 1)	SE of the mean of all differences under the one-way random-effects model, sqrt(σ²_between × Σm_i² / N² + σ²_within / N), t(n_subjects - 1). Falls back to SD / √n for a stat with no repeated nights left after NaN removal	Bland & Altman 2007 model
Parametric CI of the LoA	√3 × SE, t(n - 1)	MOVER: combines the CI of the bias with the chi-square CI of the total variance	Zou 2013
LoA half-width (regression bias, constant LoA)	1.96 × SD of the residuals; CI from SE ≈ SD / √(2n)	1.96 × total SD of the residuals; chi-square (MOVER) CI	Zou 2013
Euser LoA slope (log_transform=True)	From the SD of the log-ratios	From the total SD of the log-ratios; chi-square (MOVER) CI	Euser 2008, Zou 2013
Regressions (bias on reference, |residuals| on reference) behind the proportional-bias and heteroscedasticity tests and the regression bias / LoA	OLS, CIs and p-values with t(n - 2)	Linear mixed model with a random intercept per subject (statsmodels MixedLM, REML). Slope p-value and Wald CIs with t(n_subjects - 2). Residuals from the fixed effects only. Falls back to OLS with a warning if the fit has no between-subject variance	Myles & Cui 2007, Parker et al. 2016
Unbiased test	One-sample t-test, Cohen's d = bias / SD	t = bias / model-based SE with t(n_subjects - 1); Cohen's d = bias / total SD
Normality test (Shapiro-Wilk)	On all differences	Unchanged (p-value approximate with clustered nights; the effect-size gates limit the consequences)
Bootstrap CIs of bias, LoA, half-width, Euser slope	Resample nights	Cluster bootstrap: resample subjects, keep all their nights
Bootstrap CIs of the regression coefficients	Resample nights	Not bootstrapped: always the Wald CI of the mixed model, even with ci_method="boot". In simulations, a bootstrap over 8-20 subjects covered the true slope only 85-90% of the time at the 95% level
variance_components	sd_between and sd_within are NaN, sd_total is the sample SD	Between-, within-subject and total SD, n_subjects, n_sessions	Bland & Altman 2007, Parker et al. 2016
calibrate(), plot_blandaltman()		Unchanged, plus the hue="subject_id" option

Averaging the nights of each subject before the analysis is not a valid alternative: the bias would be unchanged but the LoA would be too narrow, because averaging removes part of the night-to-night error.

Dependencies

• statsmodels is a new optional dependency (pip install yasa[stats], included in full). It is required only when SleepStatsAgreement gets repeated measures.
• It is pinned to 0.14.5 in the minimum-versions CI job and listed in docs/index.rst.

Docs and tests

• New "Several nights per subject" section in the evaluation tutorial, with the repeated-measures example also shown in the SleepStatsAgreement docstring.
• Tests check the variance components, the bias SE and the LoA against a hand-computed ANOVA, the MOVER interval against the formulas, and the regressions against statsmodels. They also cover the input formats of subjects, invalid inputs, the equivalence of the subjects argument and the subject_id index level, the cluster bootstrap, and the single-night, no-repeated-nights and boundary fallbacks.

References

• Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135-160. https://doi.org/10.1177/096228029900800204
• Bland, J. M., & Altman, D. G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. https://doi.org/10.1080/10543400701329422
• Myles, P. S., & Cui, J. (2007). Using the Bland-Altman method to measure agreement with repeated measures. British Journal of Anaesthesia, 99(3), 309-311. https://doi.org/10.1093/bja/aem214
• Euser, A. M., Dekker, F. W., & le Cessie, S. (2008). A practical approach to Bland-Altman plots and variation coefficients for log transformed variables. Journal of Clinical Epidemiology, 61(10), 978-982. https://doi.org/10.1016/j.jclinepi.2007.11.003
• Zou, G. Y. (2013). Confidence interval estimation for the Bland-Altman limits of agreement with multiple observations per individual. Statistical Methods in Medical Research, 22(6), 630-642. https://doi.org/10.1177/0962280211402548
• Parker, R. A., Weir, C. J., Rubio, N., et al. (2016). Application of mixed effects limits of agreement in the presence of multiple sources of variability: exemplar from the comparison of several devices to measure respiratory rate in COPD patients. PLoS ONE, 11(12), e0168321. https://doi.org/10.1371/journal.pone.0168321
• Menghini, L., Cellini, N., Goldstone, A., Baker, F. C., & de Zambotti, M. (2021). A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code. SLEEP, 44(2), zsaa170. https://doi.org/10.1093/sleep/zsaa170

AI assistance

Model used: Claude Fable 5.1 via Claude Code

Checklist

• An issue was opened or discussed first (preferred)
• The branch is up to date with master, and I checked open PRs for conflicts
• The PR is small and focused, or stacked
• The description starts with my own summary
• Only necessary tests were added, and all tests pass
• The documentation is updated
• The changelog was not modified (it is updated by the maintainer before each release)

If AI-assisted:

• The exact model used is reported
• An AI review was run, the issues were fixed, and a summary was posted
• /simplify was run (preferred, Claude Code)

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.72%. Comparing base (07868d4) to head (1200303).

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #279      +/-   ##
==========================================
+ Coverage   98.61%   98.72%   +0.10%     
==========================================
  Files          13       13              
  Lines        3167     3361     +194     
  Branches      431      456      +25     
==========================================
+ Hits         3123     3318     +195     
  Misses         23       23              
+ Partials       21       20       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[raphaelvallat]

Comparison with matrixCorr::ba_rm (R)

I compared this implementation with the repeated-measures Bland-Altman function of the R package matrixCorr (version 0.12.3, where bland_altman_repeated is a deprecated alias of ba_rm).

Same model, different estimator

Both fit the same random-intercept model to the paired differences, d = bias + subject intercept + night error, with LoA = bias ± 1.96 × sqrt(σ²_subject + σ²_night), i.e. for a new night of a new subject. matrixCorr estimates the variance components with an iterative EM/GLS scheme with damped and clipped updates. YASA uses the closed-form one-way ANOVA of Bland & Altman (2007).

Results on two simulated TST datasets, with a REML fit from statsmodels (MixedLM, intercept only) as arbiter:

Dataset	Component	YASA	statsmodels REML	matrixCorr
Balanced, 10 subjects × 4 nights	sd_between	12.58	12.58	11.91
	sd_within	4.45	4.45	4.45
Unbalanced, 8 subjects, 23 nights (the test fixture of this PR)	sd_between	10.13	10.28	0.00
	sd_within	8.48	8.50	12.84
	bias 95% CI	3.6 to 23.1	3.7 to 23.0	8.1 to 18.6

On the unbalanced dataset, matrixCorr's EM collapsed the subject variance to zero although the data were simulated with a strong subject offset (sd_between = 20). Its bias CI is then about half as wide as the REML one, because the 23 nights are treated as independent. YASA matches REML. Only two datasets were run, so this is a warning sign rather than a characterization of matrixCorr.

Differences in what is estimated

• Bias. matrixCorr reports the GLS intercept, which down-weights subjects with many nights when the subject variance is large. YASA reports the plain mean of all differences, as in Bland & Altman (2007). The two coincide for balanced designs or when the subject variance is zero.
• Confidence intervals. matrixCorr uses Wald intervals with a normal quantile and a delta-method propagation through a numerical Hessian of the profiled REML likelihood. YASA uses a t quantile with n_subjects - 1 degrees of freedom for the bias and the MOVER method of Zou (2013) for the LoA, and also offers a cluster bootstrap. With 8 to 20 subjects, the normal quantile is noticeably anticonservative.
• Proportional bias. matrixCorr can add a slope on the paired mean (classic Bland-Altman convention) and keeps horizontal LoA. YASA regresses on the reference value (Menghini et al. convention, since PSG is the gold standard), fits it as a mixed model, and derives regression-based bias and LoA lines plus a heteroscedasticity test.

Features only one side has

• matrixCorr: optional AR(1) correlation between consecutive nights of a subject, and pairwise matrices for three or more methods. YASA treats the nights of a subject as exchangeable and compares exactly two scorers.
• YASA: assumption tests adapted to the clustering, log-transform and Euser slope, calibration, Bland-Altman plots with a subject hue, bootstrap CIs, the between- and within-subject SD as a user-facing output (variance_components), and the epoch-by-epoch half.

Takeaway

Nothing here suggests changing the implementation. The closed-form ANOVA is exact for balanced data, within 1 to 2% of REML for unbalanced data, and was more robust than matrixCorr's EM in the one unbalanced case tested. One design choice worth keeping in mind: the bias is the unweighted mean of all differences rather than the GLS estimate.

🤖 Generated with Claude Code

[raphaelvallat]

Validation against the full text of the references

I read the five methods papers cited by this PR (Bland & Altman 2007, Zou 2013, Parker et al. 2016, Myles & Cui 2007, Euser et al. 2008) and checked every formula the implementation relies on, then fed the worked examples of Bland & Altman and Zou into the code.

Worked examples are reproduced

Quantity	Paper	YASA
Bland & Altman 2007, within-subject mean square	0.170714026	0.170713993
Bland & Altman 2007, between-subject mean square	4.2090856	4.2090861
Bland & Altman 2007, divisor m0	4.9818182	4.9818182
Bland & Altman 2007, total SD	0.99062408	0.99062411
Bland & Altman 2007, LoA	-1.3394565, 2.5437899	-1.3394566, 2.5437899
Zou 2013, CI of the total variance (p. 637)	0.76, 28.28	0.76, 28.28
Zou 2013, CI of the lower LoA (p. 638)	-9.83, -0.33	-9.83, -0.33
Zou 2013, CI of the upper LoA (p. 638)	1.77, 11.27	1.78, 11.28

The last row differs only by the rounding of the printed inputs. One caveat on the Bland & Altman data: in the author-produced PDF (White Rose repository), the subject labels of the second column of Table 1 are shifted by one row, so subjects 5 to 8 have 6, 4, 4 and 6 pairs rather than the printed 5, 4, 4 and 7. With the labels as printed, Σm_i² is 314 instead of the paper's 312 and the ANOVA table cannot be reproduced. With the shift corrected, every number matches.

Method-by-method

• Variance decomposition (_variance_components): matches Bland & Altman 2007 exactly, including the divisor for unbalanced data and the bias as the mean of all differences, which the paper says "automatically weights the observations correctly".
• MOVER intervals (_variance_ci, _mover_loa_ci): Zou's equations (pp. 637-638) are implemented correctly. Zou builds the total variance from the variance of the subject means and the harmonic mean of the replicates; YASA applies the same MOVER combination to the ANOVA mean squares. The two coincide for balanced data and both rely on an approximate chi-square distribution for unbalanced data.
• Point estimator of the bias: Zou uses the mean of the subject means, YASA the mean of all differences (Bland & Altman). Parker et al. recommend either the raw mean or a random-effects mean, so the choice is supported. On the Bland & Altman data this gives LoA of -1.34 to 2.54 (YASA) versus -1.30 to 2.72 (Zou, Table 3).
• Quantile: Zou uses normal quantiles; YASA uses t(n_subjects - 1), which is more conservative with few subjects. Zou's simulations show near-nominal MOVER coverage with 12 subjects.
• Parker et al. 2016: same model (random intercept per participant), total SD as the root of the summed components, separate reporting of within and total SD. All reflected in variance_components. They use a parametric bootstrap-t for the CIs and note that MOVER depends on normality, which is why the normality gate and the cluster bootstrap are kept.
• Myles & Cui 2007: editorial without formulas; supports random-effects modelling and warns that averaging nights removes measurement error.
• Euser et al. 2008: the back-transformation 2(e^z - 1)/(e^z + 1) is the natural-log form of their 10-log formula. This code predates the PR; the PR only replaces the SD of the log-ratios by the total SD.

Corrections made (1200303)

Two code comments cited papers for statements the papers do not make:

• The model-based SE of the bias was attributed to Zou 2013. It is the variance of the mean of all differences under the one-way random-effects model; Zou's interval uses the mean of the subject means instead.
• Clipping a negative between-subject variance to zero was attributed to Bland & Altman 2007. The paper does not discuss this case; it is the usual convention for ANOVA estimators.

The corresponding cell of the summary table in the PR description was updated as well. No formula change was needed.

🤖 Generated with Claude Code