Medical AI Bias Audit Tool

This project audits a clinical machine learning classifier for demographic bias, measuring performance disparities across sex groups using formal fairness metrics from the Fairlearn library. It trains a Random Forest on the UCI Heart Disease dataset, quantifies fairness gaps with Fairlearn's MetricFrame, attempts to mitigate them with ThresholdOptimizer, and reports what actually happened — including where the mitigation didn't help.

Live app: https://medical-bias-audit-3vrioafrnresgpscbwgruk.streamlit.app/

Why this project

Responsible AI / fairness in ML is an active research area, and this project treats "does it work?" and "does it work fairly?" as two separate questions. A model that looks accurate overall can still systematically underdiagnose one demographic group — which in a clinical setting means missed diagnoses, not just a lower accuracy number.

Dataset

UCI Heart Disease dataset (Cleveland subset, 303 patients, 13 clinical features, binary target). Loaded via ucimlrepo; the loader (src/load_data.py) falls back to a mirror CSV with the same schema if that source is unreachable, so the pipeline still runs on restricted networks.

Protected attribute audited: sex. Class balance in this dataset is already uneven — roughly 96 female vs 207 male patients — which is itself a data-representation bias worth naming before any model is trained.

Method

1. Baseline model — Random Forest (100 trees), trained on all 13 features including sex, evaluated overall and split by sex.
2. Fairness audit — Fairlearn MetricFrame computing accuracy, selection rate, false negative rate, and false positive rate per group, plus scalar demographic parity difference and equalized odds difference.
3. Mitigation — Fairlearn ThresholdOptimizer (equalized-odds constraint, post-processing on the fitted model) as the primary mitigation attempt, compared against the naive approach of simply dropping the sex feature.
4. Visualization — three charts: grouped bar comparison, side-by-side confusion matrices, and a feature-importance-vs-correlation-with-sex scatter to flag proxy features.

Key findings (from the run in this repo)

• Baseline accuracy gap: 78.6% (male) vs 94.7% (female) on this train/test split — a 16-point gap, but in the opposite direction of the "female patients underperform" assumption that's common in fairness-audit writeups. This is an important finding in itself: bias direction is dataset- and split-dependent, not something to assume in advance.
• Demographic parity difference: 0.323 (baseline) — the model flags "high risk" for very different proportions of male vs female patients.
• Equalized odds difference: 0.059 (baseline).
• Mitigation result: applying ThresholdOptimizer changed overall accuracy from 83.6% → 85.2%, but on this 61-patient test set it did not meaningfully improve equalized odds (0.059 → 0.059) and slightly worsened demographic parity (0.323 → 0.347).
• Why: with only ~19 female patients in the test split, per-group metrics are estimated from very few samples, so a single train/test split is noisy — a mitigation can look like it "worked" or "failed" almost by chance. This is a genuine limitation of small-sample fairness auditing, not a bug: the honest fix is to repeat the audit across multiple random splits (or use cross-validated fairness metrics) before trusting any single before/after number. That extension is a natural next step for this project.
• Naive debiasing (dropping the sex feature): accuracy dropped (83.6% → 78.7%) and the accuracy gap between groups barely changed (16.2 → 15.7 points) — confirming that removing the protected attribute alone is not sufficient, since other features (chol, thal, exang had the highest correlation with sex among top-importance features here) act as proxies.

Run python src/run_all.py yourself to reproduce these numbers, or regenerate them across multiple seeds — they will shift somewhat given the small dataset.

Fairness metrics, defined

Metric	What it means	Ideal value
Demographic parity difference	Does the model predict "high risk" at the same rate for both groups?	0
Equalized odds difference	Are true-positive and false-positive rates equal across groups?	0
False negative rate (by group)	% of actual disease cases missed — in medical AI, this is a missed diagnosis	as low as possible, and equal across groups

Project structure

medical-bias-audit/
├── app.py                     # Streamlit app
├── requirements.txt
├── src/
│   ├── load_data.py            # Dataset loading + fallback
│   ├── train_baseline.py       # Baseline RF + per-group eval
│   ├── fairness_audit.py       # Fairlearn MetricFrame audit
│   ├── mitigate.py             # ThresholdOptimizer + naive debiasing
│   ├── visualize.py            # 3 charts
│   └── run_all.py              # Runs the full pipeline end-to-end
└── assets/                     # Generated charts + comparison table

Running it

pip install -r requirements.txt
python src/run_all.py        # full pipeline, prints + saves everything
streamlit run app.py         # interactive dashboard

Relationship to prior work

This project extends earlier work on an AI-based medical diagnosis system (DiseaseGPT, TNSCST-funded, Govt. of Tamil Nadu IPD-128) by examining whether such systems can perpetuate demographic disparities — i.e., moving from "does this diagnostic model work?" to "does it work fairly, and is it safe to trust a single audit number without checking for sample-size noise?"

Limitations & next steps

• Single train/test split — results should be validated with repeated stratified splits or cross-validated fairness metrics.
• Only ~96 female patients total; any per-group metric on the test set is estimated from a handful of samples.
• Only sex was audited as a protected attribute here; age_group is also computed in load_data.py and can be swapped in as sensitive_features for a second audit.