Feature Reduction as simple as phi
brew install neelsbester/tap/lophiIntel Mac users are covered via Rosetta 2 running the ARM64 binary.
If you don't have Scoop installed, run this in PowerShell first:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
Invoke-RestMethod -Uri https://get.scoop.sh | Invoke-ExpressionThen install lo-phi:
scoop bucket add lophi https://github.com/neelsbester/scoop-bucket
scoop install lophiDownload from GitHub Releases:
- macOS ARM64 (Apple Silicon):
lophi-macos-aarch64.tar.gz - Linux x86_64:
lophi-linux-x86_64.tar.gz - Windows x86_64:
lophi-windows-x86_64.exe
cargo install --path .Lo-phi offers three usage modes to fit your workflow:
The easiest way to get started. The wizard guides you step-by-step through configuration:
# Launch interactive wizard
lo-phi
# Or start with a specific file
lo-phi data.parquetThe wizard will walk you through:
- Selecting your input file (CSV or Parquet)
- Choosing the target column for analysis
- Configuring thresholds (missing values, Gini, correlation)
- Setting optional parameters (solver, weights, columns to drop)
- Reviewing and confirming your settings
Perfect for: Exploratory analysis, learning the tool, one-off feature reduction tasks
View and adjust all options at once with keyboard shortcuts:
# Open the dashboard
lo-phi --manual
# Or with a pre-selected file
lo-phi --manual data.csvKeyboard Shortcuts:
[T]Select target column[C]Edit thresholds (Missing, Gini, Correlation)[S]Configure solver options[W]Select weight column[D]Choose columns to drop[F]Convert CSV to Parquet[A]Advanced options[Enter]Run analysis[Q]Quit
Perfect for: Experienced users, rapid iteration, comparing different configurations
Pure command-line mode for scripting and automation:
lo-phi --no-confirm data.csv \
--target outcome \
--missing-threshold 0.3 \
--gini-threshold 0.05 \
--correlation-threshold 0.4Perfect for: CI/CD pipelines, batch processing, reproducible workflows
Lo-phi includes a fast CSV-to-Parquet converter with compression options:
# Via wizard (option [F] in dashboard or dedicated conversion workflow)
lo-phi
# Select: "Convert CSV to Parquet"
# Via CLI
lo-phi convert input.csv output.parquet --compression snappySupported compression codecs: snappy (default), gzip, lz4, zstd, none
Lo-phi performs three types of feature reduction:
- Missing Value Analysis - Removes features exceeding a null ratio threshold (default: 30%)
- Gini/IV Analysis - Drops features with low predictive power using Weight of Evidence (WoE) binning
- Correlation Analysis - Eliminates one feature from highly correlated pairs (default: 0.40)
After running feature reduction, Lo-phi generates:
{input}_reduced.{csv|parquet}- Your cleaned dataset with dropped features removed{input}_reduction_report.zip- Comprehensive analysis reports:gini_analysis.json- Detailed WoE binning and IV scores per featurereduction_report.json- Full analysis metadata in JSON formatreduction_report.csv- Human-readable summary with correlation details
| Parameter | Default | Description |
|---|---|---|
--missing-threshold |
0.30 | Drop features with >30% missing values |
--gini-threshold |
0.05 | Drop features with Gini coefficient <0.05 |
--correlation-threshold |
0.40 | Drop one feature from pairs with correlation >0.40 |
| Parameter | Default | Description |
|---|---|---|
--use-solver |
true | Enable MILP optimizer for optimal binning boundaries |
--solver-trend |
none | Monotonicity constraint: none, ascending, descending, peak, valley, auto |
--solver-timeout |
30s | Maximum time for solver optimization |
--solver-gap |
0.01 | MIP gap tolerance (1% optimality) |
| Parameter | Default | Description |
|---|---|---|
--binning-strategy |
cart | Binning method: cart (decision tree) or quantile |
--gini-bins |
10 | Target number of bins for Gini analysis |
--prebins |
20 | Initial bins before merging (quantile strategy) |
--cart-min-bin-pct |
5.0 | Minimum percentage of samples per bin (CART) |
--min-category-samples |
5 | Minimum samples for categorical bin |
| Parameter | Default | Description |
|---|---|---|
--target |
(required) | Binary target column for predictive analysis |
--weight |
None | Optional weight column for weighted statistics |
--drop-columns |
None | Comma-separated list of columns to exclude |
--schema-inference-length |
10000 | Rows to scan for type inference (0 = full scan) |
# Using wizard mode
lo-phi credit_data.csv
# Using CLI mode with custom thresholds
lo-phi --no-confirm credit_data.csv \
--target default \
--missing-threshold 0.25 \
--gini-threshold 0.03 \
--correlation-threshold 0.5lo-phi --no-confirm survey_data.parquet \
--target churned \
--weight sample_weight \
--use-solver \
--solver-trend autolo-phi --no-confirm transactions.csv \
--target fraud \
--drop-columns transaction_id,timestamp,customer_namelo-phi --no-confirm income_data.csv \
--target high_earner \
--binning-strategy quantile \
--prebins 30 \
--gini-bins 15# Build
cargo build --release
# Run tests
cargo test --all-features
# Run specific test with output
cargo test --all-features test_missing_analysis -- --nocapture
# Lint and format
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt
# Full CI check
make checkcargo benchLo-phi uses a pipeline architecture with three sequential analysis stages:
Load Dataset → Missing Analysis → Gini/IV Analysis → Correlation Analysis → Save & Report
Key Technologies:
- Polars - High-performance DataFrames with lazy/streaming evaluation
- Rayon - Parallel processing for IV and correlation calculations
- Ratatui/Crossterm - Interactive terminal UI for wizard and dashboard
- HiGHS Solver - MILP optimization for monotonic binning constraints
Lo-phi uses Weight of Evidence (WoE) binning to measure feature predictive power:
- CART Strategy (default): Decision tree-based splits optimizing Gini impurity
- Quantile Strategy: Equal-frequency binning with optional monotonicity constraints
- Solver Integration: MILP optimizer enforces monotonic WoE trends while maximizing IV
Key Constants:
- Pre-bin count: 50 initial quantiles before merging
- Min bin samples: 5 observations minimum per bin
- Laplace smoothing: 0.5 to prevent log(0) errors
Pearson correlation computed using Welford's online algorithm for numerical stability. Categorical features are currently excluded (see Future Enhancements).
- WoE-Encoded Correlation - Measure correlation in "predictive space" using WoE-transformed values instead of raw features
- Cramér's V for Categoricals - Detect association between categorical feature pairs (e.g., city vs postal_code)
- Feature Importance Export - Generate ranked feature importance lists for external modeling tools
This project is licensed under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Run the test suite (
cargo test --all-features) - Commit your changes
- Push to the branch
- Open a Pull Request
