Statistical Arbitrage on NSE Equities, from Scratch to Advanced

A brutally honest, expert tutorial series. It teaches the research mechanics of statistical arbitrage on Indian (NSE) equity data and is honest at every step about what a live implementation would actually require. Every number is computed live on real data, no result is shown without costs, and the notebooks keep the losers in.

This is a course about the ideas and the honest evaluation of statistical arbitrage, not a live-trading manual. Long and short equity price series are used because they are the cleanest way to learn the statistics. A long/short equity backtest is treated throughout as a research abstraction: in real Indian markets the short leg needs an implementable vehicle (intraday cash short, SLB-borrowed stock, or a stock-futures proxy), and every such choice changes costs, margin, borrow availability, taxes, tracking error and risk.

What makes it honest

• Gross vs net, always. No P&L without realistic NSE costs subtracted.
• In-sample vs out-of-sample, always. The in-sample curve is a hypothesis, not a result.
• Computed, never hardcoded. Every statistic (cointegration p-values, half-lives, Sharpes) is recomputed in the notebook and described as true "in this data window".
• We show losers. Failed pairs, broken cointegration and negative net Sharpe stay in.
• A "statistical relationship exists" is not the same as "a tradable edge exists".
• Every notebook ends with a "Where this breaks" section, the assumption that kills it in production.
• Educational only. Not investment advice.

Stack

• Data: the OpenAlgo SDK (client.history, NSE equity and index), python-dotenv for credentials. Each notebook reads from a local DuckDB cache with source="db" (fast); source="api" re-downloads from the broker.
• Numerics: numpy, pandas.
• Econometrics: statsmodels (coint, adfuller, kpss, OLS, AutoReg, coint_johansen, acf/pacf), scipy (stats, optimize, odr for total-least-squares hedge ratios).
• ML / validation: scikit-learn, used sparingly (walk-forward and purged cross-validation, PCA, scaling).
• Kalman filter: hand-rolled in numpy, so the series needs no extra dependency.
• Plotting: matplotlib, seaborn.

No proprietary toolbox and no backtest framework. The backtests are plain pandas so the assumptions stay visible.

Universe: the 50 current Nifty 50 constituents plus the NIFTY index, about ten years of daily and about two years of intraday data.

Getting started

git clone https://github.com/marketcalls/statistical-arbitrage.git
cd statistical-arbitrage

# create an environment and install the scientific stack
pip install openalgo python-dotenv numpy pandas statsmodels scipy scikit-learn matplotlib seaborn jupyter duckdb

jupyter lab

Create a .env file in the repository root with your OpenAlgo credentials:

OPENALGO_API_KEY=your_api_key
OPENALGO_API_HOST=http://127.0.0.1:5000

Then start with 00a_building_the_historical_database.ipynb to build the local DuckDB cache once. Every later notebook reads from that cache with source="db". See OpenAlgo for setting up the broker connection and SDK.

Reading order

Part 0 - Setup and orientation

• 00a_building_the_historical_database.ipynb - how the DuckDB cache is built (download once, read fast).
• 00_setup_and_data.ipynb - the data layer, liquidity filters, survivorship and adjustment caveats.
• 00b_from_research_to_real_markets.ipynb - research vs reality; the short-leg vehicles in India; relationship vs edge; the pre-live checklist.

Part 1 - Foundations

• 01_what_is_statistical_arbitrage.ipynb - trade the relationship, not the price; the family tree; market neutrality.
• 02_stationarity_and_random_walks.ipynb - I(1) vs I(0), ADF and KPSS done properly, the Hurst exponent.
• 03_correlation_is_not_cointegration.ipynb - the pivot: the most correlated pairs need not be cointegrated; spurious regression.
• 04_cointegration_mechanics.ipynb - Engle-Granger, the hedge ratio (OLS vs total least squares), the spread, the OU half-life.

Part 2 - Building a pair, then breaking it honestly

• 05_finding_pairs_without_fooling_yourself.ipynb - the cointegration scan and the multiple-testing trap.
• 06_spread_zscore_and_signal.ipynb - the z-score strategy and the first in-sample backtest (it looks great, on purpose).
• 07_the_brutal_reality_check.ipynb - out-of-sample collapse, real NSE costs, de-cointegration, look-ahead.

Part 3 - Doing it properly

• 08_dynamic_hedge_ratio_kalman.ipynb - a hand-rolled Kalman filter for a time-varying hedge ratio (and why it can lose to a static beta).
• 09_baskets_and_johansen.ipynb - cointegrating vectors among several names, and how unstable they are.
• 10_cross_sectional_factor_neutral.ipynb - residual reversion across the universe, and how turnover and costs invert the edge.
• 11_risk_sizing_and_portfolio.ipynb - neutrality, volatility targeting, covariance shrinkage, the stop-on-a-spread dilemma, retiring a broken pair.

Part 4 - From research to reality

• 12_honest_backtesting_and_validation.ipynb - walk-forward, purged CV, the deflated Sharpe, the probability of backtest overfitting, parameter-stability maps.
• 13_implementation_pathways.ipynb - leg vehicles, legging risk, impact from intraday bars, monitoring and the kill switch; the research-to-live gap.

PLAN.md holds the full design notes and rationale behind the series.

What you can do at the end

• Pull and clean NSE equity data from OpenAlgo (DuckDB) into a research-ready panel.
• Tell correlation from cointegration and test a relationship with proper specification.
• Build a pairs trade and, more importantly, judge honestly whether it survives costs, instability and out-of-sample testing.
• Move from a single pair to baskets, dynamic hedges and a cross-sectional market-neutral research model.
• Validate without fooling yourself, and articulate exactly what a live Indian implementation would require (shorting vehicle, borrow, margin, execution, monitoring).

Disclaimer

This material is for education and research only. It is not investment advice and not a recommendation to trade any instrument. Markets carry risk; do your own research and consult a registered adviser before risking capital.

License

Released under the MIT License.

Author

Rajandran R - marketcalls / OpenAlgo