Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,102 @@ This is an educational resource produced by OpenAI that makes it easier to learn

For the unfamiliar: [reinforcement learning](https://en.wikipedia.org/wiki/Reinforcement_learning) (RL) is a machine learning approach for teaching agents how to solve tasks by trial and error. Deep RL refers to the combination of RL with [deep learning](http://ufldl.stanford.edu/tutorial/).

## Quick Installation

```bash
git clone https://github.com/openai/spinningup.git
cd spinningup
pip install -e .
```

**Prerequisites:**
- Python 3.6 or higher
- OpenMPI (for parallel training)
- Platform: Linux or macOS (Windows not officially supported)

For detailed installation instructions including MuJoCo setup, see the [Installation Guide](https://spinningup.openai.com/en/latest/user/installation.html).

## Quick Start

Run your first experiment with PPO on LunarLander:

```bash
python -m spinup.run ppo --env LunarLander-v2 --exp_name quicktest
```

Watch the trained agent:

```bash
python -m spinup.run test_policy data/quicktest/quicktest_s0
```

Plot training results:

```bash
python -m spinup.run plot data/quicktest/quicktest_s0
```

## Implemented Algorithms

Spinning Up includes clean, well-documented implementations of six core deep RL algorithms, available in both **TensorFlow 1** and **PyTorch** versions:

| Algorithm | Full Name | Type | TF1 | PyTorch |
|-----------|-----------|------|-----|---------|
| **VPG** | Vanilla Policy Gradient | On-Policy | ✓ | ✓ |
| **TRPO** | Trust Region Policy Optimization | On-Policy | ✓ | ✓ |
| **PPO** | Proximal Policy Optimization | On-Policy | ✓ | ✓ |
| **DDPG** | Deep Deterministic Policy Gradient | Off-Policy | ✓ | ✓ |
| **TD3** | Twin Delayed Deep Deterministic Policy Gradient | Off-Policy | ✓ | ✓ |
| **SAC** | Soft Actor-Critic | Off-Policy | ✓ | ✓ |

All implementations include:
- [Core algorithm logic](spinup/algos/)
- [Logging and plotting utilities](spinup/utils/)
- [Practical examples](spinup/examples/)
- [Educational exercises](spinup/exercises/)

## Repository Structure

```mermaid
graph TD
A[spinningup] --> B[spinup/]
A --> C[docs/]
A --> D[test/]

B --> E[algos/]
B --> F[utils/]
B --> G[examples/]
B --> H[exercises/]
B --> I[run.py]

E --> J[pytorch/]
E --> K[tf1/]

J --> L[ppo/]
J --> M[ddpg/]
J --> N[sac/]
J --> O[td3/]
J --> P[trpo/]
J --> Q[vpg/]

K --> R[ppo/]
K --> S[ddpg/]
K --> T[sac/]
K --> U[td3/]
K --> V[trpo/]
K --> W[vpg/]

F --> X[logx.py]
F --> Y[plot.py]
F --> Z[mpi_tools.py]

style B fill:#e1f5fe
style E fill:#f3e5f5
style F fill:#e8f5e8
```

## What's Included

This module contains a variety of helpful resources, including:

- a short [introduction](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html) to RL terminology, kinds of algorithms, and basic theory,
Expand Down