Push the expectation operators in Bellman equations for Q

docs/spinningup/rl_intro.rst

@@ -394,7 +394,7 @@ The Bellman equations for the on-policy value functions are
 
     \begin{align*}
     V^{\pi}(s) &= \underE{a \sim \pi \\ s'\sim P}{r(s,a) + \gamma V^{\pi}(s')}, \\
-    Q^{\pi}(s,a) &= \underE{s'\sim P}{r(s,a) + \gamma \underE{a'\sim \pi}{Q^{\pi}(s',a')}},
+    Q^{\pi}(s,a) &= r(s,a) + \gamma \underE{s'\sim P \\ a'\sim \pi}{Q^{\pi}(s',a')},
     \end{align*}
 
 where :math:`s' \sim P` is shorthand for :math:`s' \sim P(\cdot |s,a)`, indicating that the next state :math:`s'` is sampled from the environment's transition rules; :math:`a \sim \pi` is shorthand for :math:`a \sim \pi(\cdot|s)`; and :math:`a' \sim \pi` is shorthand for :math:`a' \sim \pi(\cdot|s')`. 
@@ -406,7 +406,7 @@ The Bellman equations for the optimal value functions are
 
     \begin{align*}
     V^*(s) &= \max_a \underE{s'\sim P}{r(s,a) + \gamma V^*(s')}, \\
-    Q^*(s,a) &= \underE{s'\sim P}{r(s,a) + \gamma \max_{a'} Q^*(s',a')}.
+    Q^*(s,a) &= r(s,a) + \gamma \underE{s'\sim P}{ \max_{a'} Q^*(s',a')}.
     \end{align*}
 
 The crucial difference between the Bellman equations for the on-policy value functions and the optimal value functions, is the absence or presence of the :math:`\max` over actions. Its inclusion reflects the fact that whenever the agent gets to choose its action, in order to act optimally, it has to pick whichever action leads to the highest value.