# 强化学习背景知识

## March 17, 2023

Reading time ~5 minutes

### High-level idea

• Given a state $s$, the reward $r$ is a function that can tell the agent how good or bad an action $a$ is.
• Based on received rewards, the agent learns to take more good actions and gradually filter out bad actions.

### Categorization of methods

• Value-based method: evaluate the goodness of an action given a state using the Q-value function.
• inefficient/impractical when the number of states or actions is large or infinite.
• Policy-gradient method: derive actions directly by learning a policy $\pi(s,a)$ that is a probability distribution over all possible actions.
• suffer from a large fluctuation.
• Actor-critic method (combination of value-based and policy-gradient methods): the actor attempts to learn a policy by receiving feedback from the critic.
• critic-value loss function: $L_1=\sum(R-V(s))^2$, where the discount future reward $R=r+\gamma V(s^\prime)$
• $\gamma \in [0,1]$ is the discount factor that manages the importance levels of future rewards.
• $V(s)$ represents the expected (scalar) reward of a given state
• actor-policy loss function: $L_2=-log(\pi(a \vert s)) * A(s) - \theta H(\pi)$, where the estimated advantage function $A(s)=R-V(s)$
• $H(\pi)$ is the entropy term controlled by the hyperparameter $\theta$.
• $A(s)$ shows how advantageous the agent is when it is in a particular state.
Updated on