16.3 Actor-critic architectures and A3C algorithm

Actor-critic architectures combine value-based and policy-based methods in reinforcement learning. They use an actor network to learn the policy and a critic network to estimate the value function, addressing limitations of pure approaches and improving training stability.

The A3C algorithm enhances actor-critic systems with asynchronous training using multiple parallel actors. It employs advantage functions and shared global networks, leading to faster convergence and efficient exploration in continuous control tasks.

Actor-Critic Architectures

Motivation for actor-critic architectures

• Addresses limitations of pure value-based and policy-based methods by combining strengths
• Value-based methods estimate value function (Q-learning, SARSA)
• Policy-based methods directly optimize policy (REINFORCE, Policy Gradient)
• Combining approaches reduces variance in policy gradient estimates, improves sample efficiency, and enhances training stability

Components of actor-critic systems

• Actor network learns policy, outputs action probabilities or continuous values using neural network
• Critic network estimates value function, provides feedback to actor using neural network
• Actor and critic interact: critic evaluates actor's actions, actor improves policy based on feedback
• Training process updates actor using policy gradient with advantage estimates, critic uses temporal difference learning

A3C Algorithm

A3C algorithm and its advantages

• Asynchronous training with multiple actor-learners running in parallel, sharing global network
• Advantage function replaces raw value estimates, reducing policy gradient update variance
• Improves exploration through parallel actors, enhances stability with uncorrelated experiences
• Faster convergence and efficient use of multi-core CPUs
• Algorithm steps:
  1. Initialize global network parameters
  2. Create multiple worker threads
  3. Each worker copies global parameters, interacts with environment, computes gradients, updates global network asynchronously

Implementation of A3C for control

• Choose continuous control environment (MuJoCo, OpenAI Gym)
• Design network architectures: shared base for feature extraction, separate actor and critic heads
• Implement worker class for environment interaction, local updates, gradient computation
• Create global network with shared parameters and asynchronous updates
• Training loop: start multiple worker threads, monitor performance, implement stopping criteria
• Tune hyperparameters: learning rates, discount factor, entropy regularization coefficient
• Evaluate trained model on unseen episodes, compare to baseline methods