Distillation Policy Optimization

arXiv · February 1, 2023 · Significant research

Summary

The paper introduces a novel actor-critic framework called Distillation Policy Optimization that combines on-policy and off-policy data for reinforcement learning. It incorporates variance reduction mechanisms like a unified advantage estimator (UAE) and a residual baseline. The empirical results demonstrate improved sample efficiency for on-policy algorithms, bridging the gap with off-policy methods.

Keywords

reinforcement learning · on-policy · off-policy · actor-critic · sample efficiency

Read original article →

Get the weekly digest

Top AI stories from the GCC region, every week.

YaPO: Learnable Sparse Activation Steering Vectors for Domain Adaptation

arXiv · Jan 13

The paper introduces Yet another Policy Optimization (YaPO), a reference-free method for learning sparse steering vectors in the latent space of a Sparse Autoencoder (SAE) to steer LLMs. By optimizing sparse codes, YaPO produces disentangled, interpretable, and efficient steering directions. Experiments show YaPO converges faster, achieves stronger performance, exhibits improved training stability and preserves general knowledge compared to dense steering baselines.

Diffusion-BBO: Diffusion-Based Inverse Modeling for Online Black-Box Optimization

arXiv · Jun 30

This paper introduces Diffusion-BBO, a new online black-box optimization (BBO) framework that uses a conditional diffusion model as an inverse surrogate model. The framework employs an Uncertainty-aware Exploration (UaE) acquisition function to propose scores in the objective space for conditional sampling. The approach is shown theoretically to achieve a near-optimal solution and empirically outperforms existing online BBO baselines across 6 scientific discovery tasks.

Learning to Identify Critical States for Reinforcement Learning from Videos

arXiv · Aug 15

Researchers at KAUST have developed a new method called Deep State Identifier for extracting information from videos for reinforcement learning. The method learns to predict returns from video-encoded episodes and identifies critical states using mask-based sensitivity analysis. Experiments demonstrate the method's potential for understanding and improving agent behavior in DRL.

Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation

arXiv · Dec 15

Researchers at MBZUAI have demonstrated a method called "Data Laundering" to artificially boost language model benchmark scores using knowledge distillation. The technique covertly transfers benchmark-specific knowledge, leading to inflated accuracy without genuine improvements in reasoning. The study highlights a vulnerability in current AI evaluation practices and calls for more robust benchmarks.