Researchers propose Reward-Free Policy Optimization (RFPO) for large language models (LLMs), utilizing a pretrained critic to predict future outcomes and provide learning signals without requiring external rewards or completed rollouts.