This paper introduces Evolutionary Safety, a perspective for studying safety in AI systems undergoing recursive self-improvement. It proposes a taxonomy of safety risks and governance principles for maintaining safety guarantees in AI systems that can adapt and evolve.