Measuring Goodhart’s Law

Article's Summary:
But it’s important to keep track of how well the true objective is being optimized. We’ll focus on a setting that is particularly clean to analyze, in which we have access to the true objective. In addition, best-of-$n$ sampling has reliable performance and is straightforward to analyze mathematically, making it well-suited to empirical studies of Goodhart’s law and related phenomena. Together, these estimators allow us to easily analyze how the true objective varies with the amount of optimization applied to the proxy objective. In the settings we’ve studied so far, such as summarization, we’ve typically been able to reach a KL of around 10 nats using reinforcement learning before the true objective starts to decrease due to Goodhart’s law.

Article's Keywords: 'proxy', 'learning', 'objective', 'sampling', 'goodharts', 'reinforcement', 'samples', 'law', 'kl', 'bestofn', 'measuring', 'true'