Fixing The Good Regulator Theorem

by johnswentworth

10 min read

9th Feb 2021

World Modeling

Frontpage

Fixing The Good Regulator Theorem

The Original Good Regulator Theorem

The Problems

Making The Notion Of “Model” A Little Less Silly

Minimum Entropy -> Maximum Expected Utility And Imperfect Knowledge

Making The Notion Of “Model” A Lot Less Silly

Takeaway

26 comments

Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.

Conant & Ashby’s “Every Good Regulator Of A System Must Be A Model Of That System” opens with:

The design of a complex regulator often includes the making of a model of the system to be regulated. The making of such a model has hitherto been regarded as optional, as merely one of many possible ways.

In this paper a theorem is presented which shows, under very broad conditions, that any regulator that is maximally both successful and simple must be isomorphic with the system being regulated. (The exact assumptions are given.) Making a model is thus necessary.

This may be the most misleading title and summary I have ever seen on a math paper. If by “making a model” one means the sort of thing people usually do when model-making – i.e. reconstruct a system’s variables/parameters/structure from some information about them – then Conant & Ashby’s claim is simply false.

What they actually prove is that every regulator which is optimal and contains no unnecessary noise is equivalent to a regulator which first reconstructs the variable-values of the system it’s controlling, then chooses its output as a function of those values (ignoring the original inputs). This does not mean that every such regulator actually reconstructs the variable-values internally. And Ashby & Conant’s proof has several shortcomings even for this more modest claim.

This post presents a modification of the Good Regulator Theorem, and provides a reasonably-general condition under which any optimal minimal regulator must actually construct a model of the controlled system internally. The key idea is conceptually similar to some of the pieces from Risks From Learned Optimization. Basically: an information bottleneck can force the use of a model, in much the same way that an information bottleneck can force the use of a mesa-optimizer. Along the way, we’ll also review the original Good Regulator Theorem and a few minor variants which fix some other problems with the original theorem.

Continues in source…

Fixing The Good Regulator Theorem – LessWrong