Summary: For any normal-form game, it's possible to cast the problem of finding a correlated equilibrium in this game as a 2-player zero-sum game. This seems useful because zero-sum games are easy to analyze and more resistant to collusion.

Consider the following class of games (equivalent to the class of normal-form games):

There are $n$ players. Player $i$'s action set is $A_i$. After each player $i$ chooses their action $A_i$, the state $X$ results from these actions (perhaps stochastically), and each player $i$ receives utility $U_i\left(X\right)$.

Often, we are interested in finding the Nash equilibrium of a game like this. One strategy for this is to instantiate the $n$ players as agents. However, this could cause collusion; see here and here for previous writing on collusion. Zero-sum games seem more resistant to collusion (although maybe not 100% resistant). Additionally, 2-player zero-sum games are typically easier to reason about than $n$-player games.

So we might be interested in finding the Nash equilibrium using a zero-sum game. I don't actually know how to find a mixed Nash equilibrium, so instead I'll present a strategy for finding a correlated equilibrium (a superset of Nash equilibria which are computationally easier to find). Here's how it works:

1. The actor chooses actions $A_1,...,A_n$.
2. The critic chooses a player index $I$, observes the action $A_I$, and suggests an alternative action $A_I^{\prime }$.
3. Flip a fair coin. If it comes up heads, observe the state $X$ that results from actions $A_1,...,A_n$, and give the actor utility $U_I\left(X\right)$.
4. If it comes up tails, observe the state $X$ that results from actions $A_1,...,A_{I-1},A_I^{\prime },A_{I+1},...,A_n$, and give the actor utility $-U_I\left(X\right)$.
5. Either way, the critic's utility is the negation of the actor's utility.

It will be useful to use the concept of an $ϵ$-correlated equilibrium. While a correlated equilibrium is where no player can gain any expected utility by strategy modification, an $ϵ$-correlated equilibrium is where no player can gain more than $ϵ$ expected utility by strategy modification.

Note that the critic's policies correspond to mixtures of strategy modifications; the critic can be seen as jointly picking a player $I$ and a strategy modification $\varphi :A_I\to A_I$ for the player. Furthermore, the critic's expected utility is half the expected utility gained by the corresponding player for the average strategy modification in this mixture:

$$\text{expected critic utility}=\frac{1}{2}\left(E\right[U_I\left(X\right)|\text{tails}]-E[U_I\left(X\right)|\text{heads}\left]\right)$$

because the critic's expected utility is half the difference between player $I$'s expected utility given strategy modification ($\text{tails}$) and player $I$'s expected utility given no strategy modification ($\text{heads}$). Some facts result:

1. Suppose the actor chooses $A_1,...,A_n$ from some joint distribution that is an $ϵ$-correlated equilibrium of the original game. Then the actor's expected utility is at least $-ϵ/2$ regardless of the critic's policy.
2. Suppose the actor chooses $A_1,...,A_n$ from some joint distribution that is not an $ϵ$-correlated equilibrium of the original game. Then the critic's best response results in a utility of no more than $-ϵ/2$ for the actor.

Correlated equilibria always exist, so at a Nash equilibrium in the zero-sum game, the actor always outputs a correlated equilibrium and gets expected utility 0.

Perhaps in real life, it is inconvenient to observe the state $X$ resulting from actions $A_1,...,A_{I-1},A_I^{\prime },A_{I+1},...,A_n$, because we can only observe the state by outputting actions, and maybe we always want to output actions from a correlated equilibrium. In this case we could use counterfactual oversight to usually output $A_1,...,A_n$, but run the procedure above occasionally to gather training data. It's not clear when it's acceptable to occasionally output strategy-modified action profiles (instead of action profiles from a correlated equilibrium).