Lecture: Lagrange Multipliers (?? minutes)

FIXME

Lecture notes from Dr. Roundy's 2014 course website:

Usually, (analytically) we maximize functions by setting their derivatives equal to zero. So we could maximize the fairness by $$\frac{\partial\mathcal{F}}{\partial P_i} = 0$$ $$= -k_B (\ln P_i + 1)$$ Using the formula for the fairness function, what can this tell us about $P_i$? It doesn't make much sense at all… it means $P_i = e^{-1}$.$\ddot\frown$ There is a problem with this, which is that $P_i$ can't take just any values, because these are probabilities, and they have to add up to one. This is a constraint, and to solve a constrained maximization, we use the method of Lagrange multipliers. We first define a Lagrangian (The term Lagrangian means different things in different fields. In this case, we aren't using the normal Physics meaning for Lagrangian, but rather the definition from optimization theory, since we are optimizing.): $$\mathcal{L} = \mathcal{F} + \alpha k_B\left(1-\sum_i P_i\right)$$ Note that since the added term should be zero, we haven't changed the thing we want to maximize. Now we maximize this in the same way, but we've got some extra terms that show up in our derivatives. We could, by the way, obtain our constraint by maximizing over $α$ (the Lagrange multiplier) as well as the probabilities $P_i$.

When we minimize $\mathcal{L} $, we find $$\frac{\partial\mathcal{L} }{\partial P_i} = -k_B (\ln P_i + 1) - \alpha k_B$$ $$= 0$$ $$\ln P_i + 1 = -\alpha$$ $$P_i = e^{-1-\alpha}$$ This tells us that all the states are equally probable. To find the actual probabilities, we would need to also apply the constraint: $$= 1-\sum_i P_i$$ $$= 1 - \sum_i e^{-1-\alpha}$$ $$= 1 - e^{-1-\alpha} \sum_i$$ $$= 1 - N e^{-1-\alpha}$$ $$e^{-1-\alpha} = \frac1N$$ $$P_i = \frac1N$$ Which tells us that the probability of each state is equal to one over the total number of states. This makes some degree of sense: if all states are equally probable, then it does make sense that the probability of each state must be one in $N$.

However, this doesn't really make much physical sense, since it means states with a huge energy are just as likely as states with a very small energy. The reason is because we haven't yet taken into account the energy. We did, however, succeed in demonstrating that the maximum fairness occurs when all states are equally probable, as promised.