Theorem 4: ${lim}_{\alpha \to 1}{H}_{\alpha }\left(h\right)$ exists for all infradistributions over a finite set $X$, and equals $-{sup}_{q}dh(1;\ln (q\left)\right)$

So, as a recap, here's how it works. $\alpha >1$. The Renyi entropy for $\alpha >1$ is:

$\frac{\alpha }{1-\alpha }\ln \left({sup}_q\left(h\left(q^{\frac{\alpha -1}{\alpha }}\right)\right)\right)$

Where $q$ is taken over functions with ${\sum }_i{q}_i=1$ (we have finitely many points, we're summing it up over all of those). And our task is to take the limit as $\alpha$ tends to 1.

Our first step is to let $\alpha =1+ϵ$, and with this reindexing, we can instead take the limit as $ϵ$ tends to 0, and get the same answer. So, we're trying to find:

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left({sup}_q\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)\right)$

Well, we don't even necessarily know it exists. Looking at this, in the $ϵ\to 0$ limit, if $q$ is such that $\forall i:q_i>0$, then $q^{\frac{ϵ}{1+ϵ}}$ will tend to 1 for all $i$. And so, $h$ of that will tend to 1, and ln of that will tend to 0 from below, and hopefully the really tiny negative number cancels out with the fact that we're also dividing by a really tiny negative number. To do our approximation work, we're going to need that $q$ is bounded above 0 by something. We can show that the limit exists if we were dealing with a family of $q$ that are bounded above 0.

Accordingly, our proof strategy is going to be getting some bounds (for all $ϵ$ sufficiently close to 0) on how much the results differ if we're taking the sup over all appropriate $q$, vs if we're looking purely at the $q$ that are bounded above 0 by some (small, but not varying with $ϵ$) number. Then, we can get some upper bounds and lower bounds, which we will be able to show converge in the $ϵ\to 0$ limit. The problem is that the bounds won't be tight enough to produce the same limit, because we're blowing things up with a division by $ϵ$. So, all we're going to be able to conclude is that the relevant $ϵ\to 0$ sequence has all of its limit points lying in an interval. However, the lower our "bound above 0" cutoff is, the tighter the interval will be. So, if we take the limit as our "bound above 0" cutoff shrinks to 0, the interval will shrink to a point, and we will have shown that the limit does in fact exist. 

Let's get started on our upper bounds and lower bounds, respectively. Fix some $\delta$ which, importantly, won't vary with $ϵ$ (yes, this will lead to a bit of weirdness where we start looking at behavior when $ϵ<<\delta$). For notation, ${sup}_{q\ge \delta }$ is maximizing $q$ which fulfill the relevant property of $\forall i:q_i\ge \delta$. Accordingly, $\delta$ should be less than 1 divided by the number of possible finite outcomes. So, we can start establishing our upper bound as:

${sup}_q\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)\ge {sup}_{q\ge \delta }\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)$

And the natural logarithm is monotonic, so

$\ln \left({sup}_q\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)\right)\ge \ln \left({sup}_{q\ge \delta }\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)\right)$

And then we multiply both sides by a negative number, flipping the sign, so:

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_q\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)\right)\le \frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\left(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\right)\right)$

And bam, we have an upper bound. The lower bound is significantly more tricky. Let's start with our first question. Given a $q\ngeqq \delta$, how can we adjust it to a $q^{\prime }$ that fulfills that property? Well, just have:

$q_i^{\prime }:=\frac{q_i+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}$

We should check that A: this still adds up to 1, and B: that when $q_i=0$, $q_i^{\prime }\ge \delta$. For later, we'll also need to check C: what's the critical value where this transformation switches from increasing the value to decreasing the value? Let's hit A first.

${\sum }_i{q}_i^{\prime }={\sum }_i\frac{q_i+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}=\frac{1}{1+|X|\frac{\delta }{1-|X|\delta }}{\sum }_i(q_i+\frac{\delta }{1-|X|\delta })$

$=\frac{1}{1+|X|\frac{\delta }{1-|X|\delta }}({\sum }_i{q}_i+{\sum }_i\frac{\delta }{1-|X|\delta })=\frac{1}{1+|X|\frac{\delta }{1-|X|\delta }}(1+|X|\frac{\delta }{1-|X|\delta })=1$

Excellent. For B, we can quickly check that this function is still in range by inspection if $q_i$ is 1, because we're dividing a number by a slightly bigger number. For being above $\delta$, observe that:

$\frac{0+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}=\frac{\frac{\delta }{1-|X|\delta }}{\frac{1-|X|\delta +|X|\delta }{1-|X|\delta }}=\frac{\delta }{1-|X|\delta +|X|\delta }=\delta$

And finally, let's check the threshold where this starts moving the value of the function down instead of up. Instead of having to do a bunch of fancy equations, let $q$ be the uniform function, with $\frac{1}{|X|}$ function-mass on each outcome. We're adding an equal amount to everything, and then dividing by some amount, so we get another uniform function. Which, to sum up to 1, must be unchanged. so $\frac{1}{|X|}$ is the critical value where the value of our $q$ start decreasing in value (if they're above it), or increasing (if they're below it).

Armed with our way of translating functions so that they do indeed fulfill our desired $\delta$ bound, let's look at the difference between: $h\left(q^{\frac{ϵ}{1+ϵ}}\right)$ and $h\left(q^{\prime \frac{ϵ}{1+ϵ}}\right)$

To begin with, by monotonicity,

$h\left(q^{\frac{ϵ}{1+ϵ}}\right)\le h(sup(q^{\frac{ϵ}{1+ϵ}},q^{\prime \frac{ϵ}{1+ϵ}}))$

then, by the fact that we have a finite Lipschitz constant $L$ for infradistributions when the associated function is bounded above 0. Let $ϵ$ be small enough that $q^{\prime \frac{ϵ}{1+ϵ}}\ge 0.5$, this makes the Lipschitz constant not depend on $\delta$.

$h(sup(q^{\frac{ϵ}{1+ϵ}},q^{\prime \frac{ϵ}{1+ϵ}}))\le h\left(q^{\prime \frac{ϵ}{1+ϵ}}\right)+Ld(sup(q^{\frac{ϵ}{1+ϵ}},q^{\prime \frac{ϵ}{1+ϵ}}),q^{\prime \frac{ϵ}{1+ϵ}})$

Now, let's try to evaluate the distance between those two functions, and how it scales with $ϵ$. The distance between functions is the maximum difference in values they assign to input points. Remember, if $q_i\le \frac{1}{|X|}$, $q_i^{\prime }$ has an equal or higher value, so the supremum is gonna mimic $q^{\prime }$ there. But for everything else, $q_i$ is actually going to be higher, and the supremum will mimic $q$ instead. So, we have:

$d(sup(q^{\frac{ϵ}{1+ϵ}},q^{\prime \frac{ϵ}{1+ϵ}}),q^{\prime \frac{ϵ}{1+ϵ}})={sup}_{i:q_i>\frac{1}{|X|}}|q_i^{\frac{ϵ}{1+ϵ}}-q_i^{\prime \frac{ϵ}{1+ϵ}}|$

$={sup}_{i:q_i>\frac{1}{|X|}}(q_i^{\frac{ϵ}{1+ϵ}}-{\left(\frac{q_i+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}\right)}^{\frac{ϵ}{1+ϵ}})\le {sup}_{y\in (|X{|}^{-1},1]}(y^{\frac{ϵ}{1+ϵ}}-{\left(\frac{y+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}\right)}^{\frac{ϵ}{1+ϵ}})$

Now, what we really want to study is how that above quantity varies with $ϵ$. Fortunately, $a^{\frac{x}{1-x}}$ with $a\in (0,1]$ is an analytic function at $x=0$, so we can take the Taylor expansion of both terms to see what's happening around 0 in terms of $ϵ$.

$={sup}_{y\in (|X{|}^{-1},1]}\left(\right(1+ϵ\ln \left(y\right)+O\left({ϵ}^2\right))-(1+ϵ\ln \left(\frac{y+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}\right)+O({ϵ}^2\left)\right))$

$={sup}_{y\in (|X{|}^{-1},1]}\left(ϵ\ln \right(y)-ϵ\ln \left(\frac{y+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}\right)+O({ϵ}^2\left)\right)$

$=ϵ{sup}_{y\in (|X{|}^{-1},1]}\left(\ln \left(\frac{y}{\frac{y+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}}\right)\right)+O\left({ϵ}^2\right)$

And, graphing the function in the sup with Desmos, with $y$ as the variable, and $\delta$ and $|X|$ as sliders, it's always maximized at 1. So then we get:

$=ϵ\ln \left(\frac{1}{\frac{1+\frac{\delta }{1-|X|\delta }}{1+|X|\frac{\delta }{1-|X|\delta }}}\right)+O\left({ϵ}^2\right)=ϵ\ln \left(\frac{1+|X|\frac{\delta }{1-|X|\delta }}{1+\frac{\delta }{1-|X|\delta }}\right)+O\left({ϵ}^2\right)$

I should note something about the big-O notation. From trying it out on some test values, the coefficient gets larger as $y$ decreases. So we don't have to worry about the coefficient in the big-O blowing up unboundedly, the lower bound on our possible $y$ tames it. Putting it all together, we get (for $ϵ$ sufficiently close to 0),

$h\left(q^{\frac{ϵ}{1+ϵ}}\right)\le h\left(q^{\prime \frac{ϵ}{1+ϵ}}\right)+Lϵ\ln \left(\frac{1+|X|\frac{\delta }{1-|X|\delta }}{1+\frac{\delta }{1-|X|\delta }}\right)+O\left({ϵ}^2\right)$

And our big-O doesn't depend on what q happens to be. Given any q at all, when we rescale it to get our $q^{\prime }$, our analysis always works until the big-O terms start manifesting, but they're biggest for the smallest $y$'s, and the lowest our $y$ gets is $|X{|}^{-1}$, so we can pick a sufficiently large constant to make everything work regardless of the $q$. Also, our big-O absorbed the Lipschitz constant. Now, because our big-O doesn't care about what $q$ is, we have:

${sup}_q\left(h\right(q^{\frac{ϵ}{1+ϵ}}\left)\right)\le {sup}_{q\ge \delta }\left(h\right(q^{\frac{ϵ}{1+ϵ}}\left)\right)+Lϵ\ln \left(\frac{1+|X|\frac{\delta }{1-|X|\delta }}{1+\frac{\delta }{1-|X|\delta }}\right)+O\left({ϵ}^2\right)$

Pretty much, since the last two terms of our bound are uniform in $q$, and all our $q^{\prime }\ge \delta$ since we designed it that way, we get the above bound. Now, we can take the $\ln$ to get

$\ln \left({sup}_q\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\left)\right)\le \ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right))+Lϵ\ln \left(\frac{1+|X|\frac{\delta }{1-|X|\delta }}{1+\frac{\delta }{1-|X|\delta }}\right)+O({ϵ}^2\left)\right)$

and multiply to get:

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_q\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\left)\right)\ge \frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right))+Lϵ\ln \left(\frac{1+|X|\frac{\delta }{1-|X|\delta }}{1+\frac{\delta }{1-|X|\delta }}\right)+O({ϵ}^2\left)\right)$

Thus, our upper bound on the sequence as a whole is:

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\left)\right)$

and our lower bound on the sequence as a whole for small $ϵ$ is

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right))+Lϵ\ln \left(\frac{1+|X|\frac{\delta }{1-|X|\delta }}{1+\frac{\delta }{1-|X|\delta }}\right)+O({ϵ}^2\left)\right)$

Let's show convergence for that top one, as it'll eventually let us crack the bottom one as well. Our task now is to show upper and lower bounds for the sequence (as $ϵ\to 0$)

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\left)\right)$

Augmented with our nice bounds on how low $q$ can be, let's see if we can find some other function that's close to $q^{\frac{ϵ}{1+ϵ}}$ for $ϵ$ near 0, so we can swap out for that.

Well... if we take the Taylor expansion of $y^{\frac{ϵ}{1+ϵ}}$ w.r.t. $ϵ$ around 0, then we get $1+ϵ\ln \left(y\right)+O\left({ϵ}^2\right)$. So what if we swapped out $q^{\frac{ϵ}{1+ϵ}}$ for $1+ϵ\ln \left(q\right)$? The second part of that is negative, so it's below 1. And, since, for all $i$, $q_i\ge \delta$, once we get to $ϵ<\frac{1}{-\ln \left(\delta \right)}$, it's guaranteed to never undershoot 0 and be a legit function. Let's assess the difference between the two functions:

${sup}_i|q_i^{\frac{ϵ}{1+ϵ}}-(1+ϵ\ln (q_i\left)\right)|$

$={sup}_i|(1+ϵ\ln (q\left)\right)+O\left({ϵ}^2\right)-(1+ϵ\ln (q\left)\right)|=O\left({ϵ}^2\right)$

Again, the reason this works is, because we have a uniform lower bound on our $q_i$, they're all in $[\delta ,1]$, which bounds how bad the big-O constants can be, so this difference is uniform across our $q$. The difference between $q^{\frac{ϵ}{1+ϵ}}$ and $1+ϵ\ln \left(q\right)$ is $O\left({ϵ}^2\right)$. By Lipschitzness of $h$ on functions bounded above $0.5$ (which happens for low enough $ϵ$), we can transfer this difference outside the $h$, and the Lipschitz constant absorbs into the big-O, so:

$h\left(q^{\frac{ϵ}{1+ϵ}}\right)\le h(1+ϵ\ln (q\left)\right)+O\left({ϵ}^2\right)$

and then, transferring this to all $q$ (the big-O is uniform), we have:

${sup}_{q\ge \delta }\left(h\right(q^{\frac{ϵ}{1+ϵ}}\left)\right)\le {sup}_{q\ge \delta }\left(h\right(1+ϵ\ln \left(q\right)\left)\right)+O\left({ϵ}^2\right)$

and so, we get a lower bound where

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\left)\right)\ge \frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h(1+ϵ\ln (q\left)\right))+O({ϵ}^2\left)\right)$

Now, for the upper bound. This is easy, because by graphing, we can see that $1+ϵ\ln \left(q_i\right)\le q_i^{\frac{ϵ}{1+ϵ}}$ is always true if $ϵ\ge 0$ and $q_i\in [\delta ,1]$. Thus, by monotonicity,

$h\left(q^{\frac{ϵ}{1+ϵ}}\right)\ge h(1+ϵ\ln (q\left)\right)$

and, again, transferring this to all $q$, and then taking the $\ln$ and multiplying by our negative constant, we have:

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h\left(q^{\frac{ϵ}{1+ϵ}}\right)\left)\right)\le \frac{1+ϵ}{-ϵ}\ln \left({sup}_{q\ge \delta }\right(h(1+ϵ\ln (q\left)\right)\left)\right)$

Alright, we've got some bounds. We can keep poking more at reexpressing the bounds, but remember that since we still haven't shown that anything in particular converges, we're gonna keep getting bounds on our bounds on our bounds. At some point we need to ground out and get that some actual limit exists, and use that as a tool to solve all the rest of the bounds. 

Let's look at $h(1+ϵ\ln (q\left)\right)$ for inspiration. We can notice something interesting. For a particular $q$, we can perfectly reexpress this as $h(1-ϵf)$, where $f_i=-\ln \left(q_i\right)$. Our analogue of $q_i$ summing to 1 is ${\sum }_i{e}^{-f_i}=1$, and our analogue of $q$ being in $[\delta ,1{]}^{|X|}$ is that $f\in [0,-\ln (\delta ){]}^{|X|}$. In fact, all $f$ of this form correspond to a $q$, and vice-versa, just have $q=e^{-f}$. So, using ${sup}_{f\le -\ln \left(\delta \right)}$ as a shorthand for "we're selecting among $f$ that fulfill these properties as that's isomorphic to selecting a $q$", we would be able to go:

${sup}_{q\ge \delta }\left(h\right(1+ϵ\ln \left(q\right)\left)\right)={sup}_{f\le -\ln \left(\delta \right)}\left(h\right(1-ϵf\left)\right)$

Now, with this reexpression, fixing a particular $f^{\ast }$, are we able to solve the following equation?

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left(h\right(1-ϵf^{\ast }\left)\right)$

We're just slicing off the smallest nontrivial bit of the problem we can to hopefully get one of these damn bounds to converge to something. And it turns out, we actually can solve this one! The teensy little problem is that L'hopital's rule only applies to differentiable functions, and this... isn't exactly differentiable. So we have to dig deep into the guts of the proof behind L'hopital's rule in order to show that applying the L'hopital procedure pretty much solves this, even though our function isn't differentiable (remember, h is Lipschitz, not differentiable). Once we've solved this, we'll be able to start chaining backwards and solve everything.

Our first order of business is to show that $(1+ϵ)\ln \left(h\right(1-ϵf^{\ast }\left)\right)$, as a function of $ϵ$ is differentiable almost everywhere, concave, and monotonically decreasing. First, for concavity of $h(1-ϵf^{\ast })$, observe that:

$h(1-(pϵ+(1-p){ϵ}^{\prime }\left)f^{\ast }\right)=h(p-pϵf^{\ast }+(1-p)-(1-p\left){ϵ}^{\prime }{f}^{\ast }\right)\ge ph(1-ϵf^{\ast })+(1-p)h(1-{ϵ}^{\prime }{f}^{\ast })$

And, as $ϵ$ gets bigger, by monotonicity of $h$, $h(1-ϵf^{\ast })$ gets smaller. So, $h(1-ϵf^{\ast })$ is monotonically decreasing, and concave. Now, from math stackexchange, the composition of a concave function with a monotonically increasing concave function ($\ln$ in this case) is concave. Thus, $\ln \left(h\right(1-ϵf^{\ast }\left)\right)$ is monotonically decreasing in $ϵ$, and concave. It's 0 when $ϵ=0$, and slopes down into the negative.

Now, any line from a point on the graph of $\ln \left(h\right(1-ϵf^{\ast }\left)\right)$ to another point on it must have a very important property. It slopes down, since this function is monotonically decreasing in $ϵ$. So, given some $ϵ$ and ${ϵ}^{\prime }$, there's a linear function $a-bx$ with $a,b\ge 0$ where, regardless of $p$,

$a-b(pϵ+(1-p\left){ϵ}^{\prime }\right)=p\ln \left(h\right(1-ϵf^{\ast }\left)\right)+(1-p)\ln \left(h\right(1-{ϵ}^{\prime }{f}^{\ast }\left)\right)$

(this is just the line between two spots on our resulting concave composite function). And then we can go:

$\ln \left(h\right(1-(pϵ+(1-p\left){ϵ}^{\prime }\right)f^{\ast }\left)\right)\ge p\ln \left(h\right(1-ϵf^{\ast }\left)\right)+(1-p)\ln \left(h\right(1-{ϵ}^{\prime }{f}^{\ast }\left)\right)=a-b(pϵ+(1-p\left){ϵ}^{\prime }\right)$

The first inequality is by concavity for $\ln \left(h\right(1-ϵf^{\ast }\left)\right)$ in $ϵ$. Multiplying by $(1+pϵ+(1-p\left){ϵ}^{\prime }\right)$ produces:

$(1+pϵ+(1-p\left){ϵ}^{\prime }\right)\ln \left(h\right(1-(pϵ+(1-p\left){ϵ}^{\prime }\right)f^{\ast }\left)\right)\ge (1+pϵ+(1-p\left){ϵ}^{\prime }\right)(a-b(pϵ+(1-p){ϵ}^{\prime }\left)\right)$

and, then, I'm gonna explain this following inequality in a bit, where it came from.

$\ge p(1+ϵ)(a-bϵ)+(1-p)(1+{ϵ}^{\prime })(a-b{ϵ}^{\prime })$

$=p(1+ϵ)\ln \left(h\right(1-ϵf^{\ast }\left)\right)+(1-p)(1+{ϵ}^{\prime })\ln \left(h\right(1-{ϵ}^{\prime }{f}^{\ast }\left)\right)$

So, the equality makes sense, we just use the link between our line and ln of $h$, see above just under "net takeaway". The inequality... what the heck is that about? Well, $(1+x)(a-bx)$ is a parabola opening down, it's concave. So, plugging $pϵ+(1-p){ϵ}^{\prime }$ in for the $x$ produces a bigger value than $p(1+ϵ)(a-bϵ)+(1-p)(1+{ϵ}^{\prime })(a-b{ϵ}^{\prime })$.

Ok, so our equation as a whole is concave. It's also monotonically decreasing because (ln of $h$ of...) starts off at 0 and just heads down from there, and we're multiplying by bigger and bigger positive numbers as $ϵ$ increases.

And also, according to math stackexchange, concave functions can only be non-differentiable at countably many points! So,

$(1+ϵ)\ln \left(h\right(1-ϵf^{\ast }\left)\right)$

the above function is concave in $ϵ$, is monotonically decreasing in $ϵ$, and only has countably many non-differentiable points. So as not to rewrite this each time, we'll switch our variable from $ϵ$ to $x$, and abbreviate this as $\theta \left(x\right)$, unpacking as needed.

Now that we know our function is nice, we can turn to verifying the important core needed to apply L'hopital's rule to solve the limit. And once that's done, we can start chaining backwards to solve our whole problem.

The key part that works for our function in particular, though it may fail in the general case, is an analogue of Cauchy's mean value theorem. We're about to make a bit of a mathematical leap here. We're going to let the derivative be set-valued at nondifferentiable points. As a toy example, consider the function $|x|$. if $x\ne 0$, the derivative is well-defined. If $x=0$, then, even though we don't have a derivative here, there's a left-derivative and a right-derivative. So, we can consider the "derivative" at 0 for $|x|$ to be any number between -1 and 1. 

Now, the particular thing we need to go through the L'hopital proof for our specific case is: For $ϵ$ and $\tau \in (0,ϵ)$, there's a $\chi \in [\tau ,ϵ]$, and a possible choice of derivative (remember, we've got the set-valued derivatives now) where:

$\frac{\theta \left(ϵ\right)-\theta \left(\tau \right)}{(-ϵ)-(-\tau )}=\frac{\frac{d}{dx}\left(\theta \right(x\left)\right)\left(\chi \right)}{\frac{d}{dx}(-x)\left(\chi \right)}$

We can make the argument for this more intuitive by drawing the following picture.

Remember, our function \theta is concave, and note that that the left-hand-side of our equation is just going "what's the slope of the line" (with positive sign). From concavity, we can just translate said line up until it becomes a tangent line of our function, and that gets us our value of $\chi \in [\tau ,ϵ]$ where there's a possible derivative that induces the above equality (because $\frac{d}{dx}(-x)\left(\chi \right)=-1$ no matter what, and this also enforces the appropriate sign). So, we do indeed have our analogue of Cauchy's mean value theorem working here. Drawing a line segment between two points on the graph of $\theta \left(x\right)$, there's a point in between them where the possible derivative matches up with the slope of the line segment.

Now, define the following two functions:

$m\left(ϵ\right):={inf}_{\chi \in [0,ϵ]}\frac{\frac{d}{dx}\left(\theta \right(x\left)\right)\left(\chi \right)}{\frac{d}{dx}(-x)\left(\chi \right)}$

$M\left(ϵ\right):={sup}_{\chi \in [0,ϵ]}\frac{\frac{d}{dx}\left(\theta \right(x\left)\right)\left(\chi \right)}{\frac{d}{dx}(-x)\left(\chi \right)}$

The inf and sup is also taken over possible choices of derivative, by the way. And so, regardless of our choice of $ϵ$ and $\tau$, as long as $\tau <ϵ$, from our analogue of Cauchy's Mean Value Theorem,

$m\left(ϵ\right)\le \frac{\theta \left(ϵ\right)-\theta \left(\tau \right)}{(-ϵ)-(-\tau )}\le M\left(ϵ\right)$

Now that we've established this, lock $ϵ$ in place and let $\tau$ limit to 0.

${lim}_{\tau \to 0}\frac{\theta \left(ϵ\right)-\theta \left(\tau \right)}{(-ϵ)-(-\tau )}$

$={lim}_{\tau \to 0}\frac{(1+ϵ)\ln \left(h\right(1-ϵf^{\ast }\left)\right)-(1+\tau )\ln \left(h\right(1-\tau f^{\ast }\left)\right)}{(-ϵ)-(-\tau )}$

$={lim}_{\tau \to 0}\frac{(1+ϵ)\ln \left(h\right(1-ϵf^{\ast }\left)\right)}{(-ϵ)-(-\tau )}-{lim}_{\tau \to 0}\frac{(1+\tau )\ln \left(h\right(1-\tau f^{\ast }\left)\right)}{(-ϵ)-(-\tau )}$

$=\frac{1+ϵ}{-ϵ}\ln \left(h\right(1-ϵf^{\ast }\left)\right)$

That last equality was because $1-\tau f^{\ast }$ approaches 1, so by normalization for $h$, $h(1-\tau f^{\ast })$ approaches 1, so $\ln \left(h\right(1-\tau f^{\ast }\left)\right)$ approaches 0, and multiplying by $(1+\tau )$ dosn't change that, and the bottom doesn't limit to 0.

So, we have our result that, for all our relevant $ϵ$,

$m\left(ϵ\right)\le \frac{1+ϵ}{-ϵ}\ln \left(h\right(1-ϵf^{\ast }\left)\right)\le M\left(ϵ\right)$

And now we'll use the squeeze theorem. So, what's the limit as $ϵ$ heads to 0 for $m\left(ϵ\right)$ and $M\left(ϵ\right)$?

Well, the former is:

${lim}_{ϵ\to 0}m\left(ϵ\right)={lim}_{ϵ\to 0}{inf}_{\chi \in [0,ϵ]}\frac{\frac{d}{dx}\left(\theta \right(x\left)\right)\left(\chi \right)}{\frac{d}{dx}(-x)\left(\chi \right)}={lim}_{ϵ\to 0}{inf}_{\chi \in [0,ϵ]}\frac{\frac{d}{dx}\left(\right(1+x\left)\ln \right(h(1-x{f}^{\ast })\left)\right)\left(\chi \right)}{-1}$

$={lim}_{ϵ\to 0}{inf}_{\chi \in [0,ϵ]}(-\left(\ln \right(h(1-x{f}^{\ast }))+\frac{1+x}{h(1-x{f}^{\ast })}\frac{d}{dx}h(1-x{f}^{\ast }\left)\right)(\chi \left)\right)$

$={lim}_{ϵ\to 0}{inf}_{\chi \in [0,ϵ]}(-\left(\ln \right(h(1-\chi f^{\ast }))+\frac{1+\chi }{h(1-\chi f^{\ast })}\frac{d}{dx}(h(1-x{f}^{\ast })\left)\right(\chi \left)\right))$

and as $ϵ$ gets incredibly close to 0 (forcing $\chi$ to do so as well), the $\ln \left(h\right(1-\chi f^{\ast }\left)\right)$ turns into 0, and the $\frac{1+\chi }{h(1-\chi f^{\ast })}$ turns into 1. So, this produces:

$={lim}_{ϵ\to 0}{inf}_{\chi \in [0,ϵ]}(-\frac{d}{dx}(h(1-x{f}^{\ast })\left)\right(\chi \left)\right)$

And now, since $h(1-x{f}^{\ast })$ is concave in $x$ and monotonically decreasing... Well, no matter which slope we choose for the nondifferentiable points, the shallowest possible slope is at 0. The slope is gonna be negative. Multiplying it by a negative means that we're trying to minimize a positive number. So, we want the shallowest slope we can possibly get, which would mean plugging $\chi =0$ in. Bam, no dependence on $ϵ$ anymore.

$=-\frac{d}{dx}\left(h\right(1-x{f}^{\ast }\left)\right)\left(0\right)$

$=-{lim}_{\tau \to 0}\frac{h(1+\tau (-f^{\ast }\left)\right)-h\left(1\right)}{\tau }$

And, consulting the Wikipedia page for the Gateux Derivative, this is the Gateaux Derivative of $h$ at 1 in the direction of $-f^{\ast }$!

So, we finally have solved one of our limits, it's:$-dh(1;-f^{\ast })$. Now, what about $M\left(ϵ\right)$? Well, a similar analysis applies.

${lim}_{ϵ\to 0}M\left(ϵ\right)={lim}_{ϵ\to 0}{sup}_{\chi \in [0,ϵ]}\frac{\frac{d}{dx}\left(\theta \right(x\left)\right)\left(\chi \right)}{\frac{d}{dx}(-x)\left(\chi \right)}={lim}_{ϵ\to 0}{sup}_{\chi \in [0,ϵ]}\frac{\frac{d}{dx}\left(\right(1+x\left)\ln \right(h(1-x{f}^{\ast })\left)\right)\left(\chi \right)}{-1}$

$={lim}_{ϵ\to 0}{sup}_{\chi \in [0,ϵ]}(-\left(\ln \right(h(1-x{f}^{\ast }))+\frac{1+x}{h(1-x{f}^{\ast })}\frac{d}{dx}h(1-x{f}^{\ast }\left)\right)(\chi \left)\right)$

$={lim}_{ϵ\to 0}{sup}_{\chi \in [0,ϵ]}(-\left(\ln \right(h(1-\chi f^{\ast }))+\frac{1+\chi }{h(1-\chi f^{\ast })}\frac{d}{dx}(h(1-x{f}^{\ast })\left)\right(\chi \left)\right))$

and as $ϵ$ gets incredibly close to 0 (forcing $\chi$ to do so as well), the $\ln \left(h\right(1-\chi f^{\ast }\left)\right)$ turns into 0, and the $\frac{1+\chi }{h(1-\chi f^{\ast })}$ turns into 1. So, this produces:

$={lim}_{ϵ\to 0}{sup}_{\chi \in [0,ϵ]}(-\frac{d}{dx}(h(1-x{f}^{\ast })\left)\right(\chi \left)\right)$

And now, since $h(1-x{f}^{\ast })$ is concave in $x$ and monotonically decreasing... Well, the slope is shallowest at 0, and steepest at $ϵ$ itself. The slope is gonna be negative. Multiplying it by a negative means that we're trying to maximize a positive number. So, we want the steepest slope we can possibly get, which would mean plugging $\chi =ϵ$ in. So, we have:

$={lim}_{ϵ\to 0}sup(-\frac{d}{dx}(h(1-x{f}^{\ast })\left)\right(ϵ\left)\right)$

Now, the sup in this case is just sup over derivatives, we know which $\chi$ to put in. In particular, since $h(1-x{f}^{\ast })$ is concave in $x$ and monotonically decreasing, the steepest possible derivative we could have is the derivative where the nearby point approaches from higher $ϵ$.

Now, remember, $h(1-x{f}^{\ast })$ is a concave function and monotonically decreasing (in $x$). If we graphed the derivative (in $x$), it'd be monotonically decreasing and always at 0 or less. There's discontinuities in the graph of the derivative, the graph of the derivative would look like a staircase going down. But remember, there are only countably many discontinuities. We're taking the lowest value of the derivative (read: furthest away from 0), so we can turn it into a (discontinuous) function where, at the "stairstep" discontinuity points, we stay at the bottom of the stairstep. And so, the question is, "if we stick to the bottom of the step at points where there's a step jump, and travel towards 0 ie $ϵ$ goes to 0, do we have a limit?" Well, the value of that "staircase" derivative function is monotonically increasing (going from well below 0 to less below 0) as $ϵ$ goes to 0, and it's got an upper bound of 0, so yeah, there's a limit.

But, in order to identify the limit as $ϵ$ goes to 0 of the lowest possible derivative with just "the derivative at 0", we've got a bit of a problem. There's some free choice in which derivative value to assign at discontinuity points. Given free choice in the matter, do all those choices lead to hitting the same value at 0? Well, let's say we took the upper value of the staircase. Given any $ϵ$ where there's a discontinuity, there's a smaller $\tau \in [\frac{ϵ}{2},ϵ]$ where the derivative is well-defined (because there's only countably many points where there's a discontinuity), which attains a value closer to 0 than the closest-to-0 value on the stairstep, because our derivative function stairstep is monotonically increasing as $ϵ$ ticks down. So, even the "take the upper value for each stairstep discontinuity" function can have any particular value exceeded by a sufficiently low value for $ϵ$ when we take the lower value for the stairstep discontinuities. So, it has the same limiting value. Which, again, is the derivative at 0. So, we get:

$={lim}_{ϵ\to 0}sup(-\frac{d}{dx}(h(1-x{f}^{\ast })\left)\right(ϵ\left)\right)=-\frac{d}{dx}\left(h\right(1-x{f}^{\ast }\left)\right)\left(0\right)=-dh(1;-f^{\ast })$

Alright! We got the same value for an upper bound as a lower bound! Awesome! Now we can get started solving some limits, finally! From here on out, we'll do a lot of backing up to stuff we've already shown. First, we've already shown that:

$m\left(ϵ\right)\le \frac{1+ϵ}{-ϵ}\ln \left(h\right(1-ϵf^{\ast }\left)\right)\le M\left(ϵ\right)$

and, in the $ϵ\to 0$ limit, both the left-hand side and right-hand-side have the same limit. Namely, $-dh(1;-f^{\ast })$. The Gateaux derivative of $h$ at 1 in the direction of $-f^{\ast }$. So,

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left(h\right(1-ϵf^{\ast }\left)\right)=-dh(1;-f^{\ast })$

Alright, we've got our first toehold where we were actually able to solve a damn limit. Let's back up a bit and try somthing more ambitious. Let's attempt to solve

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)$

Again, we're going to impose upper and lower bounds. The tricky part in this case is that the previous limit was just for a particular function. Once we start taking the supremum over a bunch of different functions, we can't necessarily guarantee that they all limit at the same rate. But, let's try to reduce it to the case of one particular function.

To begin with, regardless of $ϵ$ and $f$, $h(1-ϵf)\le 1+ϵdh(1;-f)$. Why is this the case? Well, the former is a concave function, monotonically decreasing, that starts at 1 where $ϵ=0$. And the latter is the tangent line to the former function at $ϵ=0$. So, the tangent line lies above the function. Thus, we can go:

$h(1-ϵf)\le 1+ϵdh(1;-f)$

${sup}_{f\le -\ln \left(\delta \right)}\left(h\right(1-ϵf\left)\right)\le {sup}_{f\le -\ln \left(\delta \right)}(1+ϵdh(1;-f\left)\right)$

$\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)\le \ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(1+ϵdh(1;-f)\left)\right)$

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)\ge \frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(1+ϵdh(1;-f)\left)\right)$

and then, since all the Gateux derivatives are 0 or less, we can move the sup inside (changes nothing), and get:

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)\ge \frac{1+ϵ}{-ϵ}\ln (1+ϵ{sup}_{f\le -\ln \left(\delta \right)}(dh(1;-f)\left)\right)$

Ok, cool, we've got a lower bound. Now, what about an upper bound? Well, pick an $f^{\ast }$ where $dh(1;-f^{\ast })$ is extremely extremely close to ${sup}_{f\le -\ln \left(\delta \right)}\left(dh\right(1;-f\left)\right)$. We can get:

${sup}_{f\le -\ln \left(\delta \right)}\left(h\right(1-ϵf\left)\right)\ge h(1-ϵf^{\ast })$

$\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)\ge \ln \left(h\right(1-ϵf^{\ast }\left)\right)$

$\frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)\le \frac{1+ϵ}{-ϵ}\ln \left(h\right(1+ϵf^{\ast }\left)\right)$

And bam, we have an upper bound.

Now, let's take the $ϵ\to 0$ limits of our lower bound and upper bound. Our upper bound limit is:

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left(h\right(1+ϵf^{\ast }\left)\right)=-dh(1;-f^{\ast })$

Our lower bound limit is:

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln (1+ϵ{sup}_{f\le -\ln \left(\delta \right)}(dh(1;-f)\left)\right)$

But fortunately, that supremum doesn't change with $ϵ$! It's just a constant. So the function inside the $\ln$ is continuous and differentiable, and so is everything, so we can solve this with vanilla L'hopital's rule.

$={lim}_{ϵ\to 0}\frac{\frac{d}{dx}\left(\right(1+x\left)\ln \right(1+x{sup}_{f\le -\ln \left(\delta \right)}\left(dh\right(1;-f\left)\right)\left)\right)\left(ϵ\right)}{\frac{d}{dx}(-x)\left(ϵ\right)}$

$={lim}_{ϵ\to 0}\frac{\ln (1+ϵ{sup}_{f\le -\ln \left(\delta \right)}(dh(1;-f)\left)\right)+\frac{1+ϵ}{1+ϵ{sup}_{f\le -\ln \left(\delta \right)}\left(dh\right(1;-f\left)\right)}{sup}_{f\le -\ln \left(\delta \right)}\left(dh\right(1;-f\left)\right)}{-1}$

$=-\left(\ln \right(1+0)+\frac{1+0}{1+0}{sup}_{f\le -\ln \left(\delta \right)}(dh(1;-f)\left)\right)$

$=-{sup}_{f\le -\ln \left(\delta \right)}\left(dh\right(1;-f\left)\right)$

Sadly, these two bounds are different. We only have that

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)\in [-{sup}_{f\le -\ln \left(\delta \right)}(dh(1;-f)),-dh(1;-f^{\ast }\left)\right]$

Or, heck, the limit might not even exist! But all limit points must be in that interval. But... this argument worked independently of our choice of $f^{\ast }$. So, we can select it so the derivative is as close as we want to the supremum derivative! No matter how tight the interval is, we can always back up, pick a different $f^{\ast }$, and make the interval smaller! Thus, the limit must exist, and we have:

${lim}_{ϵ\to 0}\frac{1+ϵ}{-ϵ}\ln \left({sup}_{f\le -\ln \left(\delta \right)}\right(h(1-ϵf)\left)\right)=-{sup}_{f\le -\ln \left(\delta \right)}\left(dh\right(1;-f\left)\right)$