Subgame Perfect Equilibrium in Baye, Shin (1999) The following is taken from Baye, Shin (1999)
Consider a contest over a prize valued at 1 with symmetric players $1$ and $2$ who exert a level of effort $x_1$ and $x_2$ respectively. Effort cannot exceed $2/3$. Profit ($\pi$) of player 1 is
$$\pi_1=\frac{x_1-\frac{x_1x_2}{2}}{x_1+x_2-x_1x_2}-x_1$$
The simultaneous-move equilibrium is (denoted with superscript $*$)
$$x_1^*=x_2^*=1-\frac{\sqrt{2}}{2}$$
The profit for each is
$$\pi_1^*(x_1^*,x_2^*)=\pi_2^*(x_1^*,x_2^*)=\frac{\sqrt{2}-1}{2}$$
And we have that the best response of player $2$
$$R_2(x_1)=\frac{2x_1-\sqrt{(-2x_1^2+4x_1)}}{2(x_1-1)}$$
However, suppose that player $1$ 'moves' before player $2$. Then, player $1$ would deviate (upwards) from $x_1^*$, presumably, because player $2$ would reduce it's level of effort, and it would receive more of the prize. How does one show this?

In the article, Baye and Shin consider the new Stackelberg equilibrium $\pi_1^s$ where they show that $\pi_1^s(x_1^*+\epsilon)&gt;0$ for some $\epsilon&gt;0$. Specifically
$$\pi_1^s(x_1^*+\epsilon)-\pi_1^*(x_1^*+x_2^*)=\frac{\sqrt{1+2\epsilon\sqrt{2}-2\epsilon^2}-1-\epsilon\sqrt{2}+2\epsilon^2}{\sqrt{2}-2\epsilon}&gt;0$$

I've tried replicating their result by seeing how $x_2$ changes with $x_1+\epsilon$ and substituting $x_1+\epsilon$ and the new value of $x_2^s$ into $1$'s profit function, but I fail to reach their result.
True to the saying that a picture is worth a thousand words, let's plot the profit function under the Stackelberg game and see what we're dealing with.

You correctly observed (or copied from the Baye, Shin paper) that the reaction function of player 2 is:

$$
R(x_1)=\frac{2\mathit{x_1}-\sqrt{2}\,\sqrt{2\mathit{x_1}-{{\mathit{x_1}}^{2}}}}{2\mathit{x_1}-2}\
$$

Inserting this reaction function in the profit function of player 1 gives us a complicated expression:

$$
\pi(x_1)=-\frac{{{2}^{\frac{3}{2}}}\,\sqrt{2-\mathit{x_1}}\,{{\mathit{x_1}}^{2}}-2{{\mathit{x_1}}^{\frac{3}{2}}}-3\sqrt{2}\,\sqrt{2-\mathit{x_1}}\,\mathit{x_1}+4\sqrt{\mathit{x_1}}}{{{2}^{\frac{3}{2}}}\,\sqrt{2-\mathit{x_1}}\,\left( \mathit{x_1}-1\right) }
$$

(Perhaps more advanced symbolic math programs can further simplify this, mine won't). In any case this is  a function of one variable $\mathit{x_1}$, and can be plotted. For completeness I have inserted the original equilibrium as well.



As you can see the original NE is no longer a local maximum, and player 1 can increase its profits by increasing their effort.