Is there a closest free-return trajectory to a black hole? The scenario I am imagining is a test particle falling from infinity with an initial off-center velocity. In Newtonian mechanics the path would be a hyperbola. But I assume that, near a black hole, that is no longer a good approximation.
I have read about the innermost stable circular orbit, and it seems like there might be an innermost free-return distance, where any object that falls within that distance will need to use rockets to get back out. I may have even come across this idea, but I am not finding it with a Google search.
I have an argument that there must be such a distance. Imagine that you could fly past a supermassive black hole just half a meter above the event horizon and continue on to deep space without using rockets. Tidal forces aren't so great at the event horizon for a SMBH, so there should be no reason you can't extend your arm just over half a meter down and not pull back a bloody stump. But you can't retrieve things from below the event horizon. This paradox is resolved if, in fact, the only way to escape from that close to the EH is to exert massive force using rockets (or some other method). Even though you are not yet inside the EH, your free fall path will take all of you there, and the force to prevent that would pulp you anyway. I think it is related enough to the main question to ask: Is this analysis correct?
The trajectory of a ballistic body, whether in Newtonian or Relativistic physics depends on the initial energy and angular momentum.
The difference is that in Newtonian physics, if the mass is compact enough, the infalling object (if given an initial kinetic energy) will never hit the central object, unless it is a direct hit, and will scatter off to infinity.
This can also happen in GR, but only if the specific angular momentum exceeds a threshold that increases with the initial energy of the particle, but can never be less $4GM/c$, and only as small as this for an object that falls from infinity with no initial kinetic energy. If the specific angular momentum is less than this then the infalling object will always end up in the black hole (for a freefalling trajectory).
For the limiting case, the closest approach to the black hole is $2r_s$, which is the local maximum of the effective potential.
However, the body can get closer if it has more angular momentum and more initial energy. If we allow the angular momentum and energy to get arbitrarily large then the body can get to $r=1.5 r_s$.
So I think this is the answer you are looking for. This is the value for a Schwarzschild black hole. For a spinning black hole, things are more complicated.
EDIT: Some details
The effective potential in the Schwarzschild metric is given by
$$ \frac{V_{\rm eff}}{mc^2} = -\frac{r_s}{2r} + \frac{L^2}{2m^2 r^2 c^2} - \frac{r_sL^2}{2m^2r^3c^2}\ , $$
where $r_s$ is the Schwarzschild radius and $L/m$ is the specific angular momentum of the free body.
The radial equation of motion in terms of the proper time $\tau$ of the body is
$$\frac{1}{2c^2}\left( \frac{dr}{d\tau}\right)^2 = \frac{1}{2}\left( \frac{E^2}{m^2 c^4} -1 \right) - \frac{V_{\rm eff}}{mc^2} \ .$$
To find the local maximum of the effective potential, differentiate with respect to $r$ and equate to zero. This gives you that the specific angular momentum at the maximum is
$$ \frac{L^2}{m^2} = \frac{c^2 r_s r^2}{2r - 3r_s}\ . $$
Allowing the energy of the body to be $E = \gamma mc^2$ (where $\gamma$ is the Lorentz factor) one can set $dr/d\tau = 0$ at the scattering minimum radius and then subsititute in the value for $L^2/m^2$ at the maximum of the effective potential. The rationale here is that the body will get to its closest point when it has an energy corresponding to the maximum in the effective potential. If it has any more energy than this it will just plunge into the black hole. Any less and it will scatter at a larger radius.
This condition gives you a quadratic equation in $r$ which can be solved in terms of $\gamma$. If $\gamma=1$ then the quadratic becomes a linear equation with $r=2r_s$. This corresponds to the case of a particle falling in from rest at infinity and meeting a local maximum of the potential of $V_{\rm eff}=0$. The body then scatters off to (just) infinity (i.e. the marginally bound case). This is illustrated below, where the horizontal purple line is a line corresponding to an energy $E/mc^2=1$. The peak at B is at $r=2r_s$ where $V_{\rm eff}=0$ (I've also shown the equivalent Newtonian potential for contrast).

If you allow $\gamma$ to be larger then the the viable solutions for the minimum $r$ lie in the bounds $1.5 r_s \leq r \leq 2r_s$. If $\gamma \rightarrow \infty$ then the scattering minimum radius $r \rightarrow 1.5r_s$, which also requires $L/m \rightarrow \infty$. The plot below shows the potential for $L/m = 10GM/c$, with a purple line representing a body with $E/mc^2=2$ which would scatter to infinity. The peak at B is just above $r=1.5r_s$.

If you want to play with this (my) Geogebra widget, you can find it at https://www.geogebra.org/m/u7y6huer