We have the unit sphere $S^2$ in $\mathbb{R}^3$ and two points, $X$ and $Y$ on the surface of the sphere. Then, a function is defined for any point $P$ inside of the unit ball as:


$$f(P) = R\,d(P, XY)$$


where $d(P,XY)$ is the Euclidean distance from the line containing $XY$, and $R$ is the radius of the small circle of the sphere in the plane containing $XYP$. One can notice that for a point $P'$ on the surface of the sphere, the function corresponds to $f(P') = \frac{1}{2} |P'X| |P'Y|$.


I can obtain values of the function numerically by calculating $R$ as the distance of $XYP$ from the origin, but it does not seem to simplify to any useful closed-form expression. Suppressing any one of the dimensions, a plot of the function (for a choice of $X,Y$) looks like this:


[Plot](https://i.stack.imgur.com/S0Fra.png)


where the minimum is reached for the line $XY$, the function is constant on lines parallel to $XY$, and it is visibly convex.


How to show this convexity? The distance is obviously a convex function, but $R$ is not convex.



Showing convexity of a function in the unit ball
Let $A$ be a midpoint of $XY$. We may restrict ourselves to the plane $\alpha$, which is perpendicular bisector to $XY$. Choose Cartesian and polar coordiantes in $\alpha$ centered in $A$, so that $O=(a,0)$ (Cartesian) and $P=(x,y)$ (Cartesian)$=(r,\varphi)$ (polar). Then $d(P,XY)=d(P,A)=r$, $R=\sqrt{1-d(O,PA)^2}=\sqrt{1-a^2\sin^2\varphi}$, thus $$R\cdot d(P,XY)=\sqrt{r^2-a^2r^2\sin^2\varphi}=\sqrt{x^2+(1-a^2)y^2},$$
this is a norm on the plane $\alpha$, it is convex.


