Let $F\colon\mathbb R^3\to\mathbb R$ be a compactly supported smooth function. I want to compute 
$$ \int\_{\mathbb R^n}\int\_{\mathbb R^n} F(\lVert x\rVert^2,\lVert y\rVert^2,\langle x,y\rangle)~\mathrm{d}x~\mathrm{d}y,$$
where $\mathrm{d}x$ is the Lebesgue measure in $\mathbb R^n$, as a three dimensional integral. Doing so requires seems to require computing the volume of certain sphere intersections and does not seem straight-forward. Is there a clever way of doing this? Any references or keywords appreciated.



Integral substitution involving the length and angle of two vectors
I'm not sure about a clever way, but the straightforward Calculus 3 type computation is not hard at all. Making the integration in $x$ outer and switching to polar coordinates in $x$, then switching in the inner integral with respect to $y$ to the cylindrical coordinates with the axis of the cylinder parallel to $x$, we get 
$$
\omega\_{n-1}\int\_0^\infty r^{n-1}dr\int\_{-\infty}^\infty dx\omega\_{n-2}\int\_0^\infty \rho^{n-2}d\rho F(r^2,xr,x^2+\rho^2)
\\
=\omega\_{n-1}\omega\_{n-2}\iiint r^{n-1}\rho^{n-2}F(x^2,xr,x^2+\rho^2) dr\,dx\,d\rho\,.
$$
Now, putting $X=r^2, Y=xr, Z=x^2+\rho^2$, we find $dX=2rdr, dY=xdr+rdx, dZ=2xdx+2\rho d\rho$, so $dXdYdZ=4r^2\rho\,dr\,dx\,d\rho$ whence the integral in question is
$$
\frac{\omega\_{n-1}\omega\_{n-2}}4\iiint (r\rho)^{n-3}F(X,Y,Z)dX\,dY\,dZ
$$
It remains to express $r\rho=(XZ-Y^2)^{1/2}$ to get the final answer
$$
\frac{\omega\_{n-1}\omega\_{n-2}}4\iiint (XZ-Y^2)^{\frac{n-3}2}F(X,Y,Z)dX\,dY\,dZ
$$
with the domain of integration $X,Z>0, Y^2\le XZ$


