I would like to perform the integration ($u \in [0,1]$),
\begin{equation}
\int\_{a=-1}^1 \int\_{b=-\sqrt{1-a^2}}^{\sqrt{1-a^2}} u^2 \sqrt{-a^2 u^2-b^2+1} \tan ^{-1}\left(\frac{\left| a\right|
 }{\sqrt{-a^2-b^2+1}}\right) db da .
\end{equation}


I also have a companion problem
\begin{equation}
\int\_{a=-1}^1 \int\_{b=-\sqrt{1-a^2}}^{\sqrt{1-a^2}} u \sqrt{-a^2-b^2+1} \tan ^{-1}\left(u \left| a\right| \sqrt{-\frac{1}{a^2
 u^2+b^2-1}}\right) db da .
\end{equation}


Solutions to these two problems would provide a resolution to the question 
"Compute a certain "separability probability" via a constrained 4D integration over $[-1,1]^4$", posed in <https://mathematica.stackexchange.com/questions/193337/compute-a-certain-separability-probability-via-a-constrained-4d-integration-ov>


Now, for the component functions of the first integrand, we have
\begin{equation}
\int\_{a=-1}^1 \int\_{b=-\sqrt{1-a^2}}^{\sqrt{1-a^2}} u^2 \sqrt{-a^2 u^2-b^2+1} db da = \frac{4}{3} u \left(\sqrt{1-u^2} u+\sin ^{-1}(u)\right),
\end{equation}
and 
\begin{equation}
\int\_{a=-1}^1 \int\_{b=-\sqrt{1-a^2}}^{\sqrt{1-a^2}} \tan ^{-1}\left(\frac{\left| a\right| }{\sqrt{-a^2-b^2+1}}\right) db da=\frac{1}{2} (\pi -2) \pi.
\end{equation}
Can some form of integration-by-parts--or other methodology--be performed? (I realize that more may certainly be needed in addition to the two above results.)


The much-viewed question <https://math.stackexchange.com/questions/1167346/integration-by-parts-for-a-double-integral> seems relevant here.



A pair of integrals involving square roots and inverse trigonometric functions over the unit disk
Further to my comment, one can set $y = \frac{z}{\sqrt{1-z^2}}$ to obtain $$\frac{8}{3} \, u^2 \int\_0^1 \! \sqrt{1 - u^2 z^2} \sin^{-1}(z) \, dz \;,$$ which *Mathematica* evaluates to
$$
\frac{2}{3} \, u \, \bigl(\operatorname{Li}\_2(-u) - \operatorname{Li}\_2(u) + \pi \sqrt{1 - u^2} \, u - (1 - u^2) \tanh^{-1}(u) - u + \pi \sin^{-1}(u)\bigr) \;,
$$
where $\operatorname{Li}\_2(x)$ is the dilogarithm.




---


A very similar derivation for your companion problem yields the final integral $$\frac{8}{3} \, u \int\_0^1 \! \sqrt{1 - z^2} \sin^{-1}(u z) \, dz$$ which *Mathematica* evaluates to $$-\frac{2}{3u} \, \bigl(u^2 \, \operatorname{Li}\_2(-u) - u^2 \, \operatorname{Li}\_2(u) - (1 - u^2) \tanh^{-1}(u) + u\bigr) \;.$$




---


For completeness, we can also do the third integral (from the comments) in exactly the same way, yielding $$\frac{8}{\pi} \, u^2 \int\_0^1 \! \sqrt{1 - u^2 z^2} \, dz$$ as the final integral, which evaluates to $$\frac{4}{\pi} \, u \, \bigl(\sqrt{1 - u^2} \, u + \sin^{-1}(u)\bigr) \;.$$




---


Bonus round: If $I\_1$, $I\_2$, $I\_3$ are the three results above, and $$A = \frac{6}{\pi^2} \, (I\_2 - I\_1) + I\_3 \;, \\ B = \frac{12 u^2 \left(u^6+9 u^4-9 u^2-12 \left(u^4+u^2\right) \log (u)-1\right)}{\left(u^2-1\right)^5} \;,$$ then we are also interested in the integral of $AB$ over $u \in [0,1]$. That seems quite impossible, but we are in luck: integrating twice by parts, differentiating each time the factor with dilogarithms, yields a simple integral in $u$ that can be evaluated. The result is $$\frac{256-168 \zeta (3)}{\pi ^2}-38+48 \log (2) \approx 0.747925 \;.$$


