Let $A$ and $B$ be two matrices with $\det(A)=\det(B)=1$. Does it follow that


$\sqrt{\mathrm{tr}(A^TB^TBA-I)}\le\sqrt{\mathrm{tr}(A^TA-I)}+\sqrt{\mathrm{tr}(B^TB-I)}$


I suspect that this can be shown using the singular value decomposition, but I've not been able to write a proof yet. Scaling arguments suggest that the determinant condition is really needed.


If needed, it may be assumed that all singular values of both $A$ and $B$ are positive.



Trace inequality for matrices with determinant 1
As per fedja's comment, the version of the inequality written in the question does not hold. A version that does hold is given below.


$\newcommand{\trace}{\mathrm{tr}}$
 Define $\trace\_n(X) := \frac{1}{n}\trace(X)$ be the normalized trace. Let $U, V$ be unitary matrices. Then, the following inequality can be shown to hold
\begin{equation\*}
 \sqrt{1-|\trace\_n(UV)|^2} \le \sqrt{1-|\trace\_n(U)|^2} + \sqrt{1-|\trace\_n(V)|^2},
\end{equation\*}
with equality if and only if $U$ or $V$ is a unitary scalar matrix.


For an elementary proof, please see the paper: ["A trace inequality for unitary matrices"](http://www.jstor.org/discover/10.2307/2974909?uid=3739560&uid=2&uid=4&uid=3739256&sid=21101262726543) by B.-Y. Wang and F. Zhang (AMM, 101(5), 1994).


