I would like to know if for $A,B\in SO(3)$ the inequality
$$
\|AB-BA\|\_F\leq \|A-I\|\_F\|B-I\|\_F
$$
holds, where $\|\cdot\|\_F$ denotes the Frobenius norm and $I$ the identity matrix. Using the identity
$$
AB-BA=(A-I)(B-I)-(B-I)(A-I)
$$
one can show the inequality with a factor $2$. However the inequality seems to be true also without this factor. 


I encountered this problem when I tried to estimate the error of permuting two small rotations.



matrix inequality with orthogonal matrices
I confirm Peter's suggestion that the best constant is $\frac1{\sqrt2}$. For let $\omega$ be this best constant. Then the inequality amounts to writing
$$3-{\rm Tr}(A^tB^tAB)\le2\omega^2(3-{\rm Tr}\,A)(3-{\rm Tr}\,B).$$
Let me parametrize $A$ by the angle of rotation $\theta$ and the axis of rotation $u$, a unit vector. Likewise $B$ has an angle of rotation $\alpha$ and an axis $v$. The left-hand side above is $8\omega^2(1-c)(1-c')$, where $c=\cos\theta$ and $c'=\cos\alpha$.


Now $B^tAB$ is the rotation of same angle $\theta$ about $w=B^{-1}u$, and $A^t$ the rotation of angle $-\theta$ about $u$. It is an instructive calculus that
$${\rm Tr}(A^tB^tAB)=2c+c^2+(1-c)^2(w\cdot u)^2+2s^2w\cdot u.$$
Because of $w\cdot u\ge c'$, we find that $\omega$ is the best constant in
$$3-2c-c^2+(1-c)^2c'^2-2s^2c'\le8\omega^2(1-c)(1-c').$$
The left-hand side factorizes as $(1-c)(1-c')(3+c+c'-cc')$, and there remains the inequality
$$3+c+c'-cc'\le8\omega^2.$$
Therefore $\omega^2$ is the supremum of $\frac18(3+c+c'-cc')$ over $[-1,1]^2$. Because this quantity if affine in both $c$ and $c'$, the supremum is achieved at a vertex, and its value is $4$. Whence $\omega^2=\frac48=\frac12$.


