There is a strange product that takes two square roots of unit matrix, say $A$ and $B$, $A^2=I$, $B^2=I$ to a square root again, 
$$ A\star B=(A+B)^{-1}(A-B+2I), \qquad (A\star B)^2=I$$
Could anybody help me with identifying this structure? Where it comes from?


It was obtained from the Caley transform $C(A)=(1-A)^{-1}(1+A)$, $C(C(A))=A$, by expanding $C(C(A),C(B))$ and using $A^2=B^2=I$. Ones we imposed $A^2=B^2=I$ the inverse transform does not exists anymore, so we are at the singular point of the Caley transform. Somehow it looks like adding some points at infinity and extending the action on them.



A product on the square roots of unit matrix
Let me denote $E\_\pm(M)$ the eigenspaces of $M$ associated with the eigenvalues $\pm1$. Let me assume that the characteristic of the scalar field $k$ is not $2$. Then your assumption is that 
$$k^n=E\_+(A)\oplus E\_-(A)=E\_+(B)\oplus E\_-(B).$$
In addition, the assumption that $A+B$ is non-singular means $E\_+(A)\cap E\_-(B)=(0)$ and $E\_+(B)\cap E\_-(A)=(0)$, from which it follows that $\dim E\_+(A)=\dim E\_+(B)$.


Then one verifies easily
$$E\_+(A\star B)=E\_+(B),\qquad E\_-(A\star B)=E\_-(A).$$
Because 
 the dimensions of $E\_-(A)$ and $E\_+(B)$ sum up to $n$, one deduces 
$$k^n= E\_+(A\star B)\oplus E\_-(A\star B),$$
and therefore $(A\star B)^2= I\_n$.


