It is a known fact that the adjoint representation of $SU(N)$ is a proper subgroup of $SO(N^2-1)$.


I would like to know how a generic $(N^2-1)\times (N^2-1)$ special ($det =1$), orthogonal matrix $O$ decomposes into an element of $SU(N)$ in the adjoint representation, $[Ad\_U]$, times something else,
$$
 O=M\cdot Ad\_U
$$


Clearly $M$ will be an element from a group, but I cannot figure out which. My background in group theory is rather limited.



$SO(N^2-1)$ and the adjoint representation of $SU(N)$
Actually, it does not look like that. Take the case $N=3$. The representation of $\mathrm{SU}(3)$ on ${\frak{so}}(8)$ breaks up into the $8$-dimensional subspace ${\frak{su}}(3)$ and an irreducible $20$-dimensional space that is isomorphic to the complex homogeneous cubic polynomials in $3$ variables as an $\mathrm{SU}(3)$ representation. That subspace is not a Lie subalgebra of ${\frak{so}}(8)$, so there is no $20$-dimensional Lie subgroup of $\mathrm{SO}(8)$ corresponding to it. 


Maybe what you really want is something like an explicit $20$-dimensional manifold $M$ and an explicit map $\mathrm{SO}(8)\to M$ whose fibers are the left cosets of the subgroup $\mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)$? That would be more reasonable.


Here is another point to think about: There cannot be any closed subset $S\subset\mathrm{SO}(8)$ such that the map $S\times \mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)\to \mathrm{SO}(8)$ induced by multiplication is $1$-to-$1$ and onto. The reason is that such a map would be continuous (using the product topology on the domain), and a continuous bijection from a compact space to a Hausdorff space is a homeomorphism. However this would imply that 
$$
\pi\_1\bigl(\mathrm{SO}(8)\bigr) \simeq \pi\_1\bigl(S\bigr)\times \pi\_1\bigl(\mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)\bigr),
$$
and this is impossible because $\pi\_1\bigl(\mathrm{SO}(8)\bigr)\simeq \mathbb{Z}\_2$ while $\pi\_1\bigl(\mathrm{Ad}\bigl(\mathrm{SU}(3)\bigr)\bigr)\simeq\mathbb{Z}\_3$. Thus, there cannot be any reasonable 'factorization' of the kind you are seeking, even if you don't require that the first factor lie in a Lie subgroup. (This argument generalizes immediately to cover all the cases when $N\ge 3$. Obviously, for $N=2$, you can take $S = \{I\_2\}$.)


(Maybe, you were imagining something like the so-called $QR$-factorization of each element of $\mathrm{SL}(n,\mathbb{R})$ into the product of a positive definite symmetric matrix and an orthogonal matrix. Unfortunately, nothing like that works in this case.)


