Let $V$ be a finite dimensional vector space over $ \mathbb{R}$. Let
\begin{equation} \left\langle\:,\:\right\rangle:\mbox{End}(V)\otimes\mbox{End}(V)\rightarrow \mathbb{R}\end{equation}
denote the pairing given by $\left\langle A,B\right\rangle = \mbox{Tr}(AB)$. We know that this pairing is $\mbox{GL}(V)$-invariant, symmetric, non-degenerate, but not definite (i,e. neither positive definite nor negative definite).


Supposed that $\beta:V\otimes V \rightarrow \mathbb{R}$ is a non-degenerate symmetric pairing. Recall $\mathfrak{so}(V,\beta)\subset \mathfrak{gl}(V)$ is defined by 
$$\mathfrak{so}(V,\beta)=\left\{A\in\mathfrak{gl}(V)\:|\: \forall v,w\in V \: \beta(Av,w)+\beta(v,Aw)=0\right\}.$$
We know that $\mathfrak{so}(V,\beta)$ is a Lie subalgebra of $\mathfrak{gl}(V)$.


**The question:** Establish conditions on $\beta$ under which the restriction of the trace pairing (i,e. $\left\langle\:,\:\right\rangle$ restricted to $\mathfrak{so}(V,\beta)$) is positive (also conditions to be negative) definite.


**My attempt:** We know that given $\beta:V\times V\rightarrow \mathbb{R}$ a non-degenerate symmetric form, for all $A\in \mbox{End}(V)$ there exist a unique $B\in \mbox{End}(V)$ which satisfies $\beta(Av,w)=\beta(v,Bw)$ for all $v,w\in V$.


The unique $B$ as above will be denoted by $A^{\dagger}$ and called **the transpose of $A$** (with respect to $\beta$).


The assignment $A\rightarrow A^{\dagger}$ defines a linear map $(\cdot)^{\dagger}\mbox{End}(V)\rightarrow \mbox{End}(V)$ called *transposition* (with respect to $\beta$).


Therefore, we can consider 
$$\mathfrak{so}(V,\beta)=\left\{A\in\mathfrak{gl}(V)\:|\: A=-A^{\dagger} \: (\mbox{transpose with respect to }\beta)\right\}.$$
In this sense, we have


1. Restriction the trace pairing is positive definite if $\mbox{Tr}(A^{\dagger}A)<$ for all $A\in \mathfrak{so}(V,\beta)$ with $A\neq 0$.
2. Restriction the trace pairing is negative definite if $\mbox{Tr}(A^{\dagger}A)>$ for all $A\in \mathfrak{so}(V,\beta)$ with $A\neq 0$.


**My question:** Conditions 1 and 2 depend on $\beta$. Does the condition 1 and 2 answer the question? Can I characterize $A^{\dagger}$ in terms of $\beta$ of a better way? Are there more specific conditions on $\beta$ to answer the question?



Conditions on $\beta$ under which the trace pairing restricted to $\mathfrak{so}(V,\beta)$ is positive (negative) definite
Yes to all three questions. By diagonalisation of forms, WLOG, $\beta$ is diagonal with $\pm 1$ on the main diagonal. Now let us just compute (and you can get an explicit formula for your transposition as well).


If $\beta$ is positive or negative definite, $so(V,\beta)$ is the set of skew-symmetric matrices. The trace form is negative definite as easily observed.


If $\dim V =2$ and $\beta$ is indefinite, $so(V,\beta)$ is one-dimensional. The trace form is positive definite as easily observed.


If $\dim V >2$ and $\beta$ is indefinite, you look at a subalgebra of $so(V,\beta)$, of matrices that are zero outside a diagonal $3\times 3$-minor, which you choose so that $\beta$ has both $+1$ and $-1$ in the corresponding positions. You can easily write two matrices $A$ and $B$ in this subalgebra such that $Tr(A^2)>0<Tr(B^2)$. Thus, the trace form is indefinite.


