$\DeclareMathOperator\GL{GL}\DeclareMathOperator\Mat{Mat}$Consider the general linear group
$$
\GL(n) = \left\lbrace
\left(\begin{array}{cc}
A & C \\ 
M & B
\end{array}\right) \text{ with } A\in \Mat(k,k),\: B\in \Mat(n-k,n-k),\: M\in \Mat(n-k,k),\: C\in \Mat(k,n-k)
\right\rbrace
$$
of $n\times n$ invertible matrices, and the following $n\times n$ matrix
$$
J = \left(\begin{array}{cc}
I\_{k} & 0 \\ 
0 & 0
\end{array}\right)
$$
where $I\_k$ is the $k\times k$ identity matrix. Consider the action of $\GL(n)$ on the projective space $\mathbb{P}^N$ of $n\times n$ symmetric matrices modulo scalar given by $(P,S)\mapsto PSP^T$. Then
$$
PJP^T = 
\left(\begin{array}{cc}
AA^T & AM^T \\ 
MA^T & MM^t
\end{array}\right)
$$
Now, consider the subgroup $F\subset G$ defined by imposing $M = 0$ and $AA^T = cI\_k$ for some $c\neq 0$ in the base field (which we can assume algebraically closed and of characteristic zero). Let $X\_k = G/F$ be the orbit of $I\_k$. Then $X\_k$ has dimension $\frac{2nk-k^2+k-2}{2}$.


I would like to ask if anyone knows a method to compute the Picard group of $X\_k$. This should be $\mathbb{Z}$ for $k = 1$ and $\mathbb{Z}/n\mathbb{Z}$ for $k = n$. Thank you.



Picard group of $\mathrm{GL}(n)$-orbits
$\DeclareMathOperator\Pic{Pic}\DeclareMathOperator\GL{GL}\DeclareMathOperator\GO{GO}$
Let $G$ be a connected linear algebraic group over an algebraically closed field $K$ of characteristic 0.
Let $F\subseteq G$ be an algebraic $K$-subgroup,
*not necessarily connected*, and set $Y=G/F$.
Then there is a canonical isomorphism
$$
 \operatorname{coker}\left[{\sf X}^\*(G)\to {\sf X}^\*(F)\right]\overset{\lambda}{\longrightarrow}\Pic Y,
$$
where ${\sf X}^\*(G)$ denotes the character group of $G$,
and the map ${\sf X}^\*(G)\to {\sf X}^\*(F)$ is the restriction homomorphism.
This follows, for instance, from Theorem 2 in the introduction of
[M. Borovoi and J. van Hamel, Extended equivariant Picard complexes and homogeneous spaces, Transform. Groups 17 (2012), 51-86](https://link.springer.com/article/10.1007/s00031-011-9163-4), [arXiv:1010.3414](https://arxiv.org/abs/1010.3414).
The map $\lambda$ sends a character $\chi\in{\sf X}^\*(F)$ to the class of the ${\Bbb G}\_m$-torsor $Y\_\chi\to Y$,
where $Y\_\chi=(G\times {\Bbb G}\_m)/\chi\_\*(F)$ and
$$\chi\_\*\colon F\to G\times {\Bbb G}\_m,\quad f\mapsto(f,\chi(f))\ \text{ for }f\in F.$$


In our case
$$ F=\left\{
\begin{pmatrix}
A&B\\0&D
\end{pmatrix}
\ \ \Big |\ \ A\in\GO\_k,\ B\in{\rm Mat}\_{k,\,n-k}, D\in \GL\_{n-k}
\right\}.
$$
Clearly,
$${\sf X}^\*(F)={\sf X}^\*(\GO\_k)\oplus {\sf X}^\*(\GL\_{n-k}).$$
If $k<n$, we have ${\sf X}^\*(\GL\_{n-k})\cong {\Bbb Z}$ with generator $\det\_{n-k}$.
(If $k=n$, then of course ${\sf X}^\*(\GL\_{n-k})=0$.)


We write $X:={\sf X}^\*(\GO\_k)$.
For $k=1$ we have $X\simeq {\Bbb Z}$.
For $2\le k\le n$, the group $X$ is generated by $d=\det\_k$ and $c$ with one relation $2d-kc=0$.
In other words,
$$X\cong{\Bbb Z}^2/\langle (2,-k)\rangle.$$


If $k$ is odd, $k=2p+1$, then the element $(2, -k)\in{\Bbb Z}^2$ is primitive (indivisible),
and hence the group $X$ is cyclic.
Namely, we consider the following basis of ${\Bbb Z}^2$:
$$e\_1=2d-(2p+1)c,\quad e\_2=d-pc; $$
then $X\cong {\Bbb Z}^2/\langle e\_1\rangle \simeq {\Bbb Z}$ with a generator of infinite order $[e\_2]$.


If $k$ is even, $k=2p$, then the element $(2, -k)=2(1, -p)\in{\Bbb Z}^2$ is divisible by 2,
and hence the group $X$ is isomorphic to ${\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z}$.
Namely, we consider the following basis of ${\Bbb Z}^2$:
$$e\_1=d-pc,\quad e\_2=c;$$
then $X\cong{\Bbb Z}^2/\langle 2e\_1\rangle\simeq {\Bbb Z}/2{\Bbb Z}\oplus{\Bbb Z}$
with a generator $[e\_1]$ of order 2 and a generator $[e\_2]=[c]$ of infinite order.


We assume that $n\ge 2$. If $k<n$, the map ${\sf X}^\*(G)\to {\sf X}^\*(\GL\_{n-k})$ is bijective, and hence
$$\operatorname{coker}\left[{\sf X}^\*(G)\to {\sf X}^\*(F)\right]\simeq {\sf X}^\*(\GO\_k)=X.$$
Thus for $1\le k<n$
$$
\Pic Y\simeq
\begin{cases}
{\Bbb Z} &\text{if $k$ is odd;}\\
{\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z} &\text{if $k$ is even.}
\end{cases}
$$


For $k=n$ we have ${\sf X}^\*(F)=X$ and
$$\Pic Y\cong \operatorname{coker}\left[{\sf X}^\*(G)\to X\right].$$
Now
$$
\operatorname{coker}\left[{\sf X}^\*(G)\to X\right]={\Bbb Z}^2/\langle (1,0),(2,-n)\rangle\simeq {\Bbb Z}/n{\Bbb Z}
$$
with the generator $[c]$ of order $n$.


**EDIT:**
Our answers for $k=1$ and $k=n$ coincide with those of OP, but **not** for $k=2$, $n=3$.
Below we compute ${\sf X}^\*(F)$ and $\operatorname{coker}[{\sf X}^\*(G)\to{\sf X}^\*(F)]$ is the case $k=2$, $n>2$.


Recall that
$$\GO\_2=\{A\in\GL\_2\mid AA^T=c(A) I\_2\}.$$
Elementary calculations show that
$$ \GO\_2=\{U(a,b), V(a,b)\mid a^2+b^2\neq 0\},$$
where
$$
U(a,b)=\begin{pmatrix}a&b\\-b&a\end{pmatrix},\quad V(a,b)=\begin{pmatrix}a&b\\b&-a\end{pmatrix}.
$$
We have
$$U(a,b)\cdot U(a,-b)=(a^2+b^2)I\_2$$
whence
$$ U(a,b)^{-1}=U(a,-b)/(a^2+b^2).$$
Set
$$ U=\{U(a,b)\},\quad V=\{V(a,b)\},\quad v=V(1,0)={\rm diag}(1,-1).$$
Then $U$ is a subgroup of $\GO\_2$, and $\GO\_2=U\cup vU$.
The group $\GO\_2$ is not abelian. Indeed,
$$ vU(a,b)v^{-1}U(a,b)^{-1}=U(a,-b)\cdot U(a,b)^{-1}=U(a,-b)^2/(a^2+b^2).$$
We obtain that the commutator subgroup
$$(\GO\_2,\GO\_2)=U\_1:=\{U(a,b)\mid a^2+b^2=1\}.$$
It follows that ${\sf X}^\*(U)\cong {\Bbb Z}$ with generator $\omega$ given by $\omega(U(a,b))=a^2+b^2$.
Then
$\omega=d|\_U=c|U$ and $\ker\omega=U\_1$.
We have an isomorphism
$$\omega\_\*\colon U/U\_1\to {\Bbb G}\_m,\quad U(a,b)\cdot U\_1\mapsto a^2+b^2.$$
The map
$$ U\times\{1,v\}\to \GO\_2,\quad (U(a,b), 1)\mapsto U(a,b), \ \, (U(a,b),v)\mapsto vU(a,b)=V(a,b)$$
induces an isomorphism
$$U/U\_1\times \{1,v\}\overset\sim\longrightarrow \GO\_2/U\_1.$$
Thus $\GO\_2/(\GO\_2,\GO\_2)\cong {\Bbb G}\_m\times {\Bbb Z}/2{\Bbb Z}$,
and ${\sf X}^\*(\GO\_2)\cong {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$.
We have
\begin{align\*}&d(U(a,b))=c(U(a,b))=a^2+b^2,\\
 &c(V(a,b))=a^2+b^2,\ \text{ but }\ d(V(a,b))=-(a^2+b^2).
 \end{align\*}
Thus the character $\zeta:=d/c$ of $\GO\_2$ takes the value 1 on $U$ and the value $-1$ on $V$.
Clearly, $\zeta\neq 1$, but $\zeta^2=1$. Thus the character $\zeta$ is of order 2.
Clearly, ${\sf X}^\*(\GO\_2)\cong {\Bbb Z}\oplus {\Bbb Z}/2{\Bbb Z}$ with generator $d$ of infinite order
and generator $\zeta$ of order 2.


Now we assume that $n>2$. We compute $\operatorname{coker}[{\sf X}^\*(G)\to{\sf X}^\*(F)]$.
We have ${\sf X}^\*(G)\cong {\Bbb Z}$ with generator $d\_n$, and
$${\sf X}^\*(F)={\sf X}^\*(\GL\_{n-k})\oplus{\sf X}^\*(\GO\_2)\cong {\Bbb Z}\oplus{\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$$
with generators $d\_{n-k}$, $d=d\_2$, and $\zeta$.
The restriction map ${\sf X}^\*(G)\to{\sf X}^\*(F)$ sends $d\_n$ to $(d\_{n-k},d\_2,0)$.
It follows that $\operatorname{coker}[{\sf X}^\*(G)\to{\sf X}^\*(F)]\simeq {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$
with generator $[d\_2]$ of infinite order and generator $[\zeta]$ of order 2.
Thus $\Pic Y\simeq {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$.


I construct explicitly a ${\Bbb G}\_m$-torsor over $Y$ of order 2 in the Picard group.
We consider the homomorphism
$$\zeta\_\*\colon F\to G\times{\Bbb G}\_m,\quad f\mapsto (f,\zeta(f))\,\text{ for }f\in F,$$
where we extend the character $\zeta$ of $\GO\_2$ to $F$ by
$$
\zeta\begin{pmatrix}
A&B\\0&D
\end{pmatrix}
:=\zeta(A).
$$
We consider the quotient
$$ Y\_\zeta=(G\times{\Bbb G}\_m)/\zeta\_\*(F)$$
and the projection map
$$\pi\colon Y\_\zeta\to Y,\quad (g,z)\cdot \zeta\_\*(F)\mapsto g\cdot F.$$
Then the class in $\Pic Y$ of the ${\Bbb G}\_m$-torsor $(Y\_\zeta,\pi)$ is of order 2.


