**Question.** Does the equality
$$\det\left[\sin 2\pi\frac{(j-k)^2}p\right]\_{1\le j,k\le p-1}=-\frac{p^{(p-1)/2}}{2^{p-1}} $$
hold for every prime $p\equiv3\pmod4$?


I have checked the equality numerically for $p=3,7,11$. I conjecture that the equality holds for each prime $p\equiv3\pmod4$, but I don't know how to prove this.


Your comments are welcome!



Is it true that $\det\big[\sin 2\pi\frac{(j-k)^2}p\big]_{1\le j,k\le p-1}=-\frac{p^{(p-1)/2}}{2^{p-1}}$ for each prime $p\equiv3\pmod4$?
We will use the notation $e\_p(t)=\exp\left(\frac{2\pi it}{p}\right)$. First, let us show that for any $1\leq m\leq \frac{p-1}{2}$ there is an eigenvector of your matrix $A\_p$ with eigenvalue
$$
\lambda\_m=\sqrt{p}\cos\frac{2\pi m^2}{p}.
$$
To do so, for any $m\in \mathbb Z$ denote by $v\_m$ the vector from $\mathbb C^{p-1}$ with $k$-th coordinate $e\_p(2mk)$ for all $k\leq p-1$. Computation of $j$-th coordinate of $A\_pv\_m$ gives
$$
(A\_pv\_m)\_j=\sum\_{k=1}^{p-1}e\_p(2mk)\sin\frac{2\pi(j-k)^2}{p}=
$$
$$
=\frac{1}{2i}\sum\_{k=1}^{p-1}e\_p(2mk)(e\_p((j-k)^2)-e\_p(-(j-k)^2))=\frac{1}{2i}(S\_{+}(m,j)-S\_{-}(m,j)).
$$
Next, completing the sum, we get
$$
S\_{+}(m,j)=\sum\_{k=0}^{p-1}e\_p(2mk+(j-k)^2)-e\_p(j^2).
$$
Noticing that $2mk+(j-k)^2=(k+m-j)^2-m^2+2mj$, shifting the variables and using periodicity, we obtain
$$
S\_+(m,j)=e\_p(-m^2+2mj)\sum\_{k=0}^{p-1}e\_p(k^2)-e\_p(j^2).
$$
It is known that the Gauss sum here equals $i\sqrt p$, thus $S\_+(m,j)=i\sqrt pe\_p(-m^2+2mj)-e\_p(j^2)$. To compute $S\_{-}(m,j)$, notice that $S\_{-}(m,j)=\overline{S\_+(-m,j)}=-i\sqrt{p}e\_p(m^2+2mj)-e\_p(-j^2)$. Combining these, we see that
$$
(A\_pv\_m)\_j=\sqrt{p}\cos\frac{2\pi m^2}{p}e\_p(2mj)-\sin\frac{2\pi j^2}{p}.
$$
Therefore, $A\_pv\_m=\lambda\_m v\_m-s$, where $s$ is a fixed vector whose $j$-th coordinate is $\sin\frac{2\pi j^2}{p}$. Noticing that $v\_m\neq v\_{-m}$ for $1\leq m\leq \frac{p-1}{2}$ and $\lambda\_m=\lambda\_{-m}$, we see that
$$
A\_p(v\_m-v\_{-m})=\lambda\_m (v\_m-v\_{-m}).
$$
Additional bonus here is the fact that coordinates of $v\_m$ lie in $\mathbb Q(e\_p(1))$. Let us take advantage of this. Denote $v\_m-v\_{-m}=w\_m$. The cyclotomic field $\mathbb Q(e\_p(1/4))=\mathbb Q(e\_p(1),i)$ has an automorphism $\sigma$ that sends $e\_p(1)$ to $e\_p(-1)$ and $i$ to itself. One can define $\sigma$ via formula $\sigma e\_p(1/4)=e\_p((2p-1)/4)$ (which works, because $(2p-1,4p)=1$). This automorphism sends $\sin\frac{2\pi l}{p}$ to $\sin\frac{2\pi(p-1)l}{p}=-\sin\frac{2\pi l}{p}$ for integer values of $l$, so, for instance $A\_p^\sigma=-A\_p$. Therefore,
$$
A\_p(\sigma^{-1}w\_m)=\sigma^{-1}(A\_p^\sigma w\_m)=-\sigma^{-1}(A\_pw\_m)=-\lambda\_m \sigma^{-1}w\_m.
$$
Thus, all eigenvalues of $A\_p$ are $\pm \lambda\_m$ for $1\leq m\leq \frac{p-1}{2}$. This means that your identity is equivalent to
$$
(-1)^{(p-1)/2}\left(\prod\_{m\leq \frac{p-1}{2}}\sqrt{p}\cos\frac{2\pi m^2}{p}\right)^2=-\frac{p^{(p-1)/2}}{2^{p-1}}.
$$
Finally, this identity is true if and only if
$$
\left(\prod\_{m\leq \frac{p-1}{2}}2\cos\frac{2\pi m^2}{p}\right)^2=1.
$$
To establish this formula, notice that $p-m^2$ is not a square and it gives us the same cosine, so the identity is equivalent to
$$
\prod\_{m\leq p-1}2\cos\frac{2\pi m}{p}=1.
$$
This last identity follows from the fact that $2\cos\frac{2\pi m}{p}=\frac{\sin\frac{4\pi m}{p}}{\sin\frac{2\pi m}{p}}$ and $m\mapsto 2m$ is a bijection on the set of invertible residues mod $p$.


EDIT: The proof is wrong, because I forgot that $\sigma$ also acts on $\lambda\_m$. However, it seems that $-\lambda\_m$ is still always an eigenvalue and then the rest of the proof works just fine.


EDIT 2: So, it turns out that the actual proof that $-\lambda\_m$ is always an eigenvalue is quite difficult. I'll outline the proof here.
First of all, we notice that for any $k$ we have
$$
\sum\_{j=0}^{p-1}\frac{1}{\cos\frac{2\pi j}{p}+\cos\frac{2\pi k}{p}}=0.
$$
To prove that, one can notice that
$$
\sum\_{j=0}^{p-1}\frac{1}{z-\cos\frac{2\pi j}{p}}=\frac{T\_p'(z)}{T\_p(z)}=\frac{pU\_{p-1}(z)}{T\_p(z)},
$$
where $T\_p$ and $U\_{p-1}$ are Chebyshev polynomials. Also, $U\_{p-1}(-\cos\frac{2\pi k}{p})=U\_{p-1}(\cos\frac{\pi(p-2k)}{p})=0$, which concludes the proof of this observation. Now, the multisets $\{\cos\frac{2\pi m^2}{p}\}\_{0\leq m\leq p-1}$ and $\{\cos\frac{2\pi m}{p}\}\_{0\leq m\leq p-1}$ are the same, so we get, for instance, for all $m\not\equiv 0\pmod p$
$$
\sum\_{j=0}^{p-1}\frac{1}{\lambda\_j+\lambda\_m}=0.
$$
From this we see, for example, that
$$
\sum\_{j=1}^{p-1}\frac{\lambda\_j-\lambda\_0}{\lambda\_j+\lambda\_m}=p.
$$
This is a very useful identity for our proof. Next, we already showed that
$$
A\_pv\_m=\lambda\_mv\_m-s.
$$
One should also notice that $v\_0=-v\_1-\ldots-v\_{p-1}$ and
$$
s=\frac{1}{p}\sum\_{m=1}^{p-1}(\lambda\_m-\lambda\_0)v\_m.
$$
Next, take
$$
w\_m=\sum\_{j=1}^{p-1}\frac{\lambda\_j-\lambda\_0}{\lambda\_j+\lambda\_m}v\_j.
$$
Applying formulas for $A\_pv\_j$, we get
$$
A\_pw\_m=\sum\_{j=1}^{p-1}\frac{\lambda\_j-\lambda\_0}{\lambda\_j+\lambda\_m}\lambda\_jv\_j-s\sum\_{j=1}^{p-1}\frac{\lambda\_j-\lambda\_0}{\lambda\_j+\lambda\_m}.
$$
Due to our formula, the last coefficient is $p$, so
$$
A\_pw\_m=\sum\_{j=1}^{p-1}\frac{\lambda\_j-\lambda\_0}{\lambda\_j+\lambda\_m}\lambda\_jv\_j-\sum\_{j=1}^{p-1}(\lambda\_j-\lambda\_0)v\_j.
$$
Therefore, $A\_pw\_m=-\lambda\_mw\_m$, which finally concludes the proof.


