Let $a,b$ be coprime and say $0<a<b<2a$.


Consider the quadratic system:


$$\alpha\delta-\beta\gamma=1$$
$$(\alpha^2-(\alpha\delta+\beta\gamma))a^2b+\beta^2b^3+(2\alpha\beta-\beta\delta)ab^2-\alpha\gamma a^3=0$$
$$(-\alpha\gamma+2\gamma\delta)a^2b+(\delta^2-(\alpha\delta+\beta\gamma))ab^2-\beta\delta b^3+\gamma^2a^3=0$$


1. Does it admit an integral solution in $\alpha,\beta,\gamma,\delta$? Is so, then is there a polynomial time algorithm in $\log|ab|$?


I think there are no integral solutions. We get the criteria


$$\beta^2\equiv\beta\delta\equiv 0\bmod a$$
$$\gamma^2\equiv\alpha\gamma\equiv 0\bmod b$$


This constraint seems very restrictive with respect to $\alpha\delta-\beta\gamma=1$.


2. If not, then does it admit a rational solution in $\alpha,\beta,\gamma,\delta$? Is so, then is there a polynomial time algorithm in $\log|ab|$?



Does this quadratic system admit an integral or a rational solution?
In my comments I employed Maple, which uses tools like Grobner bases to solve polynomial equations. But now I'll try to do it by hand. Let $E\_1,E\_2,E\_3$ be the three equations. A rational solution of $\{E\_1,E\_2,E\_3\}$ can be turned into an integer solution of $\{E\_2,E\_3\}$ by multiplying by a common denominator, so I'll first look for integer solutions of $\{E\_2,E\_3\}$. Coprimality of $a,b$ is silently assumed throughout and variables are integer unless specified otherwise.


From $E\_2$, $\beta$ is a multiple of $a$, while from $E\_3$, $\gamma$ is a multiple of $b$. Put $\beta=ax, \gamma=by$. Now (using $a,b\ne 0$), the equations factor:
\begin{align}
 (bx+\alpha)(ay-bx-\alpha+\delta)&=0.\tag{$E\_{2a}$} \\
 (ay+\delta)(ay-bx-\alpha+\delta)&=0.\tag{$E\_{3a}$}
\end{align}
The first case is $\alpha=-bx$, which turns $E\_{3a}$ into $(ay+\delta)^2=0$ so $\delta=-ay$. So this integer
solution of $\{E\_2,E\_3\}$ is
$$ \alpha=-bx, \beta=ax, \gamma=by, \delta=-ay.$$
However, this gives $\alpha\delta-\beta\gamma=0$ so we can't
satisfy $E\_1$ even by scaling. So this solution is discarded.


The second case is $ay-bx-\alpha+\delta=0$, which is a factor
of both $E\_{2a}$ and $E\_{3a}$. So we can choose $\delta$ arbitrarily and set $\alpha=ay-bx+\delta$. So this integer solution of $\{E\_2,E\_3\}$ is
$$ \alpha = ay-bx+\delta, \beta=ax, \gamma=by. \tag{S} $$
We can locate all **rational solutions of $\{E\_1,E\_2,E\_3\}$**
by uniformly scaling $\alpha,\beta,\gamma,\delta$ to satisfy $E\_1$. However, the scale factor might be irrational so we must be careful. What we need is that $\alpha\delta-\beta\gamma=\rho^2$ where $\rho$ is some rational number. Solving that for $x$, substituting and scaling down by $\rho$,
we find that all rational solutions of $\{E\_1,E\_2,E\_3\}$ have the form
$$ \alpha=\frac{a^2y^2+awy+\rho^2}{(ay+w)\rho},
 \beta=\frac{(w^2+awy-\rho^2)a}{b(ay+w)\rho},
 \gamma=\frac{by}{\rho}, \delta=\frac{w}{\rho} $$
for arbitrary integer $y,w$ and rational $\rho$.
If we allow $y,w$ to be rational, we can dispense with $\rho$:
$$ \alpha=\frac{a^2y^2+awy+1}{ay+w},
 \beta=\frac{(w^2+awy-1)a}{b(ay+w)},
 \gamma=by, \delta=w. $$


Now looking for **integer solutions of $\{E\_1,E\_2,E\_3\}$** we must satisfy $E\_1$.
Substituting (S) into $E\_1$ we get
$$ (bx-\delta)(ay+\delta)=-1. \tag{$E\_1b$} $$
So we have $bx-\delta=1,ay+\delta=-1$ or vice-versa.
Both options give $(bx-\delta)^2=1$, so $\delta=bx\pm 1$,
and also $ay+bx=0$.
Since $a,b$ are coprime, we can satisfy $ay+bx=0$ only with $x=az, y=-bz$.


In summary, the integer solutions of $\{E\_1,E\_2,E\_3\}$ are
\begin{align\*}
\alpha&=1-abz,\beta=a^2z,\gamma=-b^2z,\delta=1+abz \\
\alpha&=-1-abz,\beta=a^2z,\gamma=-b^2z,\delta=-1+abz.
\end{align\*}
for arbitrary integer $z$.


