Given polynomials $a(t),b(t) \in K[t]$ where $K$ is a finite extension of $GF(2)$,
with
$$
a(t) \neq 0,
$$
 we consider the equation
$$
x^2+axy+by^2=1
$$
with unknowns $x,y$ also polynomials in $K[t]$, i.e., we want  

$$
x,y \in K[t].
$$
It is easy to prove by degree considerations that there are only finitely many
such $x,y.$


Question: How many such pairs $(x,y)$ they are ???


Is this related to search of units in some appropriate extension of the rational field $K(t)$ ???



A kind of `pell` equation in characteristic $2.$
This problem is, indeed, about searching for units. I started to write out a full answer, but it got very long, so I'll cut it short. Here are the main points:
(1) There are very good tools to answer this question (2) Giving a fully correct statement would be much easier if I knew you were comfortable with using the language of algebraic number theory over function fields. (3) You are incorrect when you say that there are always finitely many solutions. (4) In almost every case where there are finitely many solutions, there is only one.


To address point (3) directly, let $a=t$ and let $b=1$. So $(x,y)=(0,1)$ is a solution to this equation. Now, let $S$ be the ring $K[t][\zeta]/(\zeta^2+t \zeta+1)$. Then $N: x + y \zeta \mapsto x^2 + txy + y^2$ is a multiplicative map from $S$ to $R$, and $N(\zeta^n)=1$ for every $n$. The first few powers of $\zeta$ are $1+t \zeta$, $t+(t^2+1) \zeta$, $(t^2+1) + t^3 \zeta$, $t^3 + (t^4+t^2+1) \zeta$ ... A simple induction argument shows that the $n$th term has $\deg x\_n=n-2$ and $\deg y\_n=n-1$, so there is no repetition. 


You can have fun verifying that $(t^3, t^4+t^2+1)$ obeys $x^2+txy+y^2=1$, and computing the next few examples, if you like.




---


Now, for the general theory.


Let $R = K[t]$ and let $S = R[\zeta]/(z^2+a \zeta+b)$. Note that there is a symmetry $\sigma: S \to S$, given by $x + y \zeta \mapsto x + y (\zeta +a) = (x+ay) \zeta$. (Think of $\zeta+a$ as the other root of $t^2+a t +b$.) This is an order two involution, whose fixed points are $R$. Define $N: S \to R$ by $z \mapsto z \sigma(z)$. So $N$ is multiplicative, and thus $N(z)$ is a unit of $R$ if and only if $z$ is a unit in the ring $S$. The only units of $R$ are $K^\*$. So solutions to $N(z)=1$ are particular units of $R$. Moreover, $K^\*$ is finite, so solutions to $N(z)=1$ are a finite index subgroup of solutions to $N(z) \in K^\*$.


Writing it out in coordinates, $N(x+y \zeta) = x^2+axy+by^2$. 
So, the question is, what does $S$ look like, and what are its units?


All of this is just like in the case of $\mathbb{Z}[\sqrt{D}]$; the analogue of the map $\sigma$ is $x+y \sqrt{D} \mapsto x-y \sqrt{D}$.


Case 1a: $b/a^2$ is of the form $u^2+u$, for some $u \in K(t)$. This is like $\mathbb{Z}[\xi]/(\xi^2-D)$ when $D$ is a square. Then $S$ is a subring of $R \oplus R$ and so its unit group is a subgroup of the unit group of $R \oplus R$, namely $K^\* \times K^\*$. 
(In fact, I can knock it down to a sufficiently smaller group than this one, but I said I wasn't going to include every detail.)


Case 1b: $b/a^2$ is of the form $u^2+u$ for some $u \in L(t)$, where $L$ is a nontrivial algebraic extension of $K$. I couldn't find an example where this actually happens, but I couldn't quite rule it out. In this case, the unit group of $S$ would land in $L^\* \times L^\*$, as above and, again as above, I can knock it down to smaller groups.


In the remaining cases, $S$ is a domain. Let $\mathcal{O}$ be the ring of those elements of $\mathrm{Frac} \ S$ which are integral over $K[T]$. The unit group of $S$ embeds in $\mathcal{O}$ and, I believe, always has finite index. The unit group of $\mathcal{O}$ is described by Dirichlet's $S$-unit theorem. Unfortunately, Wikipedia only explains the case of a number field, not a function field, and I can't find you a reference online for the function field case. (Print references include Neukirch's book, or Weil's Basic Number Theory) But I can tell you the answer: The unit group of $\mathcal{O}$ is a finitely generated abelian group, and its rank is $s-1$, where $s$ is the number of places of $\mathcal{O}$ at $\infty$. Since $\mathcal{O}$ is quadratic over $K[t]$, which has one place at $\infty$, we know that $s$ is either $1$ or $2$. These cases are analogous to $\mathbb{Z}[\sqrt{D}]$ for $D$ negative and positive, correspondingly.


Unfortunately, telling you how to compute $s$ in general is pretty bad. But let me tell you some key cases. Set $a(t) = a\_d t^d + \cdots $ and set $b(t) = b\_e t^e+ \cdots$.


Case 2: $e<2d$. This is like the example I did above. In this case, the "Newton polygon of $\zeta^2+a \zeta+b$ at $\infty$" consists of two line segments. (If you don't know this terminology, now you know what to watch out for.) Then $s$ must be $2$. 


Case 3a: $e=2d$ and $\xi^2+a\_d \xi + b\_d$ has roots in $K$. In this case $s=2$ and there will, again, be infinitely many units of $S$.


Case 3b: $e=2d$ and $\xi^2+a\_d \xi + b\_d$ does not have roots in $K$. In this case $s=1$ and the unit group of $S$ is finite. I believe that, in fact, the unit group of $S$ is $K^\*$ and only the element $1$ in $K^\*$ gives a solution to your equation: Namely, the solution $(1,0)$.


Case 4: $e>2d$. In this case, one has to work harder to compute $s$. But, once again, the unit group of $S$ should either wind up being $K^\*$ or finite.


