Let $R(X)$ be the region defined by


$$\displaystyle R(X) = \{(a,b,c) \in \mathbb{R}^3 \colon |b| \leq a \leq c, \, a,c \geq 1, \, a(4ac-b^2) \leq X\}.$$


I want to know how to estimate the sum


$$\displaystyle \sum\_{(a,b,c) \in R(X) \cap \mathbb{Z}^3} 2^{\omega(4ac-b^2)},$$


where $\omega(n)$ denotes the number of prime divisors of $n$. An upper bound suffices. 


One can show that


$$\displaystyle \sum\_{(a,b,c) \in R(X) \cap \mathbb{Z}^3} 1 = O (X \log X),$$


and it's known that the average of $2^{\omega(n)}$ over $n \leq X$ is of order $\log X$. Therefore, the expected upper bound should be about $O(X (\log X)^2)$. 



Averaging $2^{\omega(n)}$ over a region
Your guess is correct. Since $2^{\omega(n)} = \sum\_{d | n} \mu^2(d)$ one has
$$
S = \sum\_{(a,b,c) \in R(X)} 2^{\omega(4ac-b^2)} = \sum\_{d \leq X} \mu^2(d) |R\_d(X)|
$$
where $R\_d(X)$ is the set of integer triples $(a,b,c) \in R(X)$ such that $4ac \equiv b^2 \pmod d$.


Let $d$ be a squarefree integer. Let $a,b$ be such that $|b| \leq a$ and let $s = \mathrm{gcd}(4a,d)$. Then for $(a,b,c)$ to be in $R\_d(X)$ one must have $s | b^2$, or equivalently $s |b$ since $s$ is squarefree, and also $c \equiv (4a/s)^{-1} b^2 \ \mathrm{mod} \ d/s$. The number of integers $c$ like that, with $a \leq c \leq b^2/(4a) + X/(4a^2)$ is 


$$
\leq \frac{s}{d}( \frac{b^2}{4a} + \frac{X}{4a^2} - a) + 1 \leq \frac{s}{d}( \frac{X}{4a^2} - \frac{3 a}{4}) + 1\leq\frac{sX}{4 a^2d} + \mathrm{1}\_{as \leq 4d/3}.
$$


Moreover the number of integers $b$ such that $|b| \leq a$ and $s | b$ is 
$$
\leq \frac{2a}{s} + 2.
$$
Since the constraint $X \geq a(4ac-b^2)$ yields $ad \leq X$, we have
$$
|R\_d(X)| \ll \sum\_{ad \leq X} (\frac{sX}{a^2d} + \mathrm{1}\_{as \leq 4d/3})(\frac{a}{s} + 1) \\
\ll \sum\_{ad \leq X} ( \frac{X}{ad} + \frac{sX}{a^2d} + \frac{a}{s} \mathrm{1}\_{as \leq 4d/3} + 1 )
\ll \frac{X \log X}{d} + \frac{X}{d} + \sum\_{ ad \leq X} ( \frac{sX}{a^2d} + \mathrm{1}\_{as \leq d} \frac{a}{s} ).
$$
Summing over $d$ we get
$$
S \ll X \log^2(X) + \sum\_{s} \sum\_{ad \leq X \\ s|4a , s|d} ( \frac{sX}{a^2d} + \mathrm{1}\_{as \leq 4d/3} \frac{a}{s} ).
$$
For each $s$, the inner sum is 
$$
\ll \sum\_{ad \leq X/s^2 } ( \frac{X}{s^2 a^2d} + \mathrm{1}\_{as \leq 16d/3} a )
$$
The first sum is $\ll X s^{-2}$, while in the second we have the constraint $a^2 \ll X s^{-3}$ which yields
$$
\ll \sum\_{a \ll X^{1/2} s^{-3/2}} a \ll X s^{-3} + X^{1/2} s^{-3/2}.
$$
So the global sum over $s$ is $\ll X$ and we are done.


Remark: one should be able to extract from this proof an asymptotic formula of the form $c X \log^2(X) + O(X \log X)$.


