Let $X\sim\text{Hypergeometric}(n,k,m)$ and $Y\sim\text{Hypergeometric}(\binom{n}{2},\binom{k}{2},M)$, where $n>k>m$ are natural numbers and $M = \binom{m}{2}$. Consider $Z = \binom{X}{2}$. I want to show that $Y$ is stochastically dominated by $Z$.


Note that $X$ and $Y$ can be written as $X =\sum\_{i=1}^mX\_i$ and $Y=\sum\_{i=1}^MY\_i$ where $X\_i\sim\text{Bern}(k/n)$ and $Y\_i\sim\text{Bern}(\binom{k}{2}/\binom{n}{2})$, and the $X\_i$'s and $Y\_i$'s are not independent. The above stochastic dominance inequality to be shown translates to


$$
\sum\_{i=1}^MY\_i\leq \binom{X}{2} = \sum\_{i<j}X\_iX\_j.
$$


where the inequality is in the stochastic dominance sense. I cannot find a natural coupling for the left and right hand side summation terms.



Hypergeometric random variables domination
In general, $Y$ is not stochastically dominated by $Z$.


Indeed, suppose that $n>k>m\to\infty$ and $p:=\dfrac n{n+k}\to p\_\*\in[1/2,1)$. Then $EX=mp$ and $Var\,X\le mpq$, where $q:=1-p$. So, $\sqrt{Var\,X}\le\sqrt{mpq}=o(EX)$. So, $X$ is concentrated near $EX=mp\to\infty$ and hence $Z/m^2=X(X-1)/(2m^2)$ is concentrated near $$\dfrac{p^2}2\to\dfrac{p\_\*^2}2.$$


Similarly, letting $q\_\*:=1-p\_\*$, we see that $Y/m^2$ is concentrated near
$$\frac M{m^2}\, \dfrac{n^2/2}{n^2/2+k^2/2}\sim\dfrac{p^2}{2(p^2+q^2)}\to\dfrac{p\_\*^2}{2(p\_\*^2+q\_\*^2)}>\dfrac{p\_\*^2}2.$$


So, here $Y$ is not stochastically dominated by $Z$.




---


The above consideration was made assuming the parametrization of the hypergeometric distribution such that the population size is $n+k$. Assuming now the parametrization such that the population size is $n$, the answer is still no in general.


Indeed, suppose that $\infty\leftarrow m^2=o(\min(k,n-k))$ and $p:=k/n$ stays away from $0$ and $1$. Then the hypergeometric probabilities for $X$ are uniformly asymptotically equivalent to the corresponding probabilities of the binomial distribution with parameters $m,p$ -- cf. e.g. this [standard consideration](https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119536963.app4). Therefore and by the normal approximation to the binomial distribution,
$$X\approx N(mp,mpq)=N(EX,mpq),\tag{1}$$
again with $q:=1-p$. So, by the [delta method](https://en.wikipedia.org/wiki/Delta_method#Univariate_delta_method),
$$Z=X(X-1)/2\approx N(EZ,(mp)^2mpq).$$
On the other hand, similarly to (1),
$$Y\approx N(EY,\tfrac12\,m^2\,p^2(1-p^2)).$$
Also, as noted in a comment by the OP, $EZ=EY=:\mu$.


Thus, $Z$ and $Y$ have the same asymptotic mean $\mu$, but the asymptotic variance of $Z$ is much greater than that of $Y$, so that $Y$ is much more concentrated near $\mu$ than $Z$ is. Therefore, neither $Y$ is stochastically dominated by $Z$ nor vice versa.


