Gromov-Witten invariants of a class of toric varieties. Stable Betti numbers of (partial) toroidal compactifications of the moduli space of abelian varieties. With an appendix by Mathieu Dutour Sikirić 