What determines the configuration of orbits in a binary system? There are essentially two configurations of orbits a system with two bodies can have:

A configuration where the paths do not cross:



A configuation where the paths cross:



Note: Size of the bodies relative to one another is unimportant here.

What determines which of the two configurations a binary system will develop into?
Following the convention $\mathbf{r} = \mathbf{r}_2-\mathbf{r}_1$, $M = m_1+m_2$, in the center-of-mass frame we have, by definition,
$$\begin{eqnarray*}\mathbf{r}_1 = -\frac{m_2}{M}\mathbf{r}\text{,}\quad&amp;\mathbf{r}_2 = \frac{m_1}{M}\mathbf{r}\text{.}\end{eqnarray*}$$
Hence, $\ddot{\mathbf{r}} = -GM\hat{\mathbf{r}}/r^2$ implies that the individual orbits are similar conic sections in this frame, and moreover in the bound case they are ellipses that share a common focus at the center of mass, with all three of the distinct foci being collinear and with the center between them. Although there is a degenerate case of a circular orbit.

Therefore, there's a geometrically simple condition for the intersecting configuration: there is intersection if, and only if, the distance to apoapsis of the larger mass (smaller orbit) is greater than or equal to the distance to periapsis of the smaller mass (larger orbit). Since a general ellipse in polar coordinates about a focus can be described by
$$r = \frac{p}{1+e\cos(\phi-\phi_0)}\text{,}$$
where $e$ is the eccentricity, and the individual orbits are proportional, we have intersection if, and only if,
$$\frac{1-e}{1+e}\leq \frac{m_1}{m_2}\leq\frac{1+e}{1-e}\text{,}$$
as the apsides occur when the cosine term is $\pm 1$.