When could the Roche limit be larger than the Hill sphere? This answer to Are Saturn's rings stable? begins with:


  Most of Saturn's rings are inside it's Roche limit, which means they will never clump together. Tidal forces prevent this from happening.


I always confuse Roche limit and Hill sphere, so I thought I'd ask a question about both.

Question: Under what conditions could a solar system body orbiting the Sun have a Roche limit larger than its Hill sphere?
The Hill sphere is approximately given by

$$R_\mathrm{Hill} \approx a_\mathrm{p} \left(\frac{M_\mathrm{p}}{3M_\ast}\right)^\frac{1}{3}$$

Where $a_\mathrm{p}$ is the radius of the planet's orbit, and $M_\mathrm{p}$ and $M_\ast$ are the masses of the planet and the star respectively. This is an approximation to the size of the Roche lobe around the secondary.

The Roche limit (not to be confused with the Roche lobe) is the limit at which tidal forces will disrupt an object held together by its own gravity. Roche derived the following formula:

$$R_\mathrm{Roche} \approx 2.44r_\mathrm{p} \left(\frac{\rho_\mathrm{p}}{\rho_\mathrm{s}}\right)^\frac{1}{3}$$

Where $r_\mathrm{p}$ is the radius of the planet, and $\rho_\mathrm{p}$ and $\rho_\mathrm{s}$ are the densities of the planet and the satellite respectively. Note that there are various different formulae for the Roche limit depending on the different assumptions being made, see the Wikipedia Roche limit article for details. For a spherical planet, this can be rewritten in terms of the planet's mass:

$$R_\mathrm{Roche}
\approx 2.44 \left(\frac{3M_\mathrm{p}}{4 \pi \rho_\mathrm{s}}\right)^\frac{1}{3}
\approx 1.51 \left(\frac{M_\mathrm{p}}{\rho_\mathrm{s}}\right)^\frac{1}{3}$$

So the condition you're interested in becomes:

$$1.51 \left(\frac{M_\mathrm{p}}{\rho_\mathrm{s}}\right)^\frac{1}{3} \gtrsim a_\mathrm{p} \left(\frac{M_\mathrm{p}}{3M_\ast}\right)^\frac{1}{3}$$

Cancelling and rearranging gives:

$$a_\mathrm{p} \lesssim 2.18 \left(\frac{M_\ast}{\rho_\mathrm{s}}\right)^\frac{1}{3}$$

For a satellite density of 3300 kg/m3 (similar to the Moon) with a host star the mass of the Sun, this corresponds to a planetary orbit of about 2.6 solar radii.

Needless to say, this is an extremely close star–planet separation that invalidates a lot of the approximations used to compute that limit. For example, the planet will be close to or within its Roche limit with respect to the star. If it is not itself being disrupted, it will likely be significantly non-spherical.