Could a habitable satellite of a gas giant have a stable subsatellite? I have set a science fiction story on a moon, orbiting a gas giant (which orbits its star at approximately the same orbit as the Earth around the Sun), and given this moon its own satellites.  The moon is Earth-sized, and the moon’s satellites are on the order of a large asteroid each (perhaps up to half the size of the Moon … Luna if there was any confusion with the number of times I said ‘moon’ in this post.)

I am a computer nerd, not an astronomer, so I’m wondering if this is something that is at least possible?  Could the orbits of the gas giant around the star, the moon around the gas giant, and the subsatellites around the moon remain stable for the requisite billion or so years for life to develop on the moon?  Could the moon be habitable?
This is a fun question, so I spend some time thinking about it. This is what I came up with.

Is this possible?

Yes, this is certainly possible. Just consider the Solar System itself. We have a massive central body, the Sun, and several tiny subsystems (planets with their moons) orbiting it.  

Is it stable?

Again, from the simple observation that the Solar System exists in its current form, we know that such a system can remain more or less stable over a significant portion of the stellar lifetime. 

A more difficult problem, I think, is if can it be formed in the first place.

Could it be habitable?

This is a tricky question as it critically depends distances between the several orbiting bodies. Lets consider the orbital mechanism and derive from what we know of the Solar System.

In the following I've assumed the gas planet is Jupiter and the fictional planetary system consists of a Sun', Jupiter', Earth' and Moon' such that the apostrophe denotes the fake object. These fake objects are identical to the real equivalents, except for their orbital configuration and relative distances. If you want a different system you can follow the same line of reasoning, but plug in different numbers for the fake objects.

Now, we know that the Earth-Moon system exists in a stable orbit around the Sun, which means that at 1 AU the gravitational force of the Sun is not strong enough to destabilize the Moon orbit. From this we can find a lower limit on the gravitational force that Jupiter' may exert on the Moon'
\begin{aligned}
  F_{J'M'} &amp;= F_{SM} \\
  \frac{ M_{J'} M_{M'}}{r_{J'M'}^2} &amp;= \frac{ M_{S} M_{M}}{r_{SM}^2}
\end{aligned}
Solving this equation we get
\begin{aligned}
 r_{J'M'} &amp;= r_{SM} \left( \frac{ M_{J'} M_{M'} }{ M_{S} M_{M} } \right)^{1/2} \\
   &amp;= r_{SM} \left( \frac{ M_{J'} }{ M_{S} } \right)^{1/2}
\end{aligned}
which evaluates to about 0.03 AU. So the Earth'-Moon' system should have an orbital radius around Jupiter' of at least 0.03 AU. 

Now the question is if the Earth-Moon system can remain bound in its orbit around Jupiter. To answer this we can safely ignore the Moon' and assume it remains tightly bound to the Earth'. The potentially destabilizing force on the Jupiter'-Earth' system is again the gravitational pull of the Sun', but now on the Earth'. We can do a similar trick as before, but now we need to account for a different mass ratio. So lets require that the fraction of forces be equal in both cases:
$$
  \frac{F_{SM}}{F_{EM}} = \frac{F_{S'E'}}{F_{J'E'}}
$$
with some algebra this gives
$$
 r_{S'J'} = r_{J'E'} \frac{r_{SM}}{r_{EM}} \left(\frac{M_E}{M_J}\right)^{1/2}
$$
which for $r_{J'E'} = $0.03 AU evaluates to a minimum orbital radius of about 0.7 AU.

This greatly surprised me as it means you can comfortably fit the entire Jupiter'-Earth'-Moon' system inside the habitable zone, meaning that in principle life should be possible.

Of course the seasons on such a planet would be rather extreme.