Explanation of resonance stability and instability I'm having some trouble grasping the idea of planetary resonances due to what seems to me like an ambiguity. It is known that the asteroid main belt is sculpted by the Kirkwood gaps, corresponding to orbits that are in mean motion resonance with the Jupiter. As such, the asteroids reaching such orbits get ejected (unstable) from such resonance orbits. Nevertheless, objects like Pluto are dynamically stable due to their resonance with Neptune. Mercury's spin-orbital resonance makes its elliptical orbit stable around the Sun. 

Any ideas how to explain the two-fold nature of the outcomes of planetary resonances ? Any references to some literature / mathematical proof would be highly appreciated.

Thank you in advance,
Mercury and Spin-orbital resonance is pretty straight forward.  Planets and Moons are gravitationally lumpy and large bodies are somewhat fluid, even rocky bodies.   Both aspects are prone to tidal forces and that can lead to spin-orbital resonance if the tidal forces are strong enough.   Mercury's somewhat high eccentric orbit balances out with a 3:2 spin-orbital resonance. 

Resonance in 3 body orbital system is much more complicated and the math is above my pay-grade.  A proper answer to this question would have lots of math where orbital stability can be measured over long time periods and thousands of orbits.  Something precision is probably lost in an intuitive explanation.

The standard, 2 paragraph answer on Kirkwood gaps is that, if the Kirkwood object is in resonance with Jupiter, then Jupiter gives it a tug whenever it passes closest to the planet and those tugs over time destabilize the 2:1, 5:2, 7:3, 3:1 and less populated 4:1 orbital resonances.

See this summary.


  The Kirkwood gaps are regions in the main asteroid belt that have been
  cleared of asteroids by the perturbing effects of Jupiter, named for
  Daniel Kirkwood who discovered them. The Kirkwood gaps are due to
  resonances with Jupiter's orbital period. For example, an asteroid
  with a semimajor axis of 3.3 AU makes two circuits around the Sun in
  the time it takes Jupiter to make one and is thus said to be in a 2:1
  resonance orbit with Jupiter. Once every two orbits, Jupiter and such
  an asteroid would be in the same relative positions, so that the
  asteroid would experience a force in a fixed direction. Repeated
  applications of this force would eventually change the semimajor axes
  of asteroids in such orbits, creating gaps at that distance. Gaps
  occur at 4:1, 7:2, 3:1, 5:2, 7:3, and 2:1 resonances


This is only a partial answer to your question, because 3:2 (Hildas) and the much less populated 4:3 (Thules) collect around the resonance, they don't move away from it, and as you noted, Pluto and other Resonant Trans Neptunian Objects (TNOs) or Plutionos collect around the resonance as well.   They also get tiny tugs from the planet each orbit, but those tugs tend to keep them around the resonance rather than pulled away from it.

As a perhaps obvious sidebar, Resonant objects, while they orbit the sun, the need to be "herded" so to speak, by a single planet.   That's why there are relatively few resonant objects between the gas giants, and the vast, vast majority of them are between Mars and Jupiter, because Mars is small enough to not have much gravitational influence, or outside Neptune, because there are no large planetary bodies in the vicinity outside Neptune.   That's why Kirkwoods and Hildas and Resonant TNOs dominate the known orbital resonance objects.   

Cis-Jovian, for inside Jupiter's orbit or Trans-Neptunian, for outside Neptune's.   In time, we might discover Planet Nine and it might have a collection of resonant Trans and Cis-Ninian objects, but that remains to be seen, and I only mention it because I wanted to be say the word Cis-Ninian.  Obviously that won't stick if/when the planet is found and given an official name, but for now, there are theoretical resonant Trans and Cis-Ninian objects.   In fact, it's that theoretical resonance that lead to the hypothesis that Planet Nine was out there and gave a clue where to look.

So . . . meat of the answer:

Jupiter's 3/2 Hilda's, for the most part, have their aphelions passing close to one of Jupiter's Lagrange points, L3, L4 or L5.  That's why the Hilda's form a kind of triangle.   They don't orbit in a triangle, they orbit in slightly perturbed elipses, but together, they form a triangle because there are 3 main categories of Hildas, those with aphelions in Jupiter's L3, L4 and L5.  



This is explained in more detail here, under Dynamics


  The Hildas taken together constitute a dynamic triangular figure with
  slightly convex sides and trimmed apices in the triangular libration
  points of Jupiter—the "Hildas Triangle".2 The "asteroidal stream"
  within the sides of the triangle is about 1 AU wide, and in the apexes
  this value is 20-40% greater. Figure 1 shows the positions of the
  Hildas (black) against a background of all known asteroids (gray) up
  to Jupiter's orbit at January 1, 2005.7
  
  Each of the Hilda objects moves along its own elliptic orbit. However,
  at any moment the Hildas together constitute this triangular
  configuration, and all the orbits together form a predictable ring.
  Figure 2 illustrates this with the Hildas positions (black) against a
  background of their orbits (gray). For the majority of these
  asteroids, their position in orbit may be arbitrary, except for the
  external parts of the apexes (the objects near aphelion) and the
  middles of the sides (the objects near perihelion). The Hildas
  Triangle has proven to be dynamically stable over a long time span.


This resonance where the Aphelion is about 60 degrees or 180 degrees away from Jupiter keeps the Hilda objects from getting too close to Jupiter, which assists their long term stability.  Too close for a resonant object is generally bad.

Pluto, may have a similar resonant pattern with Neptune (note, I need to check my sources, I misread the linked answer) - will try to follow up. 
 some details in the accepted answer to this question. 

The Kirkwood objects are mostly too close to the sun to be gravitationally tied to one of the 3 Trojan points.   Passing close to Jupiter's orbit would make a 2:1 resonance impossible.   As a result, there isn't a stable area for them to settle in, and they are pushed by Jupiter away from resonance, unlike the 3:2 resonant Hilda's which orbit through one of the Trojan regions.  That's the best I can work this out at least.   

I'm not sure I explained that as well as I should have, but that's the gist of it, as I understand it.  The requirements for resonance are very specific and the Kirkwood objects don't have a stable zone to gather around, so the periodic tugs of Jupiter sheppard them between the resonances rather than hold them to the resonances.