Is there a lower limit for the altitude of orbiting objects? As I understand it, one object can orbit another at a variety of altitudes, and the stability of the orbit is determined by (among other things) the speed of the orbiting object.  Go too slowly and you'll fall into the larger object, go too quickly and you'll fly off into space, but go at just the right speed and you'll orbit indefinitely.  And while an object could be in a stable orbit at Altitude X by going Speed A, it could also be in an equally stable orbit at Altitude Y by going Speed B.

If that's true (and given my limited understanding, feel free to say it simply isn't), is there a lower limit?  An atmosphere would obviously cause drag, but given a lack of atmosphere, could you theoretically have an asteroid stably orbiting an Earth-sized planet at an altitude of 50 feet, if it were going fast enough?

If not, why not?  What forces limit the closeness of an orbiting object?  Also, simple explanations would be most appreciated: I've heard of things like Lagrange points and such, but I don't have a good understanding of them.
For really low orbits over an atmosphereless body, the body needs to have a uniform density. Otherwise the gravity field is not symmetric, the orbit changes shape over time, and you end up with a crater.
NASA had trouble with this, lunar Mascons, during the Apollo era. In one case: 


  "The Moon has no atmosphere to cause drag or heating on a spacecraft, so you can go really low: Lunar Prospector spent six months orbiting only 20 miles (30 km) above the surface." 


In another: 


  The orbit of PFS-2 rapidly changed shape and distance from the Moon. In 2-1/2 weeks the satellite was swooping to within a hair-raising 6 miles (10 km) of the lunar surface at closest approach. As the orbit kept changing, PFS-2 backed off again, until it seemed to be a safe 30 miles away. But not for long: inexorably, the subsatellite's orbit carried it back toward the Moon. And on May 29, 1972—only 35 days and 425 orbits after its release—PFS-2 crashed.


Now that the gravity field of the moon is mapped, it turns out that there are some low orbits that are reasonably stable: 


  "What counts is an orbit's inclination," that is, the tilt of its plane to the Moon's equatorial plane. "There are actually a number of 'frozen orbits' where a spacecraft can stay in a low lunar orbit indefinitely. They occur at four inclinations: 27º, 50º, 76º, and 86º"—the last one being nearly over the lunar poles.