At some point during my research I was confronted with this problem, but I did not dedicate serious time to it. Anyway it stayed in the back of my mind and I'm still interested in hints for it. Application: asymptotic properties of Schroedinger equations, scattering.


You have two convex (compact, smooth, everything) disjoint sets in the plane. Consider a ray starting in the complement of the two sets and bouncing on the boundary of the sets in the usual way, with the ingoing and outgoing rays forming equal angles with the normal to the boundary. Q.: does it always exist a trapped ray which never leaves a ball containing the two sets? This can happen of course if the ray keeps bouncing forever between the two bodies. A trivial example is obtained if the sets have two parallel sides, and the ray is chosen perpendicular to both. Less trivial examples (even strictly convex) can be constructed by choosing a trajectory first, and then joining the dots (i.e., the turning points of the trajectory) with convex curves; with some work and some adjustments in the trajectory, you can produce plenty of examples.


But, is this always the case? given two arbitrary bodies, does it always exist a trapped ray? 


EDIT (see Pietro's comment): I mean, another trapped ray besides the 'trivial' trapped ray bouncing between the closest points of the two sets (a general version of the trivial case mentioned above of two parallel sides).


EDIT 2 (quick summary of the discussion for the benefit of future readers): the answer is yes for smooth boundaries and large (in particular, with nonempty interior) sets of initial points. A continuity argument is enough to prove this. If the boundary is non smooth problems may arise. E.g. for two polygons with a couple of facing parallel sides, the only trapped ray is the periodic one.


PS in retrospect, the question was quite elementary, but I really enjoyed to discuss it here :)



Trapped rays bouncing between two convex bodies
Yes, there is always a trapped ray. The simplest way to see it is to find the path between the two bodies that minimizes length. It is necessarily perpendicular to both surfaces.


EDIT: I see the question was edited to ask for more than this trivial answer, so the new answer: there is a unique trapped ray from any starting point, but it is not trapped in backward time unless it is on the shortest path between the bodies. One can find it by minimizing distance of a zig-zag path alternately touching the two bodies a finite number of times, then passing to a limit.


Here is a generalization: suppose you have a collection of smooth disjoint convex shapes $\{S\_i\}$ in the plane arranged in a way that no straight line intersects more than two. Then, for any doubly infinite sequence of indices $ \dots, i\_{-1}, i\_{0}, i\_{1}, \dots $ such that $i\_j \ne i\_{j+1}$, there is a unique trajectory that intersects the shapes in that order, starting with $S\_{i\_1}$ in the positive direction and $S\_{i\_0}$ going backward. If the sequence is periodic, you can find the trajectory just as for the case of two objects. For the infinite case, you can take limits. 


Even if the shapes are not convex, as long as they are smooth the trajectories still exist, but they are not necessarily unique. If you want to say something about the case when the obstacles are not smooth, you can extend the rule to make it a non-deterministic dynamical system, where a ray hitting a corner has choices which way to go.


This kind of system is classical dynamical systems, which has been well-understood since early last century. Perhaps someone more knowledgable will supply appropriate references.
It is a limiting special case of the theory of the geodesic flow on surfaces of negative curvature.


In response to a comment, here is some more detail (that doesn't itself fit into a comment).
The question was about stability and how to prove convergence under the limiting process.


To prove existence, you don't need stability: just take a sequence of longer and longer rays, and choose a convergent subsequence. This exists because of compactness of the set of possible initial directions.


To prove uniqueness: this follows from the hyperbolicity of the flow. Think of the convex obstacles as trick mirrors that make you look skinny, cylinders with a convex cross-section. The convexity implies that reflected rays diverge at least as fast as they would from a flat mirror. Successive reflected images of the two mirrors in each other get thinner and thinner, so they narrow down to a unique point. (In the three-dimensional picture, they're also narrowing vertically, just at the relatively slow rate at which images shrink with distance in Euclidean space rather than at the exponential rate resulting from mirrors that are convex to 2nd order).


One way to formalize the discussion above is by use of triangle comparison theorems. Double the complement of the convex bodies to make a surface. The surface can be smoothly approximated by a surface of nonpositive curvature if it's comforting, but that's not technically necessary; the (intuitively obvious) statement about image sizes above become cases of the [Toponogov comparison theorem](http://en.wikipedia.org/wiki/Theorem_of_Toponogov). 


