Let $X$ be a compact in the Polish space (metric, complete, separable) and $G\subseteq X\times X$ is open. For $x\in X$ we define the section of $G$:
$$
s(x) = (y\in X|\langle x,y \rangle \in \bar{G}).
$$


Here $\bar{G}$ is a closure of $G$.


The set $A'\subseteq X$ is invariant if for all $x\in A'$ holds $s(x)\subset A'$. How to verify if there are non-empty invariant subsets of a given compact $A\subset X$? Maybe there are known equivalent problems?


It will be even helpful in the case $X = [0,1]$.


I also asked it [here](https://math.stackexchange.com/questions/44109/sets-invariant-under-sections), however haven't received an answer.



Sets invariant under sections
Your setup defines *set-valued dynamics* on $X$. More precisely, you have
$$X \stackrel{p}{\leftarrow} \overline{G} \stackrel{q}{\rightarrow} X$$
where $p$ and $q$ are the obvious projection maps. The set-valued transformation in question sends $x \in X$ to the subset $qp^{-1}(x) \subset X$. I'll write $F = qp^{-1}$ for convenience, noting that $F:X \to 2^X$ 


Now, the set $I \subset X$ is invariant in your sense precisely when $F(I) \subset I$. There is a ton of literature on set-valued dynamics, and I'm certainly in no position to do the field justice. My favorite reference is Ethan Akin's work: he has [an entire book](http://rads.stackoverflow.com/amzn/click/0821849328) called "the general topology of dynamical systems" which would be relevant. On his [webpage](http://math.sci.ccny.cuny.edu/people?name=ethan_akin) you can find a ton of references: I'd start with the "Tourist's Guide" survey paper, it is extremely well-written.




---


If you don't care much about the precise structure invariant sets sitting inside some compact subset $A \subset X$, you can use the [Conley index theory](http://math.uchicago.edu/~shmuel/AAT-readings/Mischaikow,%20conley%20survey.pdf) to get sufficient conditions for the existence of a (nonempty) maximal invariant set $I \subset A$ under the (minor) additional assumption that $A$ isolates $I$. Briefly, define the *exit set* $E \subset A$ to be the collection of those $a \in A$ whose forward orbit sets $F^n(a)$ eventually leave $A$. Now, if the homotopy type of the quotient $A/E$ (called the *homotopy conley index*) is non-trivial (again, we need some niceness properties on $A$ and $E$, maybe an ANR pair or something) then you are guaranteed a non-empty invariant set $I \subset A$. Sadly, this is not if and only if: it is possible to have invariant sets inside $A$ with trivial Conley index. 


