Let $\Omega\subset\mathbb{R}^n$ open, bounded with smooth boundary, let $s\in(0,1)$. I know that the fractional Laplacian has a sequence of eigenfunctions $\{e\_k\}\_{k\in\mathbb{N}}\subset H^s(\mathbb{R}^N)$, $e\_k=0$ a.e. on $\mathbb{R}^n\setminus\Omega$, $\forall k\in\mathbb{N}$. Moreover I know that these eigenfunctions are continuous on the whole of $\mathbb{R}^n$. My question is: are these eigenfunctions smooth in $\Omega$? Can you give me some reference on this result (if they exist)?



Eigenfunctions of the fractional Laplacian are smooth?
This is consequence of the theory of Pseudo-differential operators. Your fractional Laplacian $P\_s$ is a PDO of order $2s$. Above all, it is elliptic. This is a classical result (certainly in Hörmander) that if $P\_su\in H^r\_{\rm loc}$, then $u\in H^{r+2s}\_{\rm loc}$, where $H^{\cdots}$ is the class of Sobolev spaces.


Apply this to an eigenfunction $e\_k$, which you know is continuous, hence is $H^0\_{\rm loc}=L^2\_{\rm loc}$. Bootstrapping, you find that $e\_k$belongs to
$$\bigcap\_{\ell\in\mathbb N}H^{2\ell s}\_{\rm loc}\subset C^\infty.$$


