 So I think you're going to have to decide whether or not you just want to repair the areas or reset those that are kind of out of level and deal with this, or you want to take it all up and do a properly tamped base here and then relay it down. In terms of the space between the patio, between the pavers, I imagine these are fairly wide gaps. Usually with flagstone, they're like a half inch or three-quarters of an inch or an inch or even more. And the problem is that if you put mortar in between that, it's just going to crack into a bunch of little pieces. I think what I would use is crushed gravel.