Moving to the desert might be kind of cool.
Maybe! The barren landscape and little precipitation might be tough! Would you build a house or live in a tent?
Yeah, it would definitely be tough! A house would make it more comfortable for sure. 
Correct! Also if your out at night there it might be cold! This is why its called a polar desert or a "cold desert".