Khakis or casual slacks with a polo shirt are the rule of thumb when it comes to elegant casual day wear for men. If the resort or cruise is more upscale, your leisure wear need not be more formal, but perhaps of better design and quality. Ball caps and sneakers are fine, as long as they stay outdoors.