I'm 5'6'' tall and considered pretty skinny.  The port/burgundy-color shirt in X-Small fits great with enough room for layering.  If you usually wear Small, you should buy X-Small for this shirt.  It's sturdy and stylish at the same time, so good for all kinds of activities.