Definitely had to go with the Large in this jacket. Got the White and Gravel - the Gravel is just a bit tighter in the torso but I can still button it.  Shoulders fit well and there is a slight taper at the waist.  If you have narrow shoulders and a small waist, you probably don't have to go up a size.