Is there still a red shift when moving perpendicular to the direction of incidence? This answer suggests that when there is zero radial velocity... there will be no Doppler shift but it's not likely to be intended as precise and absolute.

I remember reading about something like this in a post here but I can't find it. While to first order there is no frequency shift when moving perpendicular (we wouldn't expect there to be be due to symmetry i.e. which way? Up or down?) I think there is a second order effect which is a Doppler shift associated with the small change in angle due to astronomical aberration.  

Question: Is this a thing? If so, what would be an expression for the frequency shift and what is it called? If the magnitude of each effect is $|v/c|$ is it always a red shift and simply

$$\Delta f/f = -\frac{v^2}{c^2}?$$
Yes, there is a transverse relativistic Doppler shift. You can think of it as being caused by time dilation. https://en.m.wikipedia.org/wiki/Relativistic_Doppler_effect

There can be a redshift or a blueshift depending on when, where and who does the measurement.

e.g. a receiver with a source going around it in a circular orbit. The receiver sees a lower frequency (redshift), by a factor $\gamma = (1 -v^2/c^2)^{-0.5}$. On the other hand, a receiver orbiting the source would receive a blueshifted signal by the same factor. When $v \ll c$ then 
$$\gamma \simeq 1 + \frac{v^2}{2c^2}\ ,$$
so whilst the standard (longitudinal) Doppler shift is of order $v/c$, the transverse Dopper shift is of order $v^2/c^2$.

NB: This scenario is chosen so that the relative motion of the source and receiver are perpendicular to the line between them. All other scenarios are complicated by the usual Doppler shift you would see because there is a component of the velocity along the line joining the source and receiver.