In Ravi Vakil's lecture notes ("Foundations of Algebraic Geometry", Classes 53 and 54) one can find a relative version of Serre duality (Exercise 6.1), namely:


"Suppose $\pi: X\rightarrow Y$ is a flat projective morphism of
locally Noetherian schemes, of relative dimension $n$. Assume all of the geometric fibers
are Cohen-Macaulay. Then there exists a coherent sheaf $\omega\_{X/Y}$ on $X$, along with a trace
map $R^n\pi\_\ast\omega\_{X/Y}\rightarrow\mathcal{O} \_Y$ such that, for every finite rank locally free sheaves $\mathcal{F}$ on $X$, each of
whose higher pushforwards are locally free on $Y$,
$$R^i\pi\_\ast\mathcal{F}\times R^{n-i}\pi\_\ast(\mathcal{F^\vee\otimes\omega}\_X)\rightarrow R^n\pi\_\ast\mathcal{\omega}\_X\rightarrow\mathcal{O}\_Y$$
is a perfect pairing."


For citing purposes, I'd like to have a more canonical reference (i.e. paper or textbook) of this result, but was yet unable to find any. Moreover, I'd actually like to have that result for a flat proper morphism instead of a flat projective morphism. Is it also true in this case? 


I'm sorry if this question is trivial, I'm not really familiar with algebraic geometry.
Thank you!



Serre duality in families
In this case, Serre duality in families = Grothendieck duality. At the level of generality you are asking, I suggest the paper:


Kleiman, Steven L.: Relative duality for quasicoherent sheaves. *Compositio Math.* **41** (1980), no. 1, 39–60.


<http://www.numdam.org/item/?id=CM_1980__41_1_39_0>


But if you are seriously interested in the topic you should get into the derived category formulation. A modern exposition is in


Lipman, Joseph; Hashimoto, Mitsuyasu: *Foundations of Grothendieck duality for diagrams of schemes*. Lecture Notes in Mathematics, **1960**. Springer-Verlag, Berlin, 2009.


Lipman's contribution is available also at:


<https://www.math.purdue.edu/~lipman/Duality.pdf>


