In the theory of the Riemann zeta function, Montgomery's Pair correlation function is defined as


$$
F(\alpha) = \frac{1}{N(T)} 
\sum\_{T < \gamma, \gamma' < 2T} T^{i \alpha (\gamma - \gamma')} \cdot \frac{4}{4+(\gamma - \gamma')^2}
$$


where $N(T)$ is the total number of zeroes of the $\zeta$ function at height between $T$ and $2T$, and $\gamma, \gamma'$ are ordinates of zeroes of the Riemann zeta function.


As $T \rightarrow \infty$ the dependance on $T$ vanishes and for this reason, by abuse of notation, we think of $F$ as just a function of $\alpha$. In effect, $F(\alpha)$ is the Fourier transform of the function that measures how often two distinct zeroes of the Riemann zeta function come together. 


Montgomery proved that $F(\alpha) = \alpha$ for $0<\alpha<1$ and conjectured that $F(\alpha)=1$ for $\alpha > 1$. He also proved that $F(\alpha)$ is positive and symmetric around $0$. However implicit in all this work is the assumption that the Riemann Hypothesis holds.


My question is the following: Is it possible to formulate an "unconditional" analogue of Montgomery's pair correlation function, for example, either by 1) allowing some additional large terms in the sum, corresponding to possible zeroes off the line, or 2) by taking into account only the $40$ percent or so of the zeroes of the Riemann zeta function that we know to lie on the critical line?


Once formulated, is it possible to obtain some non-trivial information about the unconditional analogue?



Montgomery's pair correlation function without RH?
It's a nice question. There is indeed a way to formulate an unconditional analogue. The idea is to no longer interpret $\gamma$ as ordinates, but instead label the zeroes as $\tfrac{1}{2}+i\gamma$, where $\gamma$ is a complex number with imaginary part in between -1/2 and 1/2. In this way, the Riemann hypothesis is the statement that all $\gamma$ are real, and, on RH, these $\gamma$'s correspond exactly to your $\gamma$'s. One can in this case show unconditionally that


$$
\lim\_{t\rightarrow\infty} \frac{1}{N(T)}\sum\_{T<\Re(\gamma),\Re(\gamma')<2T} T^{i\alpha(\gamma-\gamma')}\frac{4}{4+(\gamma-\gamma')^2} = \alpha
$$


for fixed $0 < \alpha < 1$. (If you want this uniformly for $\alpha$ you'll have to restrict $\alpha$ away from $1$ and $0$, or modify things around $0$.) Note that $\gamma-\gamma'$ may be complex.


Observations of this sort go back at least to Rudnick and Sarnak's paper "Zeros of principal L-functions and random matrix theory," where they write of the k-point correlation sums


$$
\sum\_{\gamma\_1,\gamma\_2,...\gamma\_k \textrm{ distinct}}h\Big(\frac{\gamma\_1}{T}\Big)\cdots h\Big(\frac{\gamma\_k}{T}\Big)f(\frac{L}{2\pi}\gamma\_1,...,\frac{L}{2\pi}\gamma\_k\Big)
$$


"if $h$ and $f$ are defined for complex argument and are localized, then the sums make sense even if we do not assume RH." This means that $f(x+iy) = \int e((x+iy)\xi)\hat{f}(\xi)d\xi$.


The reason one can do this here and with the slightly different 2-point sum Montgomery considers is that the explicit formulas relating zeroes to primes that Montgomery uses in his proof remain true unconditionally so long as the $\gamma$ representing an ordinate of a zero is replaced by $\gamma$ as defined above. I don't know that, for the Montgomery 2-point sum, this argument appears explicitly in print anywhere, but it's an instructive exercise to go through the standard proof seeing that everything still works and you can arrive at the formulation I have above.


Recalling the symmetry of the zeros (that is, if $\tfrac{1}{2}+i\gamma$ is a zero, then so are $\tfrac{1}{2}-i\gamma$, $\tfrac{1}{2}+i\bar{\gamma}$, and $\tfrac{1}{2}-i\bar{\gamma}$), one can see that $F(\alpha)$ (if this limit exists) is positive and symmetric using the same argument that Montgomery does. (Positivity requires a slight modification -- it is here that one needs conjugate symmetry in $\gamma$; I can elaborate if you'd like.)


I don't think anything can be said about the zeros that are already known to lie on the critical line, because in particular I don't think anything explicit can be said about the relation between these zeroes and the primes (about which we can say at least a few useful things). Using a zero-density estimate, it may be possible to show that


$$
\lim\_{t\rightarrow\infty} \frac{1}{N(T)}\sum\_{T<\Re(\gamma),\Re(\gamma')<2T} T^{i\alpha(\Re(\gamma-\gamma'))}\frac{4}{4+(\Re(\gamma-\gamma'))^2} = \alpha
$$


for $\alpha$ restricted as above, even if RH is false, returning our attention once again to the ordinates of zeroes. This would be interesting (to me) for a variety of reasons, but without a clever idea I don't think any known zero-density estimates are enough to get here.


