I'm trying to understand the motivation of using Pitman-Yor (PY) processes in language modeling, in particular [Teh's hierarchical LM based on PY processes](http://www.gatsby.ucl.ac.uk/%7Eywteh/research/compling/acl2006.pdf). A motivation frequently stated in research literature is "because it produces power laws", pointing out a connection to [Zipf's law](http://en.wikipedia.org/wiki/Zipfs_law) which says that ordered word frequencies follow a power law.


From my current understanding (please correct me, if I'm wrong), this translates into a claim about the tails of the Dirichlet-Poisson distribution:


**Claim 1:** Let $\boldsymbol{\tilde{\pi}}\sim\mbox{PD}\left(\alpha,\theta\right)$ with $\alpha\in(0,1)$ and $\theta>-\alpha$ Then


$$\tilde{\pi}\_{n}/n^{-\lambda}\to X\mbox{ a.s.}$$


for some bounded random variable $X$ and $\lambda > 0$.


**My current understanding:**


* My starting point is the usual stick-breaking construction of the Pitman-Yor process with discount parameter $\alpha\in(0,1)$, concentration parameter $\theta>-\alpha$, and *non-atomic* base measure $H\in\mathcal{M}\_{1}\left(V\right)$:


$$G:=\sum\_{n\in\mathbb{N}}\pi\_{n}\delta\_{\phi\_{n}}\mbox{ with }\boldsymbol{\pi}\sim\mbox{GEM}\left(\alpha,\theta\right)\mbox{ and }\phi\_{n}\overset{\operatorname{iid}}{\sim}H,$$
where $\boldsymbol{\pi}$ can be represented as a stick breaking sequence: $\pi\_{n}=V\_{n}\prod\_{k=1}^{n-1}\left(1-V\_{k}\right)$ with $V\_{k}\sim\mbox{Beta}\left(1-\alpha,\theta+k\alpha\right)$.
* Drawing from G, $\theta\_{1},\ldots,\theta\_{n}\overset{\operatorname{iid}}{\sim} G$, gives rise to a sequence of exchangeable partitions $\left(\Pi\_{n}\right)\_{n\in\mathbb{N}}$ given by the equivalence relation $i\sim j\iff\theta\_{i}=\theta\_{j}$. The empirical distribution of partition sizes
$$\nu\_{n}=\sum\_{k=1}^{\infty}P\_{k}^{\left(n\right)}\delta\_{k}$$
with $P\_{k}^{\left(n\right)}=n^{-1}\left|\left\{ A\in\Pi\_{n}:\left|A\right|=k\right\} \right|$ almost surely converges against $\mbox{GEM}\left(\alpha,\theta\right)$.
* The distribution of the ordered sequence $\tilde{\pi}\_{1}\geq\tilde{\pi}\_{2}\geq\ldots$ of stick-breaking weights $\pi\_{1},\pi\_{2},\ldots$ is called $\mbox{PD}\left(\alpha,\theta\right)$.


**Questions:**


1. Is my understanding correct, that is: Is the PY process chosen in language modeling because claim 1 holds? If yes, could you please point me to a proof? (Potential hit: Lemma 3.11. of Pitman, Combinatorial Stochastic Processes - if the PY process gives rise to an $\left(\alpha,\theta\right)$-partition. He doesn't prove the lemma and I'd expect that a property of the PY process, which is taken for granted in so many papers, has an accessible proof.)
2. To which extent does this reasoning work if I pick a base measure $H$ that has point masses? Will the empirical distribution of partition sizes $\nu\_{n}=\sum\_{k=1}^{\infty}P\_{k}^{\left(n\right)}\delta\_{k}$ still converge against $\mbox{GEM}\left(\alpha,\theta\right)$?



Proof for power-law tail of Poisson-Dirichlet distribution (Pitman-Yor process & Zipf's law)
*As an intermediate solution, I put up with the following proposition. It only gives the asymptotic behavior of the expectation of the stick-breaking weights while almost surely results exist, for sure.*


**Proposition:** Let $\left(\pi\_{k}\right)\_{k\in\mathbf{N}}$ be the stick-breaking weights of a Pitman-Yor process with concentration parameter $\alpha$ and discount parameter $d$, that is $\boldsymbol\pi\sim\mbox{GEM}(\alpha,d)$. Then 


$$\mathbf{E}\left[\pi\_{k}\right]\in\begin{cases}
O\left(k^{-1/d}\right) & \mbox{for }d>0\\
O\left(\left(\frac{\alpha}{\alpha+1}\right)^{k}\right) & \mbox{for }d=0.
\end{cases} $$


**Statement in plain English:** In the case of the Dirichlet process, $\boldsymbol{\pi}\sim\mbox{GEM}\left(\alpha,0\right)$, the stick-breaking weights $\pi\_{k}$ decrease on average exponentially with $k$. In the case of the Pitman-Yor process, $d>0$, the tails of the sequence $\boldsymbol{\pi}\sim\mbox{GEM}\left(\alpha,d\right)$ are on average those of a power law with exponent $-1/d$. The higher the discount parameter, the heavier the tails. 


**Proof:** Recall that $\pi\_{k}=V\_{k}\prod\_{n=1}^{k-1}\left(1-V\_{n}\right)$ for independent $V\_{n}\sim\mbox{Beta}\left(1-d,\alpha+nd\right)$. By independence and linearity of the expectation we have 


$$\mathbf{E}\left[\pi\_{k}\right]=\frac{1-d}{1+\alpha+\left(k-1\right)d}\prod\frac{\alpha+nd}{1+\alpha+\left(n-1\right)d}. $$


For $d=0$ , this boils down to


$$\mathbf{E}\left[\pi\_{k}\right]=\frac{1}{1+\alpha}\left(\frac{\alpha}{1+\alpha}\right)^{k-1}. $$


For $d>0$, we represent the raising factorials $\left(\alpha\right)\_{n,d}$
 and $\left(1+\alpha\right)\_{n,d}$ by Gamma functions,


$$\mathbf{E}\left[\pi\_{k}\right] = C\frac{\Gamma\left(\alpha/d+\left(k-1\right)\right)}{\Gamma\left(\left(1+\alpha\right)/d+\left(k-1\right)\right)}.$$


Using the asymptotic ratio property of the Gamma function, $\frac{\Gamma(x+\epsilon)}{\Gamma(x)}\to x^{-\epsilon}$ with $\epsilon=1/d$, yields the claim.


