Let $f:X\dashrightarrow Y$ be a birational map of smooth projective varieties, i.e., there exist open subsets $U\_1, \subset X$ and $U\_2 \subset Y$ such that $f|\_{U\_1} : U\_1 \rightarrow U\_2$ is an isomorphism.


There is an homomorphism induced by $f$ given by 


$$ f\_\* : Div(X) \rightarrow Div(Y) , D\mapsto \overline{ f(D|\_{U\_1}) }$$
Is known that if $f$ is a small modification then $h^0(f\_\*(D))=h^0(D)$ for any divisor $D$ of X.


Exists a relation between $h^0(D)$ and $h^0(f\_\*(D))$ in general for a divisor $D$ of X? 


If $D$ is a divisor of $X$ and $\mathcal{L}\_D$ is the associated line bundle we consider the following notation $h^0(D) = h^0(X, \mathcal{L}\_D)$.



Zero-cohomology of birational varieties
So, the latest formulation is much better, but there are still some problems with this setup. You can push-forward cycles via morphisms, but rational maps are trickier. 


First of all if the inverse of $f$ contracts a divisor, then $f\_\*$ does not (necessarily) respect linear equivalence. Here is an example. Let $X$ be your favorite smooth projective variety and $\pi:Y\to X$ the blow-up of a (closed) point $x\in X$. Let $f:X\dashrightarrow Y$ be the inverse of $\pi$ (as a rational map). Now let $\mathfrak d$ be a very ample linear system on $X$. If $D\in\mathfrak d$ is such that $x\not\in X$, then $f\_\*D\sim\pi^\*D$ and if $D'\in\mathfrak d$ is such that $x\in D'$ and, for simplicity, the multiplicity of $D'$ at $x$ is $1$ (this happens for instance if $X=\mathbb P^n$ and $\mathfrak d$ is the hyperplane class), then $f\_\*D'=\pi^\*D'-E \sim \pi^\*D-E\sim f\_\*D-E$. 


Therefore there is a short exact sequence of sheaves on $Y$:
$$
0\to \mathscr O\_Y(f\_\*D')\to \mathscr O\_Y(f\_\*D) \to \mathscr O\_E \to 0.
$$
Taking global sections one obtains another short exact sequence:
$$
0\to H^0(Y,\mathscr O\_Y(f\_\*D'))\to H^0(Y,\mathscr O\_Y(f\_\*D)) \to H^0(E,\mathscr O\_E) \to 0.
$$
Here the non-obvious exactness follows from the fact that $f\_\*D\cap E=\emptyset$ and hence the restriction of the corresponding section to $E$ is a non-zero global section of $\mathscr O\_E$. 


This shows that (using the OP's notation): 
$$
h^0(f\_\*D')= h^0(f\_\*D)-1.
$$
On the other hand, from the projection formula it follows that
$$
h^0(D)=h^0(f\_\*D)
$$
and hence
$$
h^0(D')=h^0(f\_\*D')+1
$$




---


OK, so we see that we better assume that the inverse of $f$ does not contract divisors. I assume you meant this to be part of the assumption that $f$ is a *small modification*, but you also seemed to be asking without that. Also, the above should warn you that using notation like $h^0(f\_\*D)$ is dangerous. The usage of $h^0$ suggests using the linear equivalence *class* of $D$, but $f\_\*D$ is not well-defined for that. Of course, this can still be done right, you just have to make sure to emphasize that $D$ is an explicit divisor, not a divisor class. (So for instance this is another reason why your original $D\in\mathrm{Pic}\\, X$ was bad).


Assuming that $f^{-1}$ does not contract divisors, there is of course a problem if $f$ contracts a divisor. Just take $D''=f\_\*D'$ from the above example. Clearly, $\pi\_\*D''=D'$ and we already saw that 
$$
h^0(D'')=h^0(\pi\_\*D'')-1.
$$




---


So, we're left with the case when neither $f$ nor $f^{-1}$ contract any divisors. In this case
it is indeed true what you want. Here is why:


The assumption means that in this case we have open sets $U\subseteq X$ and $V\subseteq Y$ such that $\mathrm{codim}\_X(X\setminus U)\geq 2$, $\mathrm{codim}\_Y(Y\setminus V)\geq 2$, and $f:U\to V$ is an isomorphism. In this case clearly we have that 
$$
H^0(X,\mathscr O\_X(D)) = H^0(U,\mathscr O\_U(D|\_U)) = H^0(V,\mathscr O\_V(f(D|\_U))) = H^0(Y,\mathscr O\_Y(f\_\*D)).
$$
The middle equality is obvious the other two follows from the fact that $X$ and $Y$ are $S\_2$. This is sometimes called the Hartog property. See [this MO answer](https://mathoverflow.net/questions/45347/why-does-the-s2-property-of-a-ring-correspond-to-the-hartogs-phenomenon/45354#45354) for more.




---


Note that this does not need $X$ and $Y$ to be smooth, only $S\_2$. In order to deal with divisors you're probably better off assuming that they are normal. For some musings about that see [this MO answer](https://mathoverflow.net/questions/35736/the-canonical-line-bundle-of-a-normal-variety/46663#46663).


Also, to be fair, you asked for a relation in general, not equality so I assume you are aware of some of the above. I think that in general the relationship between $h^0(D)$ and $h^0(f\_\*D)$ will be very complicated and has to do with how $D$ relates to the exceptional divisor(s).


