Is there really a difference by throwing the vanilla pod in as well? Recipes with milk and/or cream (panna cotta, ice cream, pastry cream, ...) often ask for the seeds of a vanilla pod (in Europe at least). A fair proportion of those also mention to throw the pod in the dairy as well. Since I re-use the pod afterwards (for vanilla sugar or vanilla extract), I have to clean them and I don't like to do so.

Is there a real flavour difference in adding the pod as well, given that you can scrape almost all the seeds out? Do recipes just ask you to do so, because you'd get more seeds in the milk or cream, or does the pod itself give a special flavour?
I think you've answered your question yourself. You use the bean itself to make vanilla sugar, so obviously there is much flavour in it as well and not just in the seeds.

My experience is that you get much more flavour out of the pod if you let it simmer in warm milk/fluid.