So why are there so many colors of cheese? Â· Milk type. Goat cheese is always white because the beta carotene in the goatâs diet is broken down into Vitamin A before it enters the animalâs milk. Sheep and cows retain the beta carotene in their milk, giving their cheeses a yellowish color.