Are black holes different sizes thanks to the different sized stars that created them? If the point in space that belongs to the collapsed star (that formed the black hole) is infinitely dense, why are black holes different sizes? Is it due to them feeding on stars and everything else that they swallow?
The size of a black hole is not defined by the expanse of its matter content but (usually) its event horizon.  The radius of the event horizon  (of a non-rotating and uncharged black hole) depends on the mass:

$$r_S = \frac{2 G M}{c^2}\\ \small M : \text{black hole mass} \\ \small G: \text{gravitational constant} \\ \small c : \text{speed of light} \\ \small r_S : \text{Schwarzschild radius}$$

Yes, the size of black holes as final stage of stellar evolution stems from the original star's mass, though some mass is ejected/radiated away during the supernova.

Black holes do grow (increasing $M$ and $r_S$) when they accrete matter from disks or companion stars.

Not all black holes we (kind of) observe are from stars: for example supermassive black holes in galactic centers, which can have billions of solar masses.  However, it is not yet fully understood, how they form. 

Note: The most common black hole descriptions actually don't include any matter.  They are defined in an empty spacetime except for one singular point that is usually interpreted to hold all the matter.  The mass of the black hole is then assigned by identifying the gravitational pull in the Newtonian approximation at some distant point. 

How the interior of real, non-vacuum black holes looks like, is still up to discussion.