If the Oort cloud has billions of objects larger than 20 km, then why do we have so few comets? According to Wikipedia:

“The Oort cloud may have billions of objects with absolute magnitude

brighter than 11 (corresponding to approximately 20-kilometre diameter)”
Most Oort cloud objects stay in the Oort cloud, and we never see them.

Comets don't last long. A comet is made of a mixture of ices and dust. A comet that enters the inner solar system has one of a few outcomes:


it could pass out of the solar system, and either never be seen again, or enter such a long orbit that it isn't seen again for an extremely long time (like Hale Bopp)
it could fall into the sun (or travel so close that it is completely broken up) (like comet ISON)
it could get caught into a short period and over time have all its ices boiled away by the heat of the sun. (asteroid 2015 TB145. Halley's comet is in the process of losing its ice with each orbit it grows smaller)
it could hit a planet. (Shoemaker-Levy 9)


In all cases, the comet is lost, destroyed or deactivated.

As comets have a short lifespan, they are rare.