Why do bonds pay back the entire principal at maturity instead of paying it off gradually? Bonds usually pay interest only and the entire principal is paid back at maturity. On the other hand fixed rate retail loans pay back both interest and principal such that at the end no debt is remaining.

To me, the latter seems to be more manageable. The borrower never need to pay back a lump sum of money at the same time.

Why do governments and companies prefer the first structure of payments when borrowing money instead of paying annuity?
They just issue new debt to roll over the principal anyway. Unlike individuals, they do not have a fixed life span.
Trading amortising bonds in the secondary market is a pain, since you then need to keep track of the amortising schedule when pricing them. (This point was discussed in comments. The calculations for amortising bonds in 2018 are straightforward, but not in the era before digital computers, and that is when bond market conventions were developed.) Vanilla bonds with the same maturity and coupon are effectively identical, but this is not true for two amortisers with different maturity dates. Although this does not appear to be an issue now; trading conventions were largely determined in the pre-digital computer age. For example, the principal amount on a bond is used in for default recovery purposes; that is equal to the face value of a bullet bond, but has to be determined for an amortising bond, which is an awkward calculation to be done on the fly without access to digital computing technology.
One added note is that straight amortising bonds have a very low duration relative to a bullet principal payment. Many investors want to get as much duration as possible for a given investment amount.