Why isn't the day backwards between leap years? So every four years, we add an extra day. A leap year. This is because a year is actually $365_{1/4}$ days long. But what happens between leap years? After two years, it would get dark at 8:00 AM instead of 8:00 PM. This is obviously not the case. What's going on here?
Leap seconds keep the clock in sync with solar time of day. Leap years keep the calendar in sync with the seasons. The two (leap seconds and leap years) have nothing to do with one another. We have leap seconds because a day is now a tiny bit longer than 86400 seconds. We have leap years because a year (tropical year) is about 365.24219 days long.

If we didn't have leap years but did have leap seconds, solar noon and noon by the clock will continue to occur at more or less the same time (ignoring cyclical changes such as the equation of time and daylight saving time). The calendar would become out of sync, becoming almost a month off from the seasons in just a century. Adding an extra day every four years doesn't quite do the trick; this is why 2100, for example, will not be a leap year.