Rydberg was born 8 November 1854 in Halmstad in southern Sweden, the only child of Sven Rydberg and Maria Anderson Rydberg. When he was 4 years old his father died, and the family was forced to live on a small income. In 1873 he graduated from Halmstads elementärläroverk, where he received high grades in maths and physics. Later that year he enrolled in Lund University, and two years later he was awarded his bachelor's degree. In 1879 he was awarded his Doctor of Philosophy with his dissertation "Konstruktioner af kägelsnitt i 3- och 4-punktskontakt".
Rydberg began his career as an amanuensis in the institution. He became a docent in maths in 1880, and in 1882 became a docent in physics. At this time he began studying the standard atomic weight, because he wondered what was the reason for the seemingly random increase in weight for the atoms in Mendeleev's periodic system. He searched for a formula for several years to no avail.
His next work was about investigating the atomic spectra, explaining why these occurred. Rydbergs research was preceded by Johann Jakob Balmers, who presented an empirical formula for the visible spectral lines of the hydrogen atom in 1885. However, Rydberg's research led him to publish a formula in 1888 which could be used to describe the spectral lines not only for hydrogen but other elements as well. After his publication in 1890 on the subject, Rydberg returned to his fruitless research on the periodic table.
Rydberg applied for a professorship in 1897, but despite the recommendations of experts in the subject he was rejected. However, he became an extraordinary professor instead. It was not until 1909 that he was upgraded into a full professorship. To earn extra money he worked part-time as a numerical examiner at Sparbanken in Lund from 1891 and as an actuary in Malmö from 1905.
In 1913, Rydberg became very ill and was forced to slow down his pace of research, and in 1915 he was granted leave on account of his illness. He died on 28 December 1919 at Lund Hospital and was succeeded by his student Manne Siegbahn. Rydberg is buried at the northern cemetery in Lund and left his wife Lydia Carlsson (1856–1925), son Helge Rydberg (1887–1968) and daughter Gerda Rydberg (1891–1983).