According to Jean Ryder, the ex-wife of songwriter Mike Hawker, "I Only Want to Be With You" was written soon after she and Hawker married on 1 December 1961, being inspired by Hawker's intense romantic feelings for his new bride.
Ifølge Jean Ryder, der var sangskriveren Mike Hawkers ekskone, blev "I Only Want to Be With You" skrevet kort efter hun og Hawker var blevet gift den 1. december 1961, og var blevet inspireret af Hawkers intense følelser for sin nye brud.