James Compton-Burnett's father, Charles, was an itinerant farm labourer – among other places at Redlynch, near Salisbury, where his son was born – who later settled at French Street, in a poor area of Southampton, and went into business as a corn and coal dealer, later living at Millbrook, outside Southampton, and working as a dairyman.