Madison is a city in Morgan County, Georgia, United States. The population was 3,636 at the 2000 census. The city is the county seat of Morgan County. It is known regionally as "The Town Sherman Refused to Burn. " Most of its nearly 100 Antebellum homes have been carefully restored, which led to Holiday Travel magazine's naming Madison "The Prettiest Small Town in America. " Madison is the birthplace of World War II naval hero Albert T. Harris.