Dudley was born on November 2, 1859. Dudley was an infant slave, born into slavery; his parents were owned by Edward B. Dudley the Governor of North Carolina from 1836 to 1841. The governor had a great impact on Dudley. One of the governor's thoughts was that all people need to be educated. Dudley kept these ideas in mind, and they affected everything he did for the rest of his life.