Their two small daughters died in 1820. As described by Taylor, the children had died of “a violent bilious fever.” They had three other girls and a boy. All four grew to adulthood. Taylor knew the difficult life of a military man. He was always against their daughters marrying military men, but all the three married military persons from the United States Army.