The house fell into ruin after emancipation, when fear of the witch's influence drove the plantation's slaves away.
The dread-filled slaves fled the plantation due to angst concerning the witch.
The slaves didn't believe in witches, and stayed on the plantation.