In spite of their closeness with their children , the couple grew apart .
Spite was rumored that each was having affairs .
In spite of their closeness with their children , the couple grew apart , and it was rumored that each was having affairs .