At first, Alan of Brittany had custody of the duke, but when Alan died in either late 1039 or October 1040, Gilbert of Brionne took charge of William.
Til at begynde med havde Alan af Bretagne forældremyndigheden over hertugen, men da Alan døde i enten slutningen af 1039 eller oktober 1040, overtog Gilbert af Brionne ansvaret for Vilhelm.