	After 1331, Kőszegi retired to his diocese and gradually lost influence in the royal court. In the spring of 1335, he suffered the occupation of his residence by Charles, who confiscated Győr Castle and its accessories to put pressure on the elderly bishop and his cathedral chapter in order to ensure the succession of his illegitimate son Coloman as Bishop of Győr. In January 1336, the Babonići and the Kőszegis made an alliance with the Dukes of Austria against Charles, but Nicholas Kőszegi did not participate in the conspiracy. As a result of an agreement, Kőszegi regained his estates a couple of months before his death, in the late spring-early summer of 1336.