IDEFICS modeling code in Transformers is for finetuning and inferencing the pre-trained IDEFICS models.
To train a new IDEFICS model from scratch use the m4 codebase (a link will be provided once it's made public)