code: def copy_model_to_fp32(m, optim):
      
      fp32_params = [m_param.clone().type(torch.cuda.FloatTensor).detach() for m_param in trainable_params_(m)]
      optim_groups = [group['params'] for group in optim.param_groups]
      iter_fp32_params = iter(fp32_params)
      for group_params in optim_groups:
          for i in range(len(group_params)):
              if not group_params[i].requires_grad: continue # only update trainable_params_
              fp32_param = next(iter_fp32_params)
              assert(fp32_param.shape == group_params[i].shape)
              fp32_param.requires_grad = group_params[i].requires_grad
              group_params[i] = fp32_param
      return fp32_params
docs: Creates a fp32 copy of model parameters and sets optimizer parameters