"Well, just for the fun of it. Sorry for having fun without you," is what you should say. "I'll just go ahead and do it anyway."
"Bueno, para divertirme. Lo siento por pasarlo bien sin ti", es lo que deberíais decir. "Voy a hacerlo de todas formas".