もちろん冗談と思って彼は「あっそ」と流していましたけどかなり嫌だったそうです。ってゆうか冗談でもそんなこと言うな！
Of course he thought it was a joke and let it slide with a "hm?" but he was seriously annoyed. I mean, just don't say things like that even as a joke!