I think the joke is that 'up in arms' sounds like the students are upset about something, but when a question is asked by a teacher, students usually raise their arms and hands to be picked to answer the question.
The teacher asked a question and the students were all up in arms .