A referee is an official who enforces the rule of a sport. To settle a score means to take revenge on someone for something they did to you in the past. Settling a score can also mean to come to an agreement on a score. This is the literal interpretation.
Two referees went head to head because they had a score to settle .