More and more online communities classify contributions based on collaborative ratings of these contributions. A popular method for such a rating-based classification is the Dawid–Skene algorithm (DSA). However, despite its popularity, DSA has two major shortcomings: (1) It is vulnerable to raters with a low competence, i.e., a low probability of rating correctly. (2) It is defenseless against collusion attacks. In a collusion attack, raters coordinate to rate the same data objects with the same value to artificially increase their remuneration. In this paper, to cope with these issues, we propose gold strategies based on the level of agreement between raters. Gold strategies adopt the notion of gold objects, i.e., contributions whose true value is known. We show that selecting gold objects at random, as is common in the literature, does not increase the accuracy of DSA in a low-competence setting to a satisfying degree. Instead, our gold strategies select contributions based on the level of agreement between community members, i.e., to which extent their ratings agree on the class of a given contribution. To maximize the net benefit of gold objects, i.e., their benefit minus their costs, we propose an adaptive algorithm. It determines the number of gold objects based on runtime information. We extensively evaluate the effectiveness of gold strategies in low-competence settings and against collusion attacks by means of simulation. We find that gold strategies based on a high level of agreement between raters improve the accuracy of DSA in low-competence settings considerably. Further, the gold strategies are highly effective against collusion attacks. Finally, the adaptive algorithm determines the optimal gold ratio for each strategy and each setting with high accuracy.
['Conny Kühne', 'Klemens Böhm']
Protecting the Dawid–Skene algorithm against low-competence raters and collusion attacks with gold-selection strategies