We have a probability game, where we have $N$ number of events, each of which outcome can be $A,B$ or $C$. We do/will NOT know real probabilities afterwards: only the discrete outcome ($A, B$ or $C$) of each event.


Player 1 forecasts these events with certain probabilities (not only guess what is the outcome, but gives probabilities for each outcome option), and Player 2 as well with own probability estimates.


How we can know how accurate Player 1 and Player 2's predictions were (relation to reality) and how to measure the accuracy?


I have heard that one can use Akaike's information criterion to solve the problem. I was wondering another way but I need expert's opinion if this works:


I heard that one can start solving the problem by modeling the process by multinomic distribution and then take its Dirichlet's distribution. But how this leads to a solution?


Okay, I agree that one can write the solution like "Take some Dirichlet's distribution. Now use Akaike's information criterion". But I would like to know if this problem can be solved by using Dirichlet's distribution in some relative reasonable way so that you can't remove the distribution argument and the solution is still valid.



Which fortuneteller is better
I could be missing something here but I'd compute 
P[These events occur|Fortune teller 1 is telling the truth] and P[These events occur|Fortune teller 2 is telling the truth]. 


More explicitly 
Let N\_a, N\_b and N\_c be the number of times outcomes A,B and C occured. Then 
P(This happened given fortune teller 1 told the truth)= (N choose a,b,c)[(P\_1,a)^N\_a][(P\_1,b)^N\_b][(P\_1,c)^N\_c]
Where P\_1,a is the probabiliy fortune teller one assigns to event a. (similiar for b and c). 


Do the same computation for fortuneteller 2 and compare these 2. This tells us who assigned a higher probability to the outcome that came out. Suppose we might have some other criteria for "better" though (who is more often "close" for example but doubt it'll change the answer for "reasonable" definitions of "close").


