Thats a unique idea.You should consider the following idea to enhance your idea so that it would impact uniquely: Previous studies evaluate simulated dialog corpora using evaluation measures which can be automatically extracted from the dialog systems’ logs. Hover, the validity of these automatic measures has not been fully proven. In this study,  first recruit human judges to assess the quality of three simulated dialog corpora and then use human judgments as the gold standard to validate the conclusions drawn from the automatic measures.  observe that it is hard for the human judges to reach good agreement when asked to rate the quality of the dialogs from given perspectives. Hover, the human ratings give consistent ranking of the quality of simulated corpora generated by different simulation models. When building prediction models of human judgments using previously proposed automatic measures,  find that  cannot reliably predict human ratings using a regression model, but  can predict human rankings by a ranking model.
['Hua Ai', 'Diane J. Litman']
I am writing a research paper on a idea Assessing Dialog System User Simulation Evaluation Measures Using Human Judges
[{"from": "human", "value": "I am writing a research paper on a idea Assessing Dialog System User Simulation Evaluation Measures Using Human Judges"}, {"from": "gpt", "value": "Thats a unique idea.You should consider the following idea to enhance your idea so that it would impact uniquely: Previous studies evaluate simulated dialog corpora using evaluation measures which can be automatically extracted from the dialog systems\u2019 logs. Hover, the validity of these automatic measures has not been fully proven. In this study,  first recruit human judges to assess the quality of three simulated dialog corpora and then use human judgments as the gold standard to validate the conclusions drawn from the automatic measures.  observe that it is hard for the human judges to reach good agreement when asked to rate the quality of the dialogs from given perspectives. Hover, the human ratings give consistent ranking of the quality of simulated corpora generated by different simulation models. When building prediction models of human judgments using previously proposed automatic measures,  find that  cannot reliably predict human ratings using a regression model, but  can predict human rankings by a ranking model."}]