Evaluating Dialogue Systems via an Opinion

Until today, the evaluation of such algorithms happens in one fundamental way. They solve "hypothetical problems," i.e., dialog systems are tested by being asked to respond in specific scenarios and provide a "solution", i.e. a reply, to a "problem".

The replies are either explicitly compared to reference responses using overlap-based metrics (e.g., BLEU) or are evaluated by human annotators, which can also be seen as compared to (implicit) references. Instead, we propose to ask dialogue systems to tell whether a sample "solution" to a sample "problem" is good or bad. In other words, we ask another dialog system whether a conversation is fluent and coherent or not, and to what degree.