A non-deteriorating approach to improve Natural Language Understanding in Conversational Agents

The performance of Conversational Agents (CAs) relies heavily on intent classification and entity extraction. However, the effectiveness of these tasks is often hindered by the absence of an explicit model that accounts for their interdependence. To address this limitation, we introduce a novel approach that leverages the interdependency between intents and entities to improve the Exact Match Accuracy (EMA) of CAs. The approach evaluates the consistency of the output generated by the Natural Language Understanding (NLU) component based on a specification that outlines all valid combinations of intents and entities. If an inconsistent output is detected, the ranking of predicted intents is leveraged to determine the most probable intent that aligns with the identified entities. The technique guarantees the preservation or improvement of EMA and can be applied as a post-processing step in combination with any existing NLU method that returns a ranking of intents. The proposed approach was evaluated through an ablation study conducted using three different NLU models (DIET, DCA-Net and Bi-model) on four distinct conversational datasets: ATIS, SNIPS, NLU-Benchmark, and AWPS. The results demonstrated that our method led to a consistent improvement in the EMA across all NLU methods and datasets. The magnitude of the improvement varies depending on the method, dataset, and training data size, ranging from below 1% when using DCA-Net or Bi-model on the SNIPS dataset with full training data, to over 9% when using Bi-model or DIET on AWPS with 10% of the available training data.