One more step towards full automation

Chatbots are generally considered as conversational agents automating certain human roles, as it is widely available nowadays for customer support, travel agency, banking, productivity tools, general or dedicated assistant tasks, or even for recruiting and interviewing.

However, automation should not limit only to specific human roles, instead, it should also consider the exhibition of various human behaviors in tasks where the wide variety of potential users is needed. One such task is testing chatbots with real users. And more specifically, building and testing chatbots for domains such as patient engagement in digital health.

Testing is a relatively mechanic and mundane step, the challenge is to recruit a rich variety of potential users for the task. The goal is to expose the chatbots to a mix of potential human users that exhibit a wide range of human behavioral aspects, like expert vs. novice, polite vs. impolite, cooperative vs. non-coopeative, etc.

Background

Automated testing is not new. In the context of conversational, spoken dialogue, applications, Eckert, Levin & Pieraccini worked on simulated evaluation scenarios already back in the 90s.

Also, connecting spoken dialogue systems to each other for development purposes is known as system-to-system automatic evaluation.

Implementation

A bot-to-bot conversational test framework was implemented where human behaviors are simulated via stochastic language generation. Experiments were run for two entirely different domains: enterprise productivity and patient interaction with a digital health tool. Conversational technology, automation, and simulation of human behavior through language are the main technological building blocks.

Stochastic Natural Language Generation: modeling human behavior through the expressionism of language. A wide range of human behaviors, and their unlimited set of combination, are simulated with the use of networked Markov models.

Conversational Interaction: the dialogue manager is ran in inverse mode, with constant evaluation of the incoming responses from the chatbot being tested.

Below: snapshot of the implementation and the original design