Summary: Researchers have built a new conversational AI agent that learns from human input. Named Evorus, the system is trained alongside people, and the team reports it becomes progressively less reliant on humans as it learns.
Source: Carnegie Mellon University.
Modern conversational agents such as Siri, Alexa and Cortana handle routine requests well but can struggle with unusual queries or follow-up questions. By keeping people in the loop, researchers at Carnegie Mellon University have developed a more flexible chatbot that can handle a wider variety of interactions.
Called Evorus, the system is different from prior human-in-the-loop chatbots because it uses human responses not just to answer questions but also to train its artificial intelligence. Jeff Bigham, an associate professor at Carnegie Mellon’s Human-Computer Interaction Institute, explains that human contributors and the AI learn together, allowing the system to gradually take on more of the conversational workload.
Like an earlier project from the same lab, Chorus, Evorus recruits crowd workers on demand—using platform services such as Amazon Mechanical Turk—to propose responses to user messages. Crowd workers vote to select the best reply, and the system records the questions and the chosen answers. Over time, Evorus begins to surface those previously approved responses as candidate answers for similar future queries. The research team also developed a method by which the AI can assist in approving messages, reducing the level of crowd participation needed to validate each response.
“Where companies have spent effort training people to phrase commands so digital assistants will understand them, we’re trying the opposite: letting people speak naturally and teaching the agent to understand,” Bigham said. This approach aims to make the agent more robust in everyday conversation by shifting the burden of adaptation onto the AI rather than the user.
The team has made Evorus available for anyone who wants to participate in the research effort. The system is not yet finalized, but volunteers can join deployment trials and help the system learn from real interactions.
A formal research paper describing Evorus is already available online and will be presented by Bigham’s team at CHI 2018, the Conference on Human Factors in Computing Systems in Montreal.
Fully automated conversational systems perform well on common, straightforward commands and in domains with narrow scope—such as providing transit schedules—where a limited set of intents and responses covers most user needs. Systems that include human contributors can answer a much wider range of questions, but relying on people is costly. Apart from paid concierge or specialized services, human-powered agents are hard to scale: a session using the earlier Chorus approach cost an average of $2.48 in crowd-worker fees.
“With Evorus, we’ve found a promising balance between machine automation and crowdsourced response,” Bigham said. The goal is for the AI to gradually handle a larger share of routine inquiries, while a relatively stable pool of crowd workers remains available to respond to uncommon or long-tail questions that the AI has not yet learned to handle.
Keeping humans involved also provides a safeguard against malicious manipulation. Ting-Hao Huang, a Ph.D. student in Carnegie Mellon’s Language Technologies Institute who worked on Evorus with Bigham and Joseph Chee Chang, noted that human oversight reduces the risk of a chatbot being hijacked to produce inappropriate or harmful content—an issue that has affected other systems when deployed without sufficient human control.
During a five-month deployment involving 80 users and 181 conversations, Evorus showed measurable gains in automation and cost-efficiency. Automated responses were selected for replies about 12 percent of the time, crowd voting requirements fell by nearly 14 percent, and the average cost of crowd work per reply decreased by approximately 33 percent.
Evorus currently operates as a text-based chatbot and was deployed using Google Hangouts, which can also accept voice input and supports access from computers, phones and wearable devices. To encourage modular growth and third-party contribution, the system uses an architecture that allows external automated question-answering components to be added over time, extending the bot’s capabilities without requiring full redesign.
Funding: The work described was supported by Project InMind, an initiative at Carnegie Mellon sponsored by Yahoo!/Oath to advance technologies for personalized digital assistants.
Source: Byron Spice, Carnegie Mellon University.
Publisher: Organized by NeuroscienceNews.com.
Image credit: Carnegie Mellon University.
Original research: Results to be presented at the Conference on Human Factors in Computing Systems.
Carnegie Mellon University. “AI Makes Conversational Agents Smarter.” NeuroscienceNews. 7 February 2018.