With the prevalence of Chatbot Technology and its public APIs curated by several largest IT companies like Google, IBM, Microsoft and Amazon, developers should now find it much easier to make one. Yet still, building a truly intelligent bot that understands human language and handles long conversations is definitely one of the needed improvements. Here we introduce some of the well-established Bot framework and their public APIs.
Wit.ai
Wit.ai (acquired by Facebook) is open to public and free. It builds a simple natural language processing platform and APIs for developers to build dedicated chatbot applications and devices with integration of speech recognition and voice control.
Wit.ai also provides a very smooth and intuitive conversation interfaces and introduces Story to create examples of your conversation. So Story is a dialog layer build on their stack of NLU layer which allows bots to compile complex conversations through texts. The idea is to add flexibility to dialog which is not slot-based like other chatbot applications (e.g. making an appointment, checking weather). Developers should feel comfortable to implement simple multi-turn dialogs and flow-based conversation.
They also provide an Inbox feature, therefore you can review all utterances including those which were not correctly identified. Inbox Voice is a new training tool for speech engine, every time the users speak to the bot, the audio data would be transcribed and displayed on the page as shown below. Developers can correct the result and the bot will learn from it.
They offer their HTTP APIs, therefore it is possible to create intents and entities programmatically. What Wit.ai outperforms others is their search strategy for entities, just like other platforms, developers can create a trait entity (also named intent) in the workspace that determines value by associating the whole sentence with the value of the entity. When they need to build a chatbot that requires a lot of entity values and its expression in the knowledge base, they can leverage the other two entities.
More info on entities: https://wit.ai/docs/recipes#which-entity-should-i-use
For values that are relatively stable (e.g. the details of restaurant like ratings, menu), developers can programmatically manipulate their APIs to add values. The expressions are free-text & keyword by default.
For values that would vary over time (e.g. movies), they can use a free-text entity. By tagging the entity in these expressions, Wit will try to guess a new value. For developers’ side, they need to validate whether the value is a movie name or something else and response to Wit. In general, the more samples of queries are validated, the better the bot predicts new values.
Wit.ai now supports 11 languages (English, French, German, Italian, Dutch, Spanish, Polish, Swedish, Portuguese, Estonian and Russian), and is currently releasing 39 new languages in Beta.
Projects can be made openly or privately. Just like GitHub, everyone can fork open projects and create your own version of the applications on top of them. To quickly kick start your bot prototype by leveraging others’ projects, Wit.ai is definitely the right choice.
IBM Watson
IBM provides Conversation Service that uses the intents and entities to extract key information from users’ enquiries and provides a useful response. They even provide a quite user-friendly drag and drop-based dialogue system, so the whole dialog would be graphically represented as a decision tree.
However, it is not dynamic and scalable to a long list of intents and tons of datasets. IBM does provide APIs to programmatically create intents and entities, but developers would probably find it tedious to call their APIs.
The only way for updating an entity is to make all elements (including all its values and synonyms) into a new JSON, yet the previous version of the entity would be discarded and replaced by the latest version specified in the JSON input. Therefore, it doesn’t help a lot if developers just want to add several synonyms in the entity. Perhaps, developers can consider a hybrid approach that integrates their systems on top of such static approach now. When they discover new intent/entity values, they should update them to the workspace later.
Watson now supports 22 languages and their IBM Conversation API can be accessed up to 1000 requests and 25 intents per month for free while it can be charged USD$0.0025 per API call.
Microsoft LUIS
Microsoft LUIS (Language Understanding Intelligent Service) has its pre-existing and pre-built models that can recognize entities and intents like places, emails and calendars, etc. Moreover, Microsoft provides Bot Framework, which releases open-source Bot Builder SDK (for Node.JS & .NET) for the developers to design their dialog and add more intelligent features into their bots with its Cognitive Service API in the fields of computer vision, speech, natural language processing, knowledge extraction and web search.
They provide an app dashboard which is a data visualization and reporting tool so even non-developers can monitor their app at a single glance. Most of the data would be analyzed and visualized by graphs and charts to get more insights from the users’ input.
They offer quite a handy feature that displays instant predicted score for each utterance. It allows end users to review their training data and correct it afterwards. The more relevant & diverse examples you add to the intent, the better intent prediction you’ll get from your app.
To test our trained model, LUIS provides an intuitive tool which users can either type in any utterances in Interactive Testing or provide a dataset file in JSON format for Batch Testing to examine immediate outputs.
Once we published the application, we can use the endpoint url to send queries with intents and entities.
LUIS supports several languages as English, French, Italian, German, Spanish, Brazilian Portuguese, Japanese, Korean. With initial testing, it does pretty well in simple script conversation for English version while it works well in classifying most of the keywords into different categories. In the zh-cn culture, LUIS can learn to distinguish most Chinese characters.
Currently, LUIS is free to use for 10k API requests per month at most.
Conclusion
So if you are looking for an enterprise grade solution and would like to hinder a very best user experience for your customers, you should consider these requirements:
- Total solution provider: building a chatbot is not difficult, but a top-notch one is. In AI, data is everything. We would not only deliver the chatbot, but rather drive toward a total AI solution from an initial customer pain. Therefore, we provide end to end service for enterprise that we spend a lot of time and resources for research and source the data to make sure the data we have is top notch.
- The Natural Language Support: Among all the chatbot framework and platforms, they can support over 20 languages. However, there are significant differences in the breadth of language, some languages like Chinese is hard to be “tokenized” (A “token” is the smallest unit that can be labeled in an entity). Several tech companies’ NLP engines do not work well on Chinese words (its tremendous characters make a great overhead cost to all NLP engines), let alone recognize Cantonese words which is much more sophisticated for its mixed structure of Chinese and English.
- On-premise Solution: Notwithstanding, pubic chatbot platforms do not provide on-premise solution which all your data and information have to be located in their server in order to run your application. Therefore, if you would like to create a chatbot for your company which contains private and confidential data and information, these platforms may not be a good choice for you.
- Tailor-made language model: We know quantity is quality and nobody wants to talk with a robotic robot. So Chat log reviews are fundamental to improve the conversation experience. We will deliver our bot iteratively so the bot can improve, refine and learn continually from past records. We should avoid situations of garbage in, garbage out. Even if you have a great prediction model, weak NLP engine and wrong data will direct you to the wrong result.
To conclude, if you want to build a simple chatbot which only requires handling simple scripted conversation (slot-based or basically flow-based) and basic enquiries from the customers, Public APIs will be an excellent choice for your first try. But, if you are looking for a total and dedicated solution or you are targeting one specific language, e.g Cantonese, at this moment, you still need to build your own technology and leverage some of local startups in your region.
MindLayer provides a seamless and AI-powered customer service solution that leverages its premium Chinese Natural Language Processing (NLP) engine and trilingual ChatBot Technology. We populate our products at any touchpoint in your customer’s digital journey, making customer service available in popular messaging platforms, your company website and social media page. if you are interested, you can look at https://www.mindlayer.io/ or write us at hello@mindlayer.io
Originally published at https://medium.com on April 7, 2017.
Leave a Reply