Rootcode AI Contributes to RASA

We recently completed our most significant open-source contribution to RASA - one of the world's most used conversational AI frameworks used by millions of developers.

At Rootcode AI, we have been working a lot in the Conversational AI space for the last many years, and we’ve been using Rasa on many of our projects. When we realized we could help improve Rasa and fix one of the most significant issues we faced while using the framework, our team decided to invest time, build the fix, and release it to the public so that the worldwide community of developers using Rasa could benefit from it. In this article, we walk through our contribution, the problem we solved for Rasa, and the millions of developers who use it.

What is Rasa?

Just a decade ago, building conversational assistants that could carry out conversations with users in closed-domain contexts was a significant challenge in applied conversational AI. For example, building an intelligent conversational AI assistant that has a complete understanding of a business's product catalog and answers any form of user inquiries regarding that product catalog was an incredibly complex task to achieve. This complexity was primarily due to the involvement of multiple components when building a contextual conversational AI system, like handling bot memory, executing actions, classifying different user intentions, extracting important keywords from user responses, etc. Although the fundamental components necessary for building a conversational assistant were the same, there wasn’t one standard way to connect all these components to build effective conversational AI assistants. This meant that building a contextual conversational assistant was not only architecturally difficult but also technically complex to integrate all these different libraries for memory management, action execution, and intent/entity detection in one working system.

This was where Rasa entered the scene in 2016 when two developers wanted to build a unified common framework for closed-domain conversational AI assistants with all the tools necessary built-in. Soon after its initial release in 2016, Rasa became the industry standard for developing intelligent conversational AI assistants. Fast forward seven years, and today, Rasa is one of the most popular open-source frameworks for building chat and voice-based assistants used by millions of developers worldwide.

Being one of the early adopters of Rasa, we have been using the framework to build many of our conversational AI solutions for years.

Rasa Knowledge Base Actions

Our open-source contribution to Rasa is focused on the “knowledge base actions” component of the framework, which enables developers to build conversational AI assistants that can answer user inquiries based on domain-specific information. For example, one of the projects we worked on at Rootcode AI was to build a conversational AI assistant for the famous luxury ayurvedic brand Spa Ceylon. The solution required building a chatbot that could effectively mimic the conversational ability of a human customer support agent with years of knowledge about Spa Ceylon’s product catalog. This meant that in addition to answering customer inquiries regarding Spa Ceylon's products, the chatbot should be able to proactively recommend products to customers based on their needs and handle the conversation flow until a customer checks out.

Building such a versatile conversational assistant required embedding the chatbot with decades' worth of domain knowledge regarding Spa Ceylon’s product categories and customer preferences.

Rasa’s “Knowledgebase Actions” module enables developers to integrate such domain-specific knowledge bases with conversational AI assistants seamlessly.

The Rasa Knowledgebase Actions module works based on three main concepts:

1. Knowledge base: The knowledge base contains structured information about the information users will ask questions about. This information can be represented using a JSON document, which defines the information hierarchy in JSON structure. In the case of Spa Ceylon, the knowledge base consisted of hundreds of products and product categories.
2. Object type: In a knowledge base, there can be hierarchies of objects grouped together. For example, in the below knowledge base structure, under the category ‘restaurants’, multiple restaurants like “Donath and “Berlin Burrito Company” are listed.
3. Object: The object is the individual item itself with its attributes. For example, under the object type called ‘restaurants,’ we have 3 objects.

Image 3: Sample knowledge-base structure

The Problem

Being one of the early users of the knowledge-base actions module, we started experimenting extensively with this feature. And that is when we realized a fundamental problem in the information retrieval logic of the knowledge base.

The knowledge base action logic is designed to handle two main types of user inquiries.

1. The user wants to obtain a list of objects of a specific object type. For example, “Can you suggest some Italian restaurants?”, where the object type is ‘restaurants’.
2. The user wants to know about a specific attribute of an object. For example, “What is the price range of Berlin Burrito restaurant?”, where the object is “Berlin Burrito” restaurant and the attribute is ‘price range’.

However, a massive logical flaw was present in the implementation of this functionality. Let’s look at the sample conversations below.

Image 4: Conversation flow showing chatbot failing to respond when users ask inquiry about an object (Berlin Burrito Restaurant) directly

Image 5: Conversation flow showing chatbot working normally when users ask about object type (restaurants) first and ask about the attributes of an individual object

In the first of the above examples (image-4), when the user directly inquires about the “Berlin Burrito” restaurant (an object in our knowledge base), “Does the Berlin Burrito restaurant have outdoor seating?”, the chatbot fails to answer, and generates a fallback response asking the user to rephrase the question. But in the second example, when the user initially inquiries about restaurants (an object type), “restaurants around the town”, and then inquires about an attribute of the “Berlin Burrito” (“outside seating”), the bot functions as expected and generates an answer which says that the particular restaurant has outdoor seating.

Facing this problem, we delved through the programming logic behind Rasa Knowledge base actions to find the root cause. At the end of our analysis, we found out that the root cause for this bug was a sequence of logic that dictated that to perform an inquiry about an object in the knowledge base, for example, a specific restaurant like Berlin Burrito, the user has to first inquire about an object-category first, for example, “restaurants”.

This meant that for this logic to work, users should only ask questions to the chatbot, adhering to the structure of the knowledge base. For example, users must ask the bot to list some restaurants before asking about a specific restaurant's attributes. This rigidity in conversation flows is certainly non-intuitive for a user who wouldn’t know anything about the knowledge base or the structure behind it.

In fact, this problem has been widely shared in Rasa’s forums for years, and the only alternative solution for developers has been to develop their own knowledge-base search logic, which is very time-consuming. This logical flaw also made Rasa’s knowledge base actions feature difficult to use in production applications.

Image 6: Snapshot of a discussion in Rasa Forum regarding the drawback of knowledge base actions

Our Solution

The root cause of this limitation was that Rasa’s knowledge base module was designed in a way that required the user to first inquire about the object type in the knowledge-base structure (for example: “Can you suggest me some restaurants?”) before asking any inquiries regarding the objects themselves (for example: “Can I know the price range of Berlin Burrito Restaurants”). The idea here was that once a user asks an inquiry about the object type, in the backend, the memory management module of Rasa ( called “Tracker” in the Rasa-SDK) will extract and store the object type mentioned by the user in memory. When the user goes ahead and asks an inquiry about an object, the memory management module will only search for the corresponding objects under the object type that was extracted and stored in memory to answer the inquiry.

This is why, in the first example (image-4) in our above example conversations, the chatbot failed to answer when the user asked about an object (Berlin Burrito restaurant) directly instead of asking about its object type first.

To solve this limitation, we re-engineered the memory management module to extract objects from the user’s inquiry using Rasa’s entity extraction model and then directly inject it into the memory without depending on the hierarchical structure of the knowledge base.

The diagram below gives a high-level intuition of the changes we made to the Rasa SDK’s tracker module.

Image 7: Flow diagram explaining our contribution

We made changes to three major components in the Rasa SDK to implement this solution:

1. Added a new function in KnowledgeBase and InMemoryKnowledgeBase classes
2. Refactored Tracker Store’s behavior
3. Enhanced the logic within the ActionQueryKnowledgeBase class

New function in KnowledgeBase and InMemoryKnowledgeBase classes

KnowledgeBase and InMemoryKnowledgeBase classes are core classes of the Rasa knowledge base actions module. We have implemented a function within these classes that retrieves all object types from the knowledge base. Additionally, we have developed a function that matches the entities extracted by Rasa’s DIET (Dual Intent Entity Transformer) model to their corresponding object types in the knowledge base. For example, when a user asks, “What is the price range of Berlin Burrito?” these functions will infer that the user is referring to a restaurant and set the value of 'object type' as ‘restaurant’. Once the 'object type' is extracted, we pass the extracted 'object type' value to Rasa’s tracker.

Refactoring Tracker Store’s behavior

Before our contribution, Rasa's tracker store functionality updates the tracker after executing the action_query_knowledge_base function. This posed a challenge as for our fix to work, we had to pass the object type value to our tracker parallelly while the action was running. To overcome this, we re-engineered the default behavior of rasa’s tracker module by concurrently updating the value of SLOT_OBJECT_TYPE in the tracker with the extracted object type value while the function “action_query_knowledge_base” is being executed.

Enhancing the logic in ActionQueryKnowledgeBase class

The previous behavior of the ActionQueryKnowledgeBase class imposed a rigid conversational structure on users. In the past, when a user inquired about the price range of Berlin Burrito, the knowledge base function required the object type slot to have a non-empty value to execute the query function and search the knowledge base. With our contribution, we have eliminated this requirement, and now the function can directly infer the object type from the user's message. To achieve this, we refactored the knowledge base action’s logic to update the bot’s memory using entities in the user’s latest message. Initially, our refactored logic encountered challenges in handling mentions, which occur when users refer to objects using terms like "first one", "it" or "this". As a result, we further updated our refactored logic to handle mentions separately, ensuring that the action retains its ability to comprehend the context and provide appropriate responses to user inquiries.

Image 8: Rasa knowledge base failing to handle a direct object inquiry before our contribution

Image 9: Rasa knowledge base handling a direct object inquiry after our contribution

Through the modifications mentioned above, we have successfully eliminated the reliance on a rigid conversational structure. As a result, users can now engage in intuitive conversations without the need to adhere to the predefined structure of the knowledge base.

Conclusion

The mainstream commercialization of AI chatbots brings businesses the need to integrate conversational AI's power with their own data. As one of the world's most popular conversational AI frameworks, Rasa's knowledge base action plays a vital role in this aspect, offering powerful capabilities to integrate memory and external knowledge sources seamlessly. The Rasa knowledge base module lets developers connect their AI assistants with databases and APIs to retrieve domain-specific information on demand. However, one major drawback of the knowledge base actions module was present in the default ‘ActionQueryKnowledgeBase’ function, which requires users to list some items before asking about a particular object. This limitation, confining the user to a rigid conversational structure, can be counterintuitive and severely disrupt the conversational flow. Before our contribution, any developer who wanted to resolve this issue had to develop their own custom knowledge base actions, which was very time-consuming. This was evident through our own experience and discussions observed in several forums. Through our contribution, we have successfully resolved this issue that Rasa has been facing for over three years, enabling developers to directly utilize the default