Rori

Overview

Rori is a math tutor chatbot that aims to provide students with access to quality math lessons that will improve their foundational math skills. The chatbot serves students from foundational math through high-school level math. It presents students with microlessons that include some instruction and then practice questions. Multiple teams worked on Rori developing different components to create the experience and functionality. I worked on a back-end API that evaluates student messages to determine if their message matched the expected answer to a question or matched a pre-approved intent. Learn more about the project

Challenge

The system needed to support a wide range of answer types for the various microlessons, such as decimals, integers, fractions, equations, and various text-based answers. Even within any one answer type, students may write a given answer in different ways. For example, for a multiple choice question with the answer “A. 4”, a student might write A, a, a., 2+2=4, A. 4, A4, “that’s 4”, and more. The API needs to accurately evaluate the student message, including the most common edge cases, while maintaining a low latency.

To meet this challenge, I built a sequence of evaluation steps. It started with text processing strategies like normalization, regex, and substitutions. If matches were not found, it used more advanced tools like a multi-intent recognizer model and an LLM. The text processing sequence achieved a 98.7% accuracy in evaluating student answers while maintaining sub 100 ms evaluations. Upon adding an LLM evaluation, the average latency remained around 100ms during peak traffic with a P99 of about 800 ms while accuracy was boosted to 99.9%.

While most of my work for this project lives in private repos, you can see some of the early prototype codebases I contributed to:

• API Service
• Chatbot Analytics Dashboard
• Mathtext repository