Rori

Overview

Rori is a math tutor chatbot that aims to provide students with access to quality math lessons that will improve their foundational math skills. The chatbot serves students from foundational math through high-school level math. It presents students with microlessons that include some instruction and then practice questions. Multiple teams worked on Rori developing different components to create the experience and functionality. I worked on a back-end API that evaluates student messages to determine if their message matched the expected answer to a question or matched a pre-approved intent. Learn more about the project

Challenge

The system needed to support a wide range of answer types for the various microlessons, such as decimals, integers, fractions, equations, and various text-based answers. Even within any one answer type, students may write a given answer in different ways. For example, for a multiple choice question with the answer “A. 4”, a student might write A, a, a., 2+2=4, A. 4, A4, “that’s 4”, and more. The API needs to accurately evaluate the student message, including the most common edge cases, while maintaining a low latency.

To meet this challenge, I built a sequence of evaluation steps. It started with text processing strategies like normalization, regex, and substitutions. If matches were not found, it used more advanced tools like a multi-intent recognizer model and an LLM. The text processing sequence achieved a 98.7% accuracy in evaluating student answers against expected outcomes. Adding an LLM evaluation boosted accuracy to 99.9%