Case Studies

Our client, an educational institution, has equipped their classrooms with specialized devices that record lectures, allowing students to access them later if they missed a class or want to review the material. These devices already utilize AI for processing the recordings:

• Whisper-cpp-python model transcribes MP3 audio files into text and converts spoken content into written form for further processing,
• BERT-Summarizer generates concise summaries from the transcribed text and highlights key points from lengthy transcripts to aid understanding,
• Mistral AI Model creates test questions based on the summarized content.

These AI models are integrated into the backend of an application pre-installed on the recording devices, ensuring seamless interaction between transcription, summarization, and test generation processes.

We have been tasked with transforming this device and its existing functionality into an AI platform for lecture analysis.

We have enhanced the existing solution with new functionality that significantly expands its capabilities. The primary focus of the improvements was the creation of modules for processing audio and video materials to enable their automated analysis.

New Functionality

The platform now processes three types of input data:

• Videos from the teacher's stream,
• Videos from the smartboard stream,
• Audio recordings of lectures.

These data undergo multi-level processing, which includes:

• Removing long pauses, irrelevant topics, and non-relevant material from both audio and video files,
• Transcribing speech data into text with timestamps using the Whisper model,
• Integrating functionality for detecting key phrases or topics, which are highlighted in both text and audio for further analysis,
• Generating educational tests based on the analyzed text,
• Triggering events based on specified keywords. For instance, when a teacher says a specific phrase, the system records it, captures the timestamp, and sends additional data to external systems.

Smart System with RAG Integration

We’ve implemented a Retrieval-Augmented Generation (RAG) system, enabling the addition of new subject matter data. If a new subject is added to the curriculum, the client can input it into the database, and the model will incorporate this information, improving its ability to process incoming content effectively.

Multi-Language Recognition

The platform supports audio recognition in multiple languages. Specifically, it was configured to recognize English, Telugu, and Hindi, as these were priorities for the client.

Edge Device Compatibility

To utilize neural networks near the source of data ("on the edge"), various edge devices are employed. The solution supports operation on edge devices with ARM architecture, making it suitable for use in diverse conditions. Integration with external systems is facilitated through a user-friendly API, simplifying the deployment and use of the platform.

Following the enhancement of the device's functionality, the resulting platform has become a comprehensive tool for automating lecture processing:

• The devices efficiently process video and audio, removing irrelevant fragments and providing structured data for further use,
• The introduction of features for test generation and key phrase analysis has significantly accelerated the creation of educational materials, reducing the educators’ workload by more than 60%,
• The system has been successfully integrated with external educational platforms via an API, simplifying its adaptation to existing infrastructure,
• The platform operates reliably on edge devices with ARM architecture, demonstrating high performance even under resource-constrained conditions.