Case Studies

Our client is a production company which helps creators promote media content, like mobile and browser video games, attract more customers and generally increase its popularity. Our client helps hundreds of independent creators and media development studios, working with dozens of new projects daily, providing consultations and giving recommendations on how their products can be improved.

In order to give more accurate recommendations and determine which projects will be successful in the future, our client asked us to develop a system for analysis of digital media, like a video game recording. There are multiple features which could be analysed, like dominant colors, camera angle, how fast the camera moves, etc. Such a system would provide invaluable insight about media that is already successful, like which color pallete is the most pleasing to consumers, but also help give recommendations to developers on how their products can be improved to gain more success on the market.

The application needed to determine the dominant colors, the camera angle, and camera speed by analysing a video recording of a video game. There were multiple challenges in the development of this application:

1. Frame rate — the more frames the application analyses, the longer the processing time. We had to determine the optimum amount of frames to analyse.
2. Color groups — as color, in terms of digital media, is a collection of numbers, we couldn't use them as is due to gradients and multiple shades of the same color present in each frame. We needed to break the colors down into groups and determine which number of color groups is optimal.
3. Scene changes — the colors in each video scene are different, so to keep the color analysis as accurate as possible we had to figure out a way to automatically detect a scene change.
4. Camera speed — there are Python libraries that are able to determine the camera speed by calculating how fast the objects in the video move, though they cannot be used for videos with multiple objects that move at different speeds and in different directions.

We first determined the framerate which would yield satisfactory results while maintaining quick processing speed. The app takes every 30th frame for analysis as it is minimal viable value in terms of accuracy and performance.

Dominant Colors

Each frame is being quantized by 16 colors, meaning all colors are divided into 16 groups, which is the most optimal in terms of processing time and visual perception. We have used OpenCV to quantize each video frame and generate a specter image. After all specter images are generated, they are quantized again to get an average for all frames.

Scene Changes

Scene changes are best done via monitoring the color histogram and detecting sudden and drastic changes in dominant colors or a sudden appearance of one or more new colors. When such a change is detected, the application determines that a scene was changed and the color analysis process starts over for the new scene.

Camera Movement Speed

Determining the speed of camera movement is a sophisticated task due to a large number of objects moving at different speeds and in different directions within the frame. We have decided to use a Python library which is capable of determining the camera speed by locating static objects and building a motion vector. While the out-of-the-box solution worked well enough, there still were several problems, like the possible absence of static objects in the video or too many objects which move in a chaotic manner. We ended up augmenting the solution by developing custom object speed detection modules for more accurate results.

Color Harmonies

After the dominant colors are determined, the application compares the color specters to popular color harmonies to provide additional information for our client. Depending on how much the media in question adheres to a certain color harmony, recommendations on possible design changes can be given based on information on previous projects.

The application is currently used as part of a recommendation system for our client. It helped to analyse previous projects and detect what makes a successful media product and how new products can be improved. For us as a team this project was another interesting dive into computer vision.