Image labeling at scale: Disruption and innovation in times of COVID-19

Author: Martin Etchart, Senior Computer Vision Scientist at Ulta Beauty

As we faced the challenge of working from home amid a pandemic, while Ulta Beauty temporarily became an e-commerce only business we recognized an opportunity: boost our AI experiences by accelerating a cross-team image labeling project at-scale. Being prepared, quickly prioritizing, and embracing failure as a possible outcome is a part of what we do with Digital Innovation at Ulta Beauty.

The #TagSquad

We collected more than 1M expert labels from 300K unique face images. Throughout the process, we consolidated and tested our in-house image labeling workflow and tool, CVAT, the Open Source project that helped make this possible.

What is image labeling, and why?

Resource: Analytics Indian Mag

Let’s talk about labeling! Image labeling/annotation/tagging are synonymous terms. Some of the most usual image labeling tasks are full image attribute labeling — being presented with an image, and selecting an attribute from a set. For example: is the image of a cat or a dog? Other forms may consist of drawing a box or a closed shape in the image and selecting an attribute for the enclosed object. Some labeling tasks may be more subjective than others, requiring one image to be labeled several times by multiple labelers to reach a consensus. Once the labeling process is complete for all images, you have a labeled dataset for your AI algorithm to learn from.

Leveraging beauty expertise

Labeling tasks consisted of an image pre-filtering stage, which was a labeling task itself, with attributes like light intensity, light direction, and sharpness annotated once for each image. The attributes enabled a more efficient labeling process for beauty-oriented labeling, such as eye characteristics, skin complexion, and skin redness. These tasks are far more subjective, so up to 6 different experts labeled each image to get an accurate consensus label.

Resource: Ulta Shade finders

Scaling up (fast!)

We split the 60+ associates into four teams of 15 with a leader in place to help manage and track progress. The labeling schedule allowed for support during working hours and daily progress reports. Beauty experts were performing tasks they had never done before, moving out of their comfort zones, adapting to new work environments and circumstances, and amazing us with the quality and pace of labels collected. Team leaders were vital to effectively manage and track progress, providing an essential human touch, closing the gap between beauty and technology, maintaining a constant feedback loop, reporting roadblocks, progress, and managing schedules.

Setting up the tools

Image courtesy of Lovely mockups

A fundamental component of this project was the labeling tool itself. We utilized our in-house tool of choice, CVAT: a free, online, interactive video and image annotation tool for computer vision maintained by the OpenVINO Project as an open-source project.

We quickly and securely set up the tool. To make this possible at scale, we needed to develop additional functionality around CVTA’s existing REST API and Python CLI. The extra features allowed us to programmatically create users and tasks, pre-load labels, track progress, dump labels, analyze quality, and detect patterns. We also developed a task tool that allowed us to assign new tasks at the start of each day automatically in addition to creating a means for labelers to request a new task on their own. Daily reports were generated for team leaders and upper management with dashboards to visualize progress at given checkpoints to ensure consistency and align objectives.

As a byproduct of our efforts, we are working to improve our data models, polish our in-house labeling and training workflow, and give back to the Open Source community by contributing with code improvements for the CVAT project.

The outcome for Ulta Beauty:

  • Efficiently repurposed the in-store associate workforce and subject matter expertise within Ulta Beauty.
  • Generated a unique partnership between beauty and technology, strengthening internal collaboration.
  • Consolidated an in-house labeling tool and workflow.

Special thanks

This project was successful thanks to the collaboration of many. Thanks to the more than 60 associates labeling who deserve a special shout-out given their pace, quality, and collaboration. A special thanks to the team that pushed themselves out of the comfort zones from the AR Innovation team that placed the labeling tool, scripting, training materials, and provided daily support and extensive reporting.

Where Beauty meets Technology