Annotation Services

10-step human-in-the-loop implementation for data annotation

Ever wondered how self-driving cars navigate the chaos of traffic? Or how chatbots answer questions with uncanny accuracy? The answer lies in the meticulous, unseen world of data annotation.

Think of data annotation as the technique to teach machines to see, hear and understand the world the way humans do. Instead of coding rules, it is about providing machines with labelled examples thereby transforming raw data into something they can learn from.

Here are some examples of data annotation:

• Image labelling: This involves annotating objects in an image, such as people, cars, animals, etc. and then adding more information to each object. For example, identifying an object first as a car and labelling it as a sedan.
• Transcribing speech to text: This involves converting spoken words into written text. An example of this would be the Amazon Prime voice search.
• Text summarisation: This involves condensing a text while preserving the essence of the original narrative.
• Text classification: This involves assigning a category to a piece of text, such as a news article or product review.

Data annotation fuels many applications, such as facial recognition, image search and medical diagnosis, to name just a few. But this critical process faces certain challenges. Labelling massive datasets manually is time-consuming and expensive, and relying solely on machines leads to inaccuracies. The solution? Integrate the strength of both approaches.

Human-in-the-loop: Man joining forces with machines

For those who worry about artificial intelligence (AI) and machine learning (ML) stealing jobs, human-in-the-loop (HITL) offers a reassuring glimpse into the future. While machines excel at crunching numbers and making quick decisions, humans add both nuanced and critical thinking to the mix. It is a win-win situation: while machines get smarter and more efficient, humans stay relevant and engaged in the workforce.

Here is how HITL works:

1. Machines take the lead. An AI/ML model makes predictions, a robot performs an action, or a simulation unfolds.
2. Humans step in when needed, provide feedback, correct mistakes or make decisions that the machine struggles with.
3. Machines learn and adapt. The machine incorporates this human input to improve its performance, either by directly adjusting its algorithms or using it as training data.

Guide to effect Human-in-the-loop for data annotation

Automation has brought speed and efficiency to data labelling, but let us face it, robots are not always the answer. When it comes to complex tasks with intricate details or nuanced data, the human touch is still irreplaceable.

For example, machines can recognise basic sentiments such as positive, negative and neutral, but understanding complex emotions like sarcasm, humour or cultural nuances requires human understanding and context. This is where HITL shines. It integrates human expertise into specific steps, making machines more accurate, adaptable and ultimately, better at their jobs.

Here is a breakdown of some key steps:

Define your goals and requirements

This includes answering questions such as these:

• What type of data are you annotating? Examples include:
  • Images for self-driving cars (bounding boxes for vehicles, pedestrians, traffic signs) or medical images (tumour detection, organ segmentation).
  • Text for sentiment analysis of social media posts (positive, negative, neutral) or topic classification of news articles (sports, politics, business).
  • Audio for speech-to-text transcription of customer service calls, identifying emotions in spoken dialogue (happy, angry, sad).
• What level of accuracy and consistency do you need? This depends on the industry. Medical and finance sectors need high levels of precision, while social media sentiment analysis can do with moderate accuracy levels.
• What tasks will humans perform? For example, correcting errors made by AI/ML models, providing additional labels, resolving ambiguities, creating complex annotations, etc.
• What resources are available? (budget, team size, technology).

Choose your HITL approach

• Active learning: Uses the predictions of a trained model to prioritise data for human annotation, focusing on uncertain cases.
• Supervised learning: Data is annotated entirely by humans. This is a labour intensive, painstaking process. However, it increases the accuracy of annotations.
• Semi-supervised learning: Uses a small set of human-labelled data to train a model, then uses the model to label more data with human supervision.
• Collaborative annotation: Humans and AI tackle complex tasks together, each playing to their strengths. Humans handle intricate actions and adapt on the fly, while AI crunches numbers and scales efficiently.

Develop annotation guidelines

• Label definitions: Establish a clear definition with illustrative examples for each label. This eliminates ambiguity and subjective interpretations, and helps easy understanding and application.
• Complex scenarios: Specific instructions and reference resources address common challenges and edge cases.
• Quality control: Defined measures ensure consistency, maintain the integrity and reliability of your annotations.

Recruit and train your annotators

• Hire qualified individuals with domain expertise. Their expertise guarantees reliable annotations, minimises revisions and empowers them to tackle complex scenarios.
• Provide comprehensive training on your specific annotation tasks and guidelines.
• Establish ongoing training and support programs.

Monitor and iterate

• Track the accuracy and consistency of your annotations.
• Gather feedback from your annotators and users.
• Continuously improve your HITL process based on your findings.

Additional tips

• Start small and scale gradually.
• Use automation tools to streamline repetitive tasks.
• Measure the ROI of your HITL process.
• Consider ethical implications and data privacy.

AI/ML is spearheading innovation that requires extensive and meaningful data for training. This involves tedious annotation. A combined approach of automation and human judgement — HITL, in essence — in data annotation improves data quality that consequently boosts the accuracy and efficiency of AI models. This empowers businesses to unlock the true potential of AI and drive meaningful change.

How can Infosys BPM help?

Infosys BPM’s AI annotation services enables data science teams to build high-quality training data. Through the HITL approach, which integrates automation with human expertise, the input quality to AI models is significantly improved, resulting in high-quality output.