There are vast volumes of data generated every day. Data annotation is the process of structuring this raw data in a format that would be easily understood by computers. It involves labelling and tagging the data to indicate what type it is—audio, video, text or images. Labelling data across large datasets is the basis on which AI systems are trained to recognize patterns and instructions. This process helps AI systems make accurate predictions using everything they have learned.
While annotation needs to be accurate at all times, there are certain areas where it is especially critical—ranging from medicine and healthcare, satellite and drone imagery to autonomous vehicles.
A significant part of data annotation these days is AI-assisted and automated. Across different spheres, AI annotation has been found to reduce the manual effort needed. For instance, according to a study published in npj Digital Medicine, a study involving pathologists and biological image annotation found that an AI-assisted system reduced annotator workload by an average of 90.6 percent, while improving the effectiveness of the resulting training data by 4.34 percent. With results like these, it is universally agreed that the speed and efficiency of automated data annotation cannot be matched by humans, leading to most of the data annotation work—at least in the first instance—being done by machines.
However, in many cases there are some nuances in the data that may not be detected by automated tools. For example, expressions of sarcasm or humor are not easily caught by machines.
Consequently, there are two broad types of annotation that have emerged—one which is completely automated and the other that relies on human-in-the-loop annotation, where human judgement is brought in at certain parts of the workflow.
Fully automated annotations are the best way forward when the model’s pre-annotation accuracy is high and the labels given by AI can be accepted without the need for further checking. In the case of human-in-the-loop annotation, the steps involve a first draft of the annotation being done by AI. As a next step, a human reviewer enters the flow and checks, corrects or, if needed, rejects the annotation done by AI. The final annotated data is then ready for the next step.
It must be noted that just having a human in the loop does not guarantee accuracy: it is quite possible that humans inadvertently introduce errors or completely miss errors that exist.
Here are some of the specific cases where human-in-the-loop annotation is needed.
- Regulated or compliance-sensitive data
- Safety-critical systems
- Ambiguous or context-dependent content
- Out-of-distribution cases or edge cases
- Contractual and legal document review
Some types of data are protected by very strict laws. Examples include patient health records in the US, protected under the Health Insurance Portability and Accountability Act (HIPAA); and, personal data in Europe, protected under the General Data Protection Regulation (GDPR). When this kind of sensitive data is being handled, it has to be qualified by a human, because if anything goes wrong or is missed, an AI system cannot be held accountable for the error. This creates a documented history that human oversight was at play for the critical checks.
In areas such as autonomous vehicles and medical devices, it is vital for the labelling to be done precisely and with a high degree of accuracy to prevent physical harm. For example, in the case of self-driving cars, if pedestrians are misidentified or a medical diagnostic tool misses a tumor, it could lead to errors with severe consequences.
AI systems are not good at interpreting humor, sarcasm, cultural references and emotional nuances, and most often misidentify and misclassify such situations. Humans bring with them their cultural and linguistic understanding that models lack. This is one of the clearest cases when human understanding is needed for reliable results.
These are rare cases where the data does not resemble anything that the AI has been trained on. An example is an emerging disease or a new type of fraud or a new product category. In such cases, the AI might confidently produce wrong answers. Humans need to be in the loop here to identify that it is an out-of-the-ordinary case, and AI systems are unable to match patterns with anything they know.
Domain expertise is needed to interpret contractual clauses, regulatory language or legal precedents. Misinterpreting any of these could lead to financial or legal exposure. Human expertise is essential to verify the data labels in such cases. Depending solely on a general-purpose AI model to handle the task is not enough.
The human involvement in data annotation is an absolute necessity. Contextual understanding and domain expertise brought forth by people are something that AI systems are unlikely to match. The goal is not to oppose automation, but to highlight that AI annotation tools work best when they are paired with human oversight and not positioned as a replacement for it.
How Infosys BPM can help
Artificial intelligence (AI) and deep learning technologies are driving the next wave of global innovation. As enterprises integrate self-learning models to boost efficiency, enhance human capability and improve customer experiences, high-quality data becomes critical. Infosys BPM’s Data Annotation Services for AI/ML help client data science teams build scalable, high-quality AI training data through a combination of platform automation and human-in-the-loop annotation.
