Trust and Safety
Effective Incident Management for Trust & Safety Teams
One of the most easily recollect-able incidents in recent history for us may be the CrowdStrike-related outage in July 2024. It resulted in around 8.5 million systems crashing without being able to restart, with an estimated worldwide financial damage of US$10 billion. This incident highlighted how crucial effective incident management is for modern businesses, even as they aim for operational excellence. Reputation is on the line, with significant financial implications. An IBM Cost of a Data Breach Report says the global average cost of a data breach in 2024 was USD 4.88M, the highest ever. This statistic reaffirms the magnitude of responsibility of the incident management teams in organisations.
Understanding incidents
Before we delve into incident management, let’s see what incidents are. Typical examples of incidents are devices or systems crashing, cyber attacks like phishing or malware that make systems unavailable or trigger erratic behaviours, etc. But, when it comes to online platforms, the term takes on a different context. Here, there is a need to ensure that the content shared by users meets stipulated guidelines, regulations and laws of the land. A Trust and Safety team manages this aspect. For them, incidents are malicious content that slipped through content moderation and got posted online. When such content violates the platform’s policies, harms entities or systems, or gains rapid online traction (trends), it poses significant risks. There are compliance and regulatory issues apart from reputational and financial implications. Hence, Trust and Safety teams are key to an organisation's success and reputation.
According to a Trustlab survey, the topmost challenge faced by Trust and Safety teams is adapting to new and evolving online threats, followed by securing sufficient resources and budgets. It appears they have their job cut out. They need to make the best out of what they have to protect users and platforms with a carefully crafted combination of policies, processes, protocols and more. The need of the hour is to balance proactive and preventive measures while ensuring any incidents that occur despite the best efforts are caught rapidly and managed effectively. Here is a roadmap to achieve that.
The roadmap to effective incident management
- Evaluate the needs and secure necessary resources
- Define policies and protocols
- Why did it happen?
- Did any users flag the content?
- Who got impacted, and for how long, etc.
- Is it gaining traction online and being reported in the media?
- Are there any immediate risks to sensitive events (like elections), or will it trigger any crisis?
- Prioritise safety and wellness
Trust and Safety Teams must thoroughly study the landscape, the emerging trends and tools, watch out for what others are doing, and prepare a list of resources. To secure stakeholder buy-in, prioritise items by categorising them as Must-Have or Good-to-Have. For each item, explicitly state the need it fulfils, the benefits it provides, and the overall impact it will have. This clear justification is crucial for gaining consensus. This step is critical because, as the Trustlab survey indicates, decision-making involves multiple teams beyond Trust and Safety (45.9%). Ensuring all stakeholders understand the challenges and implications is vital for effective collaboration and informed decisions.
Trust and Safety teams worldwide are turning to AI tools to counter ever-evolving online threats. Being proactive and preventing incidents before they happen is key here. And AI-powered tools and predictive analytics are a must-have arsenal in the kitty of Trust and Safety teams, and could as well be the game-changer.
Draft clear policies for user content and content moderation to indicate what is allowed and what is not. Avoid confusion and build trust among the users through transparent implementation of the policies. Ensure all stakeholders involved in the Trust and Safety decisions and implementation are in sync with these policies. Policies may need periodic reviews and tweaks to ensure adequacy for the latest threats. Apart from these, protocols must be in place to handle incidents. Protocols should cover aspects such as:
The protocols should define the actions to be taken to manage the incident, keeping the context in mind. The actions could be removing the content or the account itself. After taking corrective action, conduct a root cause analysis of the issue to understand why it happened. If the content triggered an incident without violating any guidelines, an in-depth analysis can help understand potential policy gaps. Based on the findings, tweak the policies and protocols for adequacy. Communicate transparently about the incidents to the impacted parties to build trust in how you are safeguarding their accounts and information. A thorough investigation and proper media communication about major incidents also helps prevent misinformation.
Implement Safety by Design into the platform and leverage AI tools and analytics to keep up with constant threats. Ensure there are open discussions around Trust and Safety aspects, including the mental health and wellness of personnel. Dealing with abusive people and content constantly can affect the mental health of the team members. Make people aware of this aspect and provide adequate support to overcome any issues. Some support systems leading companies provide are access to mental health professionals, wellness workshops and rotational schedules to avoid fatigue.
In conclusion, effective incident management strategies are critical for a robust Trust and Safety strategy. The organisation must empower Trust and Safety teams with the right tools, resources and support.
How can Infosys BPM help?
Infosys BPM's offerings for industries cover a diverse spectrum, like healthcare, insurance, financial services and more. Our BPM services leverage Gen AI capabilities to deliver end-to-end solutions that transform business outcomes.
