A Collaborative Approach to Assessing the Challenges of AI-Driven Harms

Summary

Amidst a rapid proliferation of synthetic image- and video-based content in online ecosystems, our research addresses urgent gaps in understanding AI-driven harms by examining real-world cases of extremist visual content and evaluating existing AI-powered moderation solutions. This project seeks to document how generative AI enables and accelerates visual hate speech; assess the feasibility of AI-supported content moderation tools for detecting harmful visual content; and develop evidence-based recommendations for policymakers and platform developers to mitigate these harms while preserving legitimate expression. ​In fostering international collaboration between UK and Canadian researchers with demonstrated expertise in extremism, AI-driven harms, and digital media, this project seeks to highlight shared concerns and opportunities to strengthen regulatory responses in both British and Canadian contexts.

Four rapid case studies: ​​

• AI-Generated Content Posted in Support of “Unite the Kingdom”: examined how AI-generated visual content was weaponised during the September 13, 2025 Unite the Kingdom rally — one of Britain’s largest far-right demonstrations – in Instagram reels and TikTok videos.
• GenAI and Symbolic Violence: explored the visual culture of AI-generated content on TikTok and Instagram to identify recurring themes and stylistic conventions of videos that target young men.
• Gendered Harms in Manosphere Memes: explored the nuances of harm in memes shared by manosphere-affiliated Instagram accounts and identifies concerns around misogynist visuals created and shared within manosphere ecosystems.
• Automated Content Moderation for AI Visuals: assessed the feasibility of automated content moderation for far-right and male supremacist videos and images using SightEngine, an accessible online moderation tool.

Collaborating with University of Southampton and Waterloo University, Canada.

Activities and Achievements

In addition to research for all 4 case studies, the team held 2 events during the project. A Knowledge Exchange workshop spanned two days of discussion and collaboration from 23-24 February at the University of Manchester, where the research team shared the results from each case study, navigated policy gaps and agreed upon recommendations. The launch event for preliminary findings took place on 30 March at the University of Manchester, with a total of 15 people attending in-person across sectors and academic institutes.

Outputs and Outcomes

A. Czerwinsky, A. Kingdon, S. MacDonald & K. Padda (July 2026). Understanding and Responding to AI-Driven Visual Harms: A Collaborative Approach in British and Canadian Contexts.

This white paper explores how AI-driven harms manifest in synthetic images and videos shared within online environments, with the goal of understanding current gaps in regulation and response.

Download