A Collaborative Approach to Assessing the Challenges of AI-Driven Harms

Summary

Amidst a rapid proliferation of synthetic image- and video-based content in online ecosystems, our research addresses urgent gaps in understanding AI-driven harms by examining real-world cases of extremist visual content and evaluating existing AI-powered moderation solutions. This project seeks to document how generative AI enables and accelerates visual hate speech; assess the feasibility of AI-supported content moderation tools for detecting harmful visual content; and develop evidence-based recommendations for policymakers and platform developers to mitigate these harms while preserving legitimate expression. ​

Four rapid case studies: ​​

• visual content from ‘Unite the Kingdom’ protests​​
• meme-based extremism in manosphere spaces​​
• fascist aesthetics in AI-generated videos created by Sora2​​
• opportunities for AI-supported flagging of harmful content through existing moderation tools like SightEngine.​

Collaborating with University of Southampton and Waterloo University, Canada.