Human vs. Machine Deep Fake Image Detection Competition at DEF CON

UL Research Institutes’ Digital Safety team joined our partners, Lockheed Martin and Aptima, at DEF CON in Las Vegas—the longest-running and largest hacking conference in the world. At the show, we created a deep fake image detection game for participants and SemaFor.

What is SemaFor?

SemaFor (short for Semantic Forensics) is a platform dedicated to advancing analytics for deepfake detection. The platform employs automated detectors to assess whether audio, images, or video content has been generated or altered.

At DEF CON’s AI Village, we showcased SemaFor’s ability to spot synthetic images and compared its performance against the sharp instincts of human participants.

Results

• Participants: ~5,000 guesses
  • Accuracy: 77%
• SemaFor:
  • Accuracy: 93%

Key Takeaways

The DEF CON game underscored that synthetic media detection is not a solved problem—it’s an ongoing, troublesome issue. While SemaFor substantially outperformed humans overall, both struggled with certain edge-case images, showing how quickly generative technologies are advancing.

To stay ahead, our team is committed to hosting open challenges where researchers, technologists, and curious participants can develop new, cutting-edge evaluation algorithms and detection methods.

Building on This Approach

Building on this approach, UL Research Institutes is launching the SAFE: Synthetic Video Detection Challenge at ICCV 2025.

This effort will push innovation in detecting and attributing manipulated video, with a focus on:

• Generalizability across diverse domains
• Robustness against evolving generative techniques
• Scalability for real-world deployment

By mobilizing the research community through these challenges, we aim to expose weaknesses, drive breakthroughs, and strengthen global efforts in media integrity and trust.