Can you trust what you see when Metadata is stripped?
Once upon a time seeing was believing. Those days are gone. Today with the enhanced capabilities of AI, seeing can no longer be blindly trusted, at least not in digital form factors. This new reality puts a question mark behind many images we see online. Is the photo of that surfer real, or AI? As we have noted in the past, images can easily be manipulated for “visual deception.” One prominent study refers to this as “synthetic deception” and raises the red flag by calling it the instability of visual trust”. Fortunately, we are not left without tools that enhance our discernment and judgment.
Recently we completed a test with AI TruthTeller® on three hundred photos. We selected these photos from known sources. Every photo could be tracked to a device or source we know and trust. Why? We wanted to establish a predictable baseline that is accurate and reliable. This allowed us to validate whether AI TruthTeller®, a SaaS product used for AI detection, is correct in its assessment and identification of real photographs.
AI TruthTeller® uses private large language models and HITL (human-in-the-loop) processes for AI detection. This is relevant because one of the concerns with AI models is false positives. False positives occur when LLM’s look at a real photograph and says it is AI – this is false; and that’s a problem, it gives a positive outcome that is false. Our test of AI TruthTeller® on 300 photographs was done precisely to test for false positives and validate accuracy.
So, how did we do? AI TruthTeller® was wrong on just three of the images. That’s right, three photos were incorrectly identified as AI when in fact they are pictures taken with a camera. That’s a 99% accuracy rate when looking at real photographs. That’s pretty good in my book.
In fairness, we know there are many different types and grades of photos and we won’t score the same on every type of image file. At an obvious level we see jpg, png, webp, tiff, svg and more. In addition, when evaluating photography we need to consider animals, outdoor landscapes, professional headshots, water shots and product images. Each one has subtle differences in light, colors and definitions. Image detection in a digital world is not necessarily simple. And what about the 300 images we tested? This particular set of images was primarily of people; people of all ages and in a variety of settings - portraits, family outings, athletic competitions and so on. In addition, to the private collection of three hundred pictures we selected and tested nine photos that we could post here. They are representative of the total set and are identified for each photographer.
Another point to consider with image detection is where the image is hosted and how has it been processed. When working with images, we use pixel analysis and algorithms from LLM’s but also forensic evidence from the image file itself. When it’s available, we use device level information. However, for privacy concerns a great deal of metadata is stripped when uploading an image to social media platforms. Consider the following chart and what it says about images found on social media platforms:
If the major social media platforms strip metadata from uploaded photographs, then why do we look for these elements when assessing a photo? AI TruthTeller® is designed for broad AI detection and not only for social media. There are many situations where images bypass these platforms and are sent directly to a recipient. Text messaging, chat bot submissions, product returns to companies, family history research activities, headshot submissions to HR departments, creative arts, property rentals, tourism marketing and many more.
Back to the original question and the surfer. Can you trust what you see? Yes, if it is a real photograph and you are using AI TruthTeller® in an environment that gives you access to the original file. The surfer shot is a real photo. Ninety-nine percent is a strong benchmark and a great position to work from. If you are on social media, you can still have high confidence in the app and the assessed judgments, but it’s beneficial to validate through multiple sources and techniques (click here to learn more).