Meritcs 日本語

← Publications

Catching SSIM's Blind Spots Without Ground Truth — Two Diagnostic Tools

A plain-language explanation of DOI 1.20 and DOI 1.20.1, written for readers without specialized background.

This series has so far characterized SSIM's blind spots (regions where brightness differences get amplified, and regions where texture gets erased) using ground-truth (GT) images. Here, we introduce two diagnostic tools that catch these same blind spots without needing GT for each new image you check. The calibration values themselves (μstat/σstat) still have to be set once per model-and-dataset combination using GT (this calibration was done against one specific dataset, NIH ChestX-ray8; whether the same values hold on a different dataset is untested). GT isn't eliminated entirely -- it shifts from being needed for every image, to being needed once per model-and-dataset combination.

Catching the "brightness errors stand out too much" danger zone, GT-free (DOI 1.20)

Have you ever taken a photo with your phone of a blank wall, or a cloudless sky? Because there's nothing else in the frame, even the faintest unevenness in brightness stands out clearly. In a busy, detailed photo, the same amount of unevenness would go completely unnoticed. SSIM's brightness-scoring term has the same property: in regions where the ground-truth image is perfectly flat with no texture at all, even a tiny brightness difference has an outsized effect on the score (this series calls it the "Amplification Phase").

The value used to spot this danger zone (μnotta) is computed by picking a root of a quadratic equation -- a method that can genuinely be imprecise. But here, that imprecision doesn't actually matter. Restricting to blocks where the GT side is perfectly flat, real brightness values cluster tightly onto one of 256 possible levels. The classification threshold falls at a point that doesn't line up exactly with any of those levels, so the question becomes much coarser: not "exactly which level," but "which side of the threshold." A somewhat imprecise estimate is still enough to get that coarser question right.

Tested across four models, this achieves 96.0%–100% accuracy -- and not a single truly at-risk block was missed, in any of the four models (100% recall).

Catching the "texture gets erased" danger zone, GT-free (DOI 1.20.1) -- comp_contrast, a hidden exact ruler

The other blind spot is when an AI model erases texture that should genuinely be there (Structural Masking). This one became diagnosable GT-free through a different route than the one above.

The core discovery: an existing feature turned out to be exactly the same quantity

This series has long measured "contrast strength (σ)" by dividing images into small squares. Separately, for a different purpose, it had already been measuring a feature called comp_contrast.

Two independent measurements of the same block turn out to be the same number Starting from one 8-by-8 image block, two separate calculations are made for two different purposes: sigma_gt, and comp_contrast_gt divided by 255. The two paths converge on an exact match, with zero error. One 8×8 image block σgt used throughout this series comp_contrast_gt ÷ 255 a separate diagnostic feature = exact match, zero error

Verified across all four models and 326 million blocks — no exceptions.

What turned up: dividing comp_contrast by 255 gives exactly σ, with zero error (255 is the maximum pixel value for an 8-bit image, 28−1).

This isn't "close" or "similar" -- it's the same mathematical quantity, measured twice, by two different methods, for two different purposes, without anyone noticing. Verified in SQL against all four models and all 81,564,041 blocks each (about 326 million blocks combined across all four) -- not a single exception (this verification is backed by a screen-recorded video and file hashes, published on Zenodo).

Why this matters

Until now, finding a reconstruction's own contrast strength required working backward through SSIM's formula -- a calculation with a known flaw: it produces two candidate answers, with no reliable way to tell which one is correct. (This is actually the same kind of method used for μnotta in DOI 1.20 above. There, that imprecision happened not to matter; for σ, it did.) An empirically-tuned correction factor was considered as a workaround, but that amounted to searching for "a value that happens to work on this particular dataset" -- never a fully confident fix.

Once comp_contrast turned up as an exact ruler, both the workaround and the search for a correction factor became entirely unnecessary. No tuning, no approximation -- just a direct, error-free value.

How well does this work -- and where is it still weak?

Using this exact ruler, a simple rule based purely on the model's own output gives, across all four models:

Overall accuracy98.8%–99.5%
Extremely high combined judgment accuracy
Masking detection rate (recall)99.1%–99.95%
Catches almost every true case of erased texture
Non-masking retention rate (recall)42.8%–78.8%
Reflects each model's own reconstruction habits

Both the overall accuracy and the rate of correctly catching truly-erased-texture blocks are very high. But correctly clearing blocks that genuinely still have their texture is lower and model-dependent, at 42.8%–78.8%.

What this number actually means -- the smartphone "beauty mode" analogy

This 42.8%–78.8% figure doesn't mean the diagnostic method itself is imprecise. It reflects an accurate fact about the model's own reconstruction.

Take a photo of a person or fabric with a smartphone camera, and automatic smoothing (noise reduction) can erase real fine detail -- skin texture, the weave of fabric -- leaving it looking artificially smooth. How aggressive this smoothing is varies a lot between phone models and apps.

This analysis doesn't look at whole images -- it divides each image into small square blocks and checks, block by block, whether real texture was there, and whether it survived reconstruction. It's essentially counting, one block at a time, cases where "there really was texture here, but smoothing erased it."

The number represents: out of 100 blocks where the real image genuinely has fine detail, how many still have that detail after reconstruction. A lower number means smoothing is more aggressive, and more likely to erase detail that should have survived.

Concretely: Upscayl High-Fidelity scores 42.8% -- of 100 blocks with real texture, about 43 keep it, and about 57 get smoothed away. Digital-Art scores 78.8% -- about 79 out of 100 keep their texture, with only about 21 smoothed away. Same kind of smoothing, very different strength depending on the model.

Out of 100 blocks with real texture, how many keep it after reconstruction Two 10-by-10 grids of 100 squares each, one per model, where each square is one ground-truth block that genuinely has texture. Filled squares are blocks where the model's reconstruction kept that texture; faded squares are blocks where it was smoothed away. Upscayl High-Fidelity keeps 43 of 100; Digital-Art keeps 79 of 100. Upscayl High-Fidelity 43 of 100 blocks keep their texture Digital-Art 79 of 100 blocks keep their texture texture kept texture smoothed away

Each square is one ground-truth block that genuinely has texture — filled means the model's reconstruction kept it, faded means it was smoothed away. Same kind of smoothing, very different strength per model.

Because this ruler (σ, via comp_contrast) is itself measured with zero error, this spread isn't measurement noise -- it's a direct reflection of each model's own reconstruction habits.

Want more detail?

Technical detail (verification SQL, screen recordings, file hashes) is published in the Zenodo records for DOI 1.20 and DOI 1.20.1.