FAQ

Frequently asked questions

What does CC BY-NC mean? Can I use this research?

CC BY-NC (Creative Commons Attribution-NonCommercial) means anyone can use, share, and build on our work freely, as long as they credit us and don't use it commercially. For non-commercial research and educational purposes, yes — go ahead, no permission needed beyond attribution. See Terms for the full scope.

Can I use it commercially?

Not without asking us first. Commercial use requires separate permission — a deliberate choice to prevent our research from being used for profit without any benefit flowing back to the people who did the work. See Approach for why, and Terms for the exact scope.

Can this content be used to train an AI model?

For non-commercial research, yes — same as any other non-commercial use of our work, no separate permission needed beyond attribution. What requires contacting us first is any use by a for-profit organization in furtherance of its business — whether that ends up in a product or service you sell, or stays purely internal (an in-house tool, unpublished internal research, process improvement, and so on). Under CC BY-NC, "non-commercial" turns on whether the use is directed toward commercial advantage, not on whether the result is ever sold or made public. We're open to discussing terms, which may include compensation. See Terms and the license field on each Zenodo record for the exact terms.

Do I need permission to cite it?

No. Citing our work in your own research, with normal academic attribution, doesn't require separate permission — that's exactly what we want people to do.

What is Zenodo and DOI?

Zenodo is a free, permanent research repository operated by CERN. A DOI (Digital Object Identifier) is a unique, permanent reference number assigned to something published there, so it can always be found and cited, even years later. See About for more.

What is SSIM?

SSIM (the Structural Similarity Index) is a widely used metric for measuring how similar two images are — often used to judge how well an AI model reconstructed or upscaled an image compared to the original. Our research is about understanding exactly when SSIM's score can and can't be trusted.

What are comp_* features?

Eight measurements (sharpness, edges, contrast, texture, noise, and others) we compute directly from a reconstructed image to diagnose its quality along dimensions SSIM alone doesn't capture. Full definitions are in DOI 1.8; see Technology.

What is DuckDB?

An open-source, embedded analytical database — think "SQLite, but built for large-scale data analysis instead of transactions." It's what lets us run and screen-record real SQL queries against hundred-million-row datasets on an ordinary PC. See Technology.

Do you rank or recommend specific AI models?

No. Our goal is to provide a methodology for testing whether a metric like SSIM can be trusted in a given situation. What counts as a "better" model varies by use case and field, so we don't think that judgment is ours to make — it belongs to each field's own community of practice. Any per-model results we publish are demonstrations of the method, not a leaderboard.

Could this mean past model comparisons using SSIM got something wrong?

It's possible, in principle. If two models are compared using an aggregate SSIM score, and one of them happens to produce more blocks near zero luminance or zero texture — regions where the metric's own terms behave very differently than at typical brightness and texture levels — its score can be pulled down, or in the opposite masking case, propped up, by an effect that has nothing to do with actual reconstruction quality. We can't say whether this happened in any specific past comparison; that would require re-running the same diagnostic on that comparison's own data. See our About page for a concrete worked example. But the possibility is exactly why we think a per-block breakdown, rather than trust in a single averaged number, is worth having available.

Can I verify the results myself?

Yes — that's the whole point. The databases themselves aren't published, due to their file size, but every technical report includes the exact SQL queries used, hash values for the underlying data files, and (for our newer reports) a screen-recorded video of the query actually running — you can verify results against these. The pipeline code that builds the databases, including heatmap_tool.py, is published in full as its own report (DOI 1.19.1), together with a recording of it actually running — so you can build the same kind of database from your own image data, not just verify ours. See Publications and Technology.

Do DOI 1.20 and 1.20.1's ground-truth-free classifiers mean ground truth is never needed?

No — they shift when ground truth is needed, not whether it's needed at all. The thresholds these classifiers rely on, mu_stat and sigma_stat, are each calibrated from ground truth once per model-and-dataset combination (using the Two-Segment Optimal Split method described in earlier reports; this calibration was done against one specific dataset, NIH ChestX-ray8, and whether the same values hold on a different dataset is untested). Once that one-time calibration exists, the same threshold can be applied to subsequent, ground-truth-free reconstructions on that same dataset to flag likely Amplification-Phase-risk blocks (DOI 1.20) or Structural-Masking-risk blocks (DOI 1.20.1) — reducing the ground-truth requirement from "every image checked" to "once per model-and-dataset combination." It's also a narrow, specific result: it applies to these two named axes only, not to SSIM diagnosis in general. See a plain-language walkthrough, or the two reports themselves, for full scope and caveats.

Can I use this on my own AI model or dataset?

Yes — that's a large part of why we released DOI 1.19.1 in full. build_notta_tta_comp_db.py accepts any ground-truth/no-augmentation/test-time-augmentation image folder, not just our own NIH ChestX-ray8 data, so you can build the same kind of database from your own model's output and run the same kind of SQL investigation on it. One caveat: some values inside heatmap_tool.py, such as the sigma_stat threshold behind the "extended caution zone," are empirically measured for our specific dataset and models — applying them to your own data as-is would show a warning zone that doesn't match your data. You'd need to re-run the Two-Segment Optimal Split method described in our technical reports on your own data first. See the beginner-friendly code walkthrough and Technology.

What is a hash value?

A short, fixed-length "fingerprint" computed from a file's contents (we use SHA-256). If even a single bit of the file changes, the hash value comes out completely different. Matching the hash value stated in a technical report against the hash of the actual data file used lets you confirm that the file shown in the video is genuinely the same file the report describes — without needing to compare the entire file byte by byte.

Why do the technical reports include videos of SQL queries running?

Rather than just stating a query and its result in text, we record the query actually running on a local PC and producing exactly the result reported. Each recording opens with the PC's system information (OS, specs) to show it's a real machine, not a fabricated result. See the Technology page for details. For queries that take a very long time to run (some over 30 minutes), we may publish a screenshot of the result instead of a full video, for practical reasons.

Does this site use cookies or track visitors?

Only with your consent. We use Google Analytics (GA4) to understand how the site is used, but the tracking script only runs after you accept it in the cookie banner — decline it and no analytics cookies are set. You can change your choice at any time via "Cookie preferences" in the footer. See Privacy for details.

I found something on this site confusing or unclear. Can I ask you to make it clearer?

Yes — please do. This site covers technical material, so some explanations may not land as clearly as we'd like. Any kind of note is welcome — "this term is hard to follow," "the SQL prerequisites aren't clear," "found a typo," or anything else. Email us directly at contact [at] merit-metrics [dot] org and we'll review it and improve the explanation where needed.