About
About Meritcs
Meritcs /ˈmɛrɪtɪks/ — pronounced “Meritics”
A blend of "Merit" and "Metrics" — judging whether a quality metric has actually earned trust, or only appears to.
MERIT — Metric Reliability & Integrity.
Establishing, mathematically and empirically, exactly where a quality metric like SSIM can and cannot be trusted is the consistent thread running through everything we do.
A question about SSIM being evaluated as a single average over an entire image.
Iterative refinement of earlier visualization models led to the Shell / Membrane / Yolk structure.
Systematic block-level analysis, using the egg metaphor's 3D/2D system, led to the closed-form derivation.
Applied for independent, third-party review, and was approved by the community's curator.
Both viewers had already appeared in DOI 1.2's demonstration videos; DOI 1.5 later formally documented both systems and corrected a classification.
A JOIN-based data-integrity bug required re-verifying published work, and led to the design of an entirely new, JOIN-free data pipeline.
Origins
Founding motivation
This research began with a question: why is SSIM evaluated as a single average over an entire image?
Development of the visualization model
To examine what that average was concealing, several visualization models were developed and tested. Earlier iterations proved unworkable once extended into three-dimensional space. Continued refinement produced the "egg metaphor": a three-layer model of Shell, Membrane, and Yolk.
What block-by-block analysis revealed
Using a 3D/2D visualization system built on this model, block-level analysis was carried out systematically, comparing individual blocks one at a time.
The low-scoring block hadn't disappeared — it had simply become invisible within the average.
Statistical significance of small differences
This example shows how block-level variation can get buried in an image's average score. A related concern emerged from the broader literature: models are frequently ranked and compared on SSIM differences as small as 0.01. Given the instability documented above, such fine margins carry limited statistical meaning — a concern with direct implications wherever SSIM is used to evaluate medical imaging. This concern motivated closer, block-by-block examination, which led to a closed-form derivation of the underlying statistical behavior.
Luminance Bias
One block from this examination made the mechanism concrete: a completely flat, pure-black ground-truth block (zero mean luminance, zero variance) compared against a reconstruction shifted by a single 8-bit level — a difference imperceptible to the eye — produced a computed SSIM of 0.867. The drop traced entirely to the luminance term, which behaves very differently near zero luminance than at typical brightness levels. This single case became the starting point for deriving the closed-form boundary at which the term's behavior changes.
Both blocks look identical to the eye — the reconstruction differs by one 8-bit level out of 255 — yet SSIM's luminance term alone scores the pair 0.867.
Structural Masking
The reverse direction happens too: a second block from the same dataset had a ground-truth block with real, measurable texture (a comp_texture score of 5.05, comp_entropy of 2.29), reconstructed as completely flat — the model erased the texture entirely (comp_texture and comp_entropy both dropped to 0). Despite this genuine loss of detail, the computed SSIM was 0.975. The same masking mechanism is at work here, this time triggered by the reconstruction's own near-zero variance rather than the ground truth's.
The reconstruction erased real, measurable texture entirely — yet SSIM still scored the pair 0.975.
The visualization framework used throughout this analysis was itself later formally documented as its own contribution: the egg-metaphor framework was developed into two working systems, the Merutan DB 3D Viewer and the Merutan DB Viewer (2D), documented on the Demonstrations page. Both viewers had already appeared in DOI 1.2's demonstration videos; a dedicated DOI later formally documented both systems, together with a corrected classification.
Methodology
Broader context
Transparency and sample size in AI research
Beyond this initial result, a broader pattern is apparent in AI research: a persistent lack of transparency in reported methods, combined with a reproducibility crisis across the field. A related concern is sample size — many published conclusions rest on evaluations of only a handful of images.
Both issues share a common root cause: no independent means exists to verify the underlying claim. Open science addresses this directly by publishing the complete process, data, and methods, allowing anyone to verify exactly how much evidence a result rests on. This methodology was adopted as standard practice, using Zenodo and its DOI system — a permanent identifier assigned to each publication — as the primary publication venue.
What is CERN?
CERN (the European Organization for Nuclear Research) is the world's largest particle physics laboratory, headquartered near Geneva, Switzerland. Founded in 1954, it operates the Large Hadron Collider and is known for discovering the Higgs boson in 2012. It also has a history of building open information infrastructure -- Tim Berners-Lee invented the World Wide Web while working there in 1989.
Where the name "Zenodo" comes from
Zenodo is a free repository CERN built and operates as part of the OpenAIRE project, funded by the European Commission, launched in May 2013. It's named after Zenodotus, the first librarian of the ancient Library of Alexandria and the first known creator of metadata. It has since grown to about 25 million visits a year, more than 3 million uploads, and over a petabyte of data.
Independent review
Admission to a curated AI/ML community
Following the publication of DOI 1.0, an application was submitted to Zenodo's Artificial Intelligence and Machine Learning community — a curated, membership-reviewed community dedicated to AI and machine learning research, built on infrastructure provided by the CERN Data Centre, OpenAIRE, and the European Commission, and among the more active communities on Zenodo. The application was reviewed by the community's curator and approved.
This review confirmed a founding conviction: verifiability is the foundation on which researchers earn trust. Meritcs exists to put that conviction into practice.
Data-integrity correction
A subsequent internal audit identified a data-integrity bug in the database construction process: certain SQL JOIN operations could silently pull values from the wrong block. Addressing it required re-verifying previously published work. The investigation led to the design of an entirely new, JOIN-free data pipeline that structurally eliminates this class of error, rather than patching individual instances.
Publication process
Every DOI cited above went through the same three-step process:
Every step is designed to be checked, not just trusted.
Acknowledgments
This website was established following a suggestion by Dr. Wataru Ichikawa, Vice-Chairman of JACCRO, to create an organizational platform that would enable us to independently disseminate our research findings.
His advice on the structure and presentation of academic manuscripts also directly led to the identification of an actual error, resulting in the correction of the C1/C2/C3 constants published as DOI 1.10. We are deeply grateful for both his insightful suggestion and his invaluable academic guidance.
We would also like to express our sincere appreciation to JACCRO (http://www.jaccro.com/), with which Dr. Ichikawa is affiliated. JACCRO is a Japanese non-profit organization dedicated to planning and supporting high-quality, multi-institutional clinical trials and clinical research for patients with cancer, thereby contributing to advances in cancer treatment.
