← Documentation

import_ground_truth_data.py

Computes the eight comp_* diagnostic features directly from image pixels, and imports them alongside a separately pre-computed set of global/perceptual metrics into a SQLite database. DOI 1.8's own report identifies this as the reference code behind the comp_* definitions used throughout DOI 1.0 through 1.7. Those definitions had never been formally published anywhere; DOI 1.8 exists specifically to document them, in writing, for the first time.

Location	Included in the DOI 1.8 Zenodo archive (Merutan_DOI1.8.zip) — the only place it's been published, though it was already in use well before that archive existed
Used	DOI 1.0 through 1.7, per DOI 1.8's own report — the reference code behind the comp_* definitions used throughout those earlier reports. DOI 1.8 didn't introduce it; it's the report that later formally specified, in writing, definitions that had until then only existed as code.
Status	Current, as the reference for the comp_* definitions. (For the join step that fed those definitions into the verification databases of that era, see Superseded Files.)

What it expects as input

A detailed_* directory: an images_base subfolder of original images, plus one subfolder per model, each containing one subfolder per image ID. Every one of those per-image subfolders must already hold two files this script does not generate itself — analysis_results.json (global/perceptual metrics: SSIM, MS-SSIM, PSNR, MAE, LPIPS, CLIP similarity, ΔE, LAB color statistics) and p6_local_quality_data.csv (per-block local SSIM) — produced by a separate, external analysis step upstream of this script.

What it does

For every image it finds:

1. Loads the pre-computed analysis_results.json as-is into the global_comparison_metrics table — this script doesn't recompute any of those values, only imports them.
2. For every block listed in p6_local_quality_data.csv, loads the corresponding pixel region from both the original and processed images and computes all eight comp_* features directly from the pixels: sharpness (Laplacian variance), contrast (pixel standard deviation), entropy (histogram-based), two artifact measures (block-boundary discontinuity, and edge-region variance for ringing), texture (a GLCM-based composite), noise (adaptive DCT high-frequency energy), and edges (Sobel gradient density).
3. Writes one row per block — the CSV's local SSIM plus all eight freshly-computed comp_* values, for both the original and processed pixels — to the local_comparison_metrics table.

Why it exists

This script's calculate_sharpness, calculate_contrast, calculate_entropy, detect_artifacts, calculate_texture, calculate_noise, calculate_edges, and calculate_ringing functions are the concrete, published answer to "what does comp_sharpness actually compute?" DOI 1.8's own report states these are the same definitions used throughout the DOI 1.0–1.7 reports. The definitions themselves had never been written up anywhere, only existing as this code, until DOI 1.8 was published specifically to close that gap.

For the join step that later had the data-integrity bug (and the from-scratch rewrite that replaced it, build_notta_tta_comp_db.py, used from DOI 1.12 onward), see the Superseded Files page. See the Glossary for the short version of each comp_* definition.

What it produces

A SQLite database with two tables:

• global_comparison_metrics — one row per model/image pair, holding the perceptual and global metrics taken directly from that image's analysis_results.json, plus its LAB color statistics.
• local_comparison_metrics — one row per block, holding the CSV's local SSIM plus all eight comp_* feature values computed here, for both the original and processed pixels (16 values per block in total).

Note the column names in this database (sharpness_original_block, sharpness_processed_block, and so on) differ from the comp_sharpness_notta / comp_sharpness_tta convention used in the blocks table referenced throughout this series — this database is a standalone artifact of this script's own run, not that table.

How to run it

python3 import_ground_truth_data.py \
    --root_dir /path/to/detailed_modelname \
    --db_path output.db \
    [--original_images_dir DIR] [--processed_images_dir DIR]

Run once per detailed_* directory (one per model). --original_images_dir and --processed_images_dir are optional — left unset, the script looks for an images_base subfolder for the originals and falls back to the root directory itself for the processed images.

Rerunning

Not skip-based like build_notta_tta_comp_db.py: every image found under root_dir is reprocessed on every run, and every row is written with INSERT OR REPLACE, so rerunning is safe but always redoes the full pixel-level comp_* computation rather than picking up where it left off.