build_notta_tta_comp_db.py (+ build_notta_tta_sigma_db.py)

Builds the four per-model databases behind every verification SQL report in this series from DOI 1.12 onward — one row per 8×8-pixel image block, with SSIM and the eight comp_* features computed independently for the ground-truth, no-augmentation, and test-time-augmentation versions of every reconstruction.

Location	Originally included in the DOI 1.12 Zenodo archive (`Merutan_DOI1.12.zip`), unmodified since; released again on its own, unedited, together with `heatmap_tool.py` and a screen-recorded demo, as DOI 1.19.1
First established	DOI 1.12 (as the fix for the join-based data-integrity bugs described on the Technology page)
Used by	DOI 1.12 through 1.16 — the four databases it produces are reused unchanged, never rerun or edited, by every later report in the series
Status	Current — no superseded version exists

New to this codebase? See the beginner-friendly, section-by-section code walkthrough →

What it does

For every image that has a ground-truth version, a no-augmentation reconstruction, and a test-time-augmentation reconstruction, this script:

Loads all three images and splits each into non-overlapping 8×8-pixel blocks.
Computes SSIM for each block, twice — ground truth vs. no-augmentation, and ground truth vs. test-time-augmentation — keeping the luminance, contrast, and structure sub-terms separate, not just the final combined score.
Computes all eight comp_* features (sharpness, edges, two artifact measures, contrast, texture, entropy, noise) for each block, independently, from the ground-truth pixels, the no-augmentation pixels, and the test-time-augmentation pixels — 24 feature values per block in total.
Writes one row per block, plus one summary row per image, to a local database file.

Why it exists

This script is the concrete implementation of the design change described on the Technology page: computing SSIM and comp_* directly from the same set of loaded images in a single pass, rather than joining separately-generated tables together afterward. That earlier join step was a repeated source of silent data-integrity bugs at the scale of hundreds of millions of rows — see that page for the full story.

What it produces

One DuckDB database file per run — pointed at one ground-truth folder and two reconstruction folders (no-augmentation and test-time-augmentation) — containing two tables. If you're comparing multiple models, run it once per model, each against its own pair of reconstruction folders; that's how we produced the four databases (Standard, SwinIR, High-Fidelity, Digital-Art) behind our own published reports.

images — one row per source image, recording its file paths and dimensions.
blocks — one row per 8×8-pixel block, grouped into: the block's position (image, row, column); its ground-truth statistics (local brightness and local standard deviation); its SSIM sub-terms and total score against both reconstructions; and its 24 comp_* feature values (8 features × ground-truth/no-augmentation/test-time-augmentation).

How to run it

Requires Python 3 with numpy, pandas, duckdb, pillow, opencv-python-headless, and scikit-image installed, plus heatmap_tool.py in the same directory — this script imports several functions from it directly. heatmap_tool.py's own per-image visualization feature is still being refined and will get its own dedicated technical report once that work is finished; its source code is published as-is alongside this script in DOI 1.19.1 below.

python3 build_notta_tta_comp_db.py \
    --gt-dir /path/to/gt_images \
    --notta-dir /path/to/notta_recon \
    --tta-dir /path/to/tta_recon \
    --out std_notta_tta_comp.duckdb \
    [--workers N] [--chunk-size 20] [--limit N]

A single run compares one pair of reconstruction folders against one ground-truth folder. If you're comparing several models against the same ground truth, run it once per model, reusing the same --gt-dir each time. --workers controls how many images are processed in parallel (default: all CPU cores minus two, so the machine stays usable during a run); --limit caps how many new images are processed, useful for a quick test before committing to a full run.

Resumability

Safe to interrupt at any point — a crash, a reboot, or a manual stop. On restart with the same command, any image already recorded in the output database's images table is skipped automatically, so an interrupted run simply picks up where it left off rather than starting over or risking duplicate rows.

Released standalone (DOI 1.19.1)

View DOI 1.19.1 on Zenodo: https://doi.org/10.5281/zenodo.22838600

This script and heatmap_tool.py are also published together as their own report, exactly as they stood on 2026-09-01 — the same code that produced every number in DOI 1.12 through 1.16. Anyone can point it at their own ground-truth/no-augmentation/test-time-augmentation image folders and build the same kind of database from their own data and models. The recording below shows an actual run (--limit 20), executed twice in a row to demonstrate resumability: the second run correctly detects the 20 images already processed and moves on to the next batch.

Screen recording: two consecutive --limit 20 runs, unedited.

The DOI 1.19.1 release also documents two limitations carried over unedited from this state: heatmap_tool.py's generated HTML explanatory text is Japanese-only, and its SIGMA_STAT dictionary is hardcoded to this project's own dataset and four models (SIGMA_C is the universal, closed-form value and needs no such caveat). Anyone applying this to their own dataset or model should re-derive their own SIGMA_STAT value rather than reuse these as-is.

Extended for DOI 1.21: `build_notta_tta_sigma_db.py`

View DOI 1.21 on Zenodo: https://doi.org/10.5281/zenodo.23271571

build_notta_tta_sigma_db.py is a variant of this script, written for DOI 1.21. It depends on the same heatmap_tool.py and follows the same overall approach (one independent, join-free database per model, resumable in chunks), but extends the schema: alongside the derived comp_* features this script stores, it additionally stores σ_notta/μ_notta/σ_tta/μ_tta as raw values. This was needed to measure DOI 1.21's floating-point ghost duplication correction directly, rather than through the derived features alone.

Screen recording: two consecutive --limit 20 runs, unedited — the same resumability test as the DOI 1.19.1 recording above, run against this extended schema.

See the DOI 1.21 SQL walkthrough for how the resulting database is used.