← Documentation

generate_axis5_tier3_full_db.py

Applies an already-fitted Axis5 model to a target database and joins the result against analysis_delta's Tier3 states, producing axis5_full — one piece of the larger analysis system referenced in the origin story on About, and the specific piece DOI 1.7's Section 4 results were queried from.

LocationIncluded in the DOI 1.7 Zenodo archive (Merutan_DOI1.7.zip)
RoleThe inference-only pipeline: reads the weights and thresholds already saved in axis5_history by sorted_phase_v13_full.py, applies them to a target database, and exports the result as axis5_full. comp_* is computed raw, without mean subtraction: xb,i × wi.
Used byDOI 1.6 and DOI 1.7 — its output, axis5_full, is what those reports' Tier3/Axis5 SQL results and the four screen-recorded videos released with DOI 1.7 were run against. (Those videos ran against a further-assembled database named meltan_integrated_viewer_*.duckdb; this script's output reproduces every value those videos and SQL queried from it in Section 4 — see What it produces below for how the two relate.)
StatusCurrent, as documented in DOI 1.7 — a different concern entirely from the JOIN-based verification pipeline that was corrected and replaced (see Superseded Files). No bug or correction has been published against this script.

What it does

  1. Prompts (via a file-picker dialog) for a source database, and creates a new output database named after it, suffixed _axis5_results_v2.duckdb.
  2. Attaches the source database read-only, reads the most recently saved model configuration from its axis5_history table (feature list, means, inverse covariance matrix, feature weights — the same values sorted_phase_v13_full.py fitted and saved), and applies that fixed model to every block in the source's analysis_full table.
  3. Computes each block's Mahalanobis distance D from that saved model, derives P50/P75 and the Tukey-fence thresholds T1/T2 from the resulting distribution, and writes an axis5 severity value (−2 through 1) plus a primary_cause column (which single comp_* feature contributed most to each block's distance) into a new axis5_results_v2 table.
  4. Joins axis5_results_v2 against the source's analysis_delta table on (image_id, row, col), pulling in each block's noTTA/TTA Tier3 values, local SSIM, and Δssim, and derives a combined tier3_state label (Stable / Persistent / Escape / BlindSpot — see the Glossary) — the result is written as axis5_full, the table DOI 1.7's Section 4 SQL queries directly.
  5. Copies axis5_history into the output database unchanged, appending the P50/P75/T1/T2 thresholds actually used in this run to its stored configuration for the record.
  6. Records run metadata (creator, license, generation timestamp, source filename, and the thresholds used) into a merutan_metadata table in the output database. A Parquet-export path with SHA-256 hashing exists in the code but is currently disabled.

What it produces

A DuckDB database (…_axis5_results_v2.duckdb) with four tables: axis5_results_v2 (Axis5 inference results before the Tier3 join), axis5_full (the joined table — Axis5 plus Tier3 state, the one queried directly in DOI 1.7's report), axis5_history (the copied model configuration and thresholds), and merutan_metadata (run provenance). Run once per model. This script's own output database is not the same file as the meltan_integrated_viewer_*.duckdb files referenced in DOI 1.7's videos and SQL — a further, unpublished step (generate_integrated_viewer_db.py) combines this script's axis5_full with the corresponding Axis1 and Axis6 databases from the same pipeline (the larger system referenced on About) into that final database. That database's schema has been public since DOI 1.0's own appendix (Section A.2); the raw file itself isn't distributed, only because of its size (DOI 1.0 says so directly, Section A.3.3) — the same practical reason later verification databases in this series aren't distributed either. What's published here is narrower than the full assembly pipeline, but reproduces what DOI 1.7 actually needed: running this script (plus sorted_phase_v13_full.py before it) reproduces every value DOI 1.7's Section 4 queried from meltan_integrated_viewer_*.duckdb.

How to run it

Interactive, with no command-line arguments: running the script opens a file-picker dialog to choose the source database. The output database is written alongside it, in the same folder.